Latest revision as of 06:22, 10 July 2018

{{#vardefine:ADT_MAUAL | 6539}} {{#vardefine:ADT_AUTO | 6540}} {{#vardefine:ANYQEMU_SATO_MANUAL | 6563}} {{#vardefine:ANYQEMU_SATO_AUTO | 6564}} {{#vardefine:ANYBSP_SATO_AUTO | 6558}} {{#vardefine:ANYBSP_LSB_AUTO | 6561}} {{#vardefine:ANYBSP_WIC_AUTO | 6560}} {{#vardefine:MMAX_CORE2_MANUAL | 6554}} {{#vardefine:MMAX_I7_MANUAL | 6557}} {{#vardefine:NUC_I7_MANUAL | 6556}} {{#vardefine:NUC_GEX86-64_MANUAL | 6555}} {{#vardefine:MMAX_GEX86-32_MANUAL | 6553}} {{#vardefine:GAL_SATO_AUTO | 6565}} {{#vardefine:CROPS_AUTO | 6541}} {{#vardefine:CROPS_MANUAL | 6542}} {{#vardefine:ECLIPSE_MANUAL | 6543}} {{#vardefine:KERNEL_MANUAL | 6714}} {{#vardefine:METAYOCTO_MANUAL | 6544}} {{#vardefine:TOASTER_AUTO | 6545}} {{#vardefine:TOASTER_MANUAL | 6546}} {{#vardefine:ESDK_MANUAL | 6547}} {{#vardefine:ESDK_AUTO | 6548}} {{#vardefine:OE-CORE_AUTO | 6549}} {{#vardefine:BITBAKE_AUTO | 6550}} {{#vardefine:RUNTIME_AUTO | 6551}} {{#vardefine:BUILD-APPLIANCE_AUTO | 6552}}

Reversion history

Version	Modifier	Comments
1.0	Jose Perez Carranza	First draft

References

Documents that support this test plan

Introduction

This is the the overall test plan for version 2.3 of the Yocto Project, for all the generic information this document will be pointing to QA Master Test Plan

Objectives

This test plan for Yocto Project 2.3 release supports the following objectives:

To define the the tools to be used throughout the testing process.
To communicate to the responsible parties the items to be tested, set expectations around schedule, and define environmental needs.
To define how the tests will be conducted.
Release Criteria
Risks and contingencies

QA_Master_Test_Plan#Objectives_and_Tasks

Test Strategy

Is followed the general strategy defined on QA_Master_Test_Plan#Testing_Strategy and below is showed specific information applicable only for 2.3 Release

Bugs Tracking

Bugs that need to be implemented by QA Team
- 2.3 QA Assigned Bugs

Bugs that need to be verified by QA Team
- 2.3_QA_Bugs_To_Verify

Old Bugs that need to be Verified
- Old resolved bugs and features

Test Items

The Yocto Project QA team verifies each release against recent versions of the most popular Linux distributions that provide stable releases.

* centos 7 
* Debian 8
* Fedora 23 
* Fedora 24
* OpenSuse132 
* OpenSuse421 
* ubuntu 14.04 
* ubuntu 16.04

Test Cases

Yocto 2.3 Test Cases

Components to be tested

The components to be tested for version 2.3 of the Yocto Project against the different milestones are included in this section:

Code of Colors
Active Tracked on Testpoia
Active Non Tracked on Testopia
In Design Will be tracked on Tesopia
Inactive

COMPONENT	TEMPLATE
ADT	TRTEMP_2.3_MANUAL_ADT
ADT	TRTEMP_2.3_AUTO_ADT
BSP	TRTEMP_2.3_AUTO_BSP/QEMU_core-image-sato-sdk_ANYQEMU
	TRTEMP_2.3_AUTO_BSP/QEMU_core-image-sato-sdk_ANYBSP
	TRTEMP_2.3_AUTO_BSP/QEMU_core-image-lsb-sdk_ANYBSP
	TRTEMP_2.3_AUTO_BSP/QEMU_core-image-sato-sdk-wic_ANYBSP
	TRTEMP_2.3_AUTO_BSP/QEMU_core-image-sato-sdk_intel-quark
	TRTEMP_2.3_MANUAL_BSP/QEMU_core-image-sato-sdk_ANYQEMU
	TRTEMP_2.3_MANUAL_BSP/QEMU_core2-32_MinnowMax
	TRTEMP_2.3_MANUAL_BSP/QEMU_corei7-64_MinnowMax
	TRTEMP_2.3_MANUAL_BSP/QEMU_corei7-64_NUC
	}} TRTEMP_2.3_MANUAL_BSP/QEMU_GenericX86-64_NUC
	TRTEMP_2.3_MANUAL_BSP/QEMU_GenericX86_MMAX32
CROPS	TRTEMP_2.3_AUTO_CROPS
CROPS	TRTEMP_2.3_MANUAL_CROPS
Eclipse-Plugin	TRTEMP_2.3._MANUAL_Eclipse-Plugin
Kernel	TRTEMP_2.3_MANUAL_Kernel
Meta-Yocto	TRTEMP_2.3_MANUAL_Meta-yocto
Toaster	TRTEMP_2.3_AUTO_TOASTER_UI
Toaster	TRTEMP_2.3_MANUAL_TOASTER_UI-managed-mode
eSDK	TRTEMP_2.3_AUTO_eSDK
eSDK	TRTEMP_2.3_MANUAL_eSDK
OE-CORE / BitBake	TRTEMP_2.3_AUTO_OE-Core
OE-CORE / BitBake	TRTEMP_2.3_MANUAL_BitBake
Runtime (Compliance)	TRTEMP_2.3_AUTO_Runtime
Build Apliance	TRTEMP_2.3_MANUAL_Build-Appliance
pTest
Performance	Performance Test
WUpd (Software Updater)
Distributed Builds
MRAA
Meta-SwUpd

All component to be tested on 2.3 are in #Test Items a detailed list of features per component can be found in Test Plan of the components QA_Master_Test_Plan#Component

Features pending to be tested

Here is a list of areas that are not yet covered and need some work, implement test cases and add it to a component that belongs or create a new test plan for new component.

Features
Bug ID	Description
Bug 10723	Build-Appliance - Run builds in the build-appliance image
Bug 10724	Verify if -c menuconfig work
Bug 10724	Verify if -c devshell work
Bug 10725	Verify if BBPOSTCONF and BBPRECONF work in bitbakek
Bug 10726	Verify if oe-pkgdata-util print help when no parameters specified
Bug 10727	Verify if buildhistory_analsysis.py is tested
Bug 10728	Verify combo-layer
Bug 6926	Verify yocto-bsp/yocto-layer
Bug 10724	Verify bitbake -u depexp -g XXX

The complete List of QA Implementations for YP 2.3 can be found at Features Implementation by QA Team

Features Not to be tested

Approach

Tests will be conducted per the documented test cases stored in Testopia. The test manager will create test runs for each tester. The tester will execute the tests in Testopia and mark each case as Pass / Fail / Skip /Blocked. The tester should leave notes on actual results and any other relevant details when possible.

Once complete, the test manager should review the test run reports in Testopia and report back to the team by mail and in the wiki as a test report.

Pass / Fail Criteria

This criteria is defined in Yocto_Project_v2.3_Status#Yocto_Project_v2.3_Release_Criteria

Exit Criteria

As defined in QA_Master_Test_Plan#Entry and Exit Criteria

Test Deliverables

Live Status 2.3_QA_Status
Final execution Report #Execution History
Mail sent to the mailing list

Test Environment

There are different environments and tools for every type of testing refer to QA_Master_Test_Plan#Test_Areas for more detailed information.

Schedule

M1 (Oct. 31, 2016 to Dec. 23, 2016 -- 9 weeks: planning done in week 1, development week 2-6, stabilization week 7-8, release week 9)

M1 Feature/Tasks Development / Cutoff is Dec. 12, 2016

ID	Requester	Assignee	E	Summary (6 tasks) ⇒	Severity	P	Whiteboard	Status
9088	Aníbal Limón	Francisco Pedraza	3	Extensible SDK add tests for devtool	enhancement	Medium+		RESOLVED
10753	Francisco Pedraza	Francisco Pedraza	5	QA Kernel-dev test cases basic functionality	enhancement	Medium+		VERIFIED
10394	Jair Gonzalez	Jair Gonzalez	2	Creation of component to emulate keyboard with USB-KM232	enhancement	High		RESOLVED
7376	Ross Burton	Jose Perez C	3	oe-selftest doesn't run lib/oe/tests/*	enhancement	Medium+		VERIFIED
8433	Ross Burton	Jose Perez C	4	Add test for package postinsts	enhancement	Medium+		RESOLVED
8584	Ross Burton	Juan Ramos	3	Enable building API documentation	enhancement	Medium+	21 Nov 2016: RESOLVED	VERIFIED
			20

M1 Bugs

ID	Requester	Assignee	E	Summary (4 tasks) ⇒	Severity	P	Status
5319	Mark Hatle	Francisco Pedraza	3	Need a test to verify firstboot postinst script order	normal	Medium+	RESOLVED
10432	Leonardo Sandoval Gonzalez	Francisco Pedraza	2	eSDK test test_install_libraries_headers fail when distro is nodistro	normal	Medium+	VERIFIED
10481	Jair Gonzalez	Jair Gonzalez	1	[Test Case 1059] Parselogs is failing on genericx86 (32 and 64 bits) WIC giving error: GPT: Use GNU Parted to correct GPT errors.	normal	Medium	RESOLVED
10594	Jose Perez C	Jair Gonzalez	2	Ensure all options of WIC script are tested	normal	Medium+	VERIFIED
			8

M1 Stabilize schedule

Release Candidate and Milestone Release schedule

RC1: 12/12/16
M1 release: 12/23/16

The planed schedule for 2.3 can be found at Yocto_2.3_Schedule

Live Schedule

MILESTONE	STARTED	FINISHED	STATUS
M1 rc1	12/19/2016	12/23/2016	DONE
M2 rc1	01/24/2017	01/31/2017	DONE
M2 rc3	02/03/2017	02/08/2017	DONE
M3 rc1	03/15/2017	03/23/2017	DONE
M3 rc2	03/28/2017	04/05/2017	DONE
M3 rc1 Meta-Intel	04/05/2017	04/12/2017	DONE
M4 rc2	04/21/2017	04/29/2017	DONE
M4 rc6	05/02/2017	05/08/2017	DONE
2.3 Meta-Intel rc1	05/11/2017	05/16/2017	DONE
2.3 Meta-Intel rc4	05/22/2017	05/25/2017	DONE
2.3.1 rc2	07/12/2017	07/18/2017	DONE
2.3.2 rc1	09/12/2017	09/20/2017	DONE
2.3.3 rc1	12/15/2017	12/20/2017	DONE
2.3.4 rc1	6/13/2018	7/10/2018	DONE

Execution History

MILESTONE	REPORT
M1 rc1	WW52 - 2016-12-19 - Full Test Cycle 2.3 M1 rc1
M2 rc1	WW05 - 2017-01-24 - Full Test Cycle 2.3_M2 rc1
M2 rc3	WW06 - 2017-02-03 - Full Test Cycle 2.3_M2 rc3
M3 rc1	WW11 - 2017-03-15 - Full Test Cycle 2.3 M3 rc1
M3 rc2	WW14 - 2017-03-28 - Full Test Cycle 2.3 M3 rc2
M3 rc1 Meta-Intel	WW15 - 2017-04-05 - Full Test Cycle - 2.3 M3 - meta-intel
M4 rc2 (2.3 release - pyro)	WW17 - 2017-04-21 - Full Test Cycle 2.3 rc2
M4 rc4 (2.3 release - pyro)	WW19 - 2017-05-02 - Full Test Cycle 2.3 rc4
Meta-Intel rc1 (2.3 release - pyro)	WW20 - 2017-05-11 - Full Test Cycle - 2.3 rc1 - meta-intel
Meta-Intel rc4 (2.3 release - pyro)	WW21 - 2017-05-22 - Full Test Cycle - 2.3 rc4 - meta-intel
2.3.1 rc2	WW29 - 2017-07-18- Full Test Cycle - 2.3.1 rc2
2.3.2 rc1	WW38_-_2017-09-20-_Full_Test_Cycle_-_2.3.2_rc1
2.3.3 rc1	WW51_-_2017-12-20-_Full_Test_Cycle_-_2.3.3_rc1
2.3.4 rc1	WW25_-_2018-06-22-_Full_Test_Cycle_-_2.3.4_rc1

@@ Line 325: / Line 325: @@
 | 02/08/2017
 | DONE
-|- bgcolor="Gray"
+|- bgcolor="00FF00"
-| M3
+| M3 rc1
-|
+| 03/15/2017
-|
+| 03/23/2017
-|
+| DONE
-|- bgcolor="Gray"
+|- bgcolor="00FF00"
-| M4
+| M3 rc2
-|
+| 03/28/2017
-|
+| 04/05/2017
-|
+| DONE
-|- bgcolor="Gray"
+|- bgcolor="00FF00"
-|
+| M3 rc1 Meta-Intel
-|
+| 04/05/2017
-|
+| 04/12/2017
-|
+| DONE
-|- bgcolor="Gray"
+|- bgcolor="00FF00"
-|
+| M4 rc2
-|
+| 04/21/2017
-|
+| 04/29/2017
-|
+| DONE
+|- bgcolor="00FF00"
+| M4 rc6
+| 05/02/2017
+| 05/08/2017
+| DONE
+|- bgcolor="00FF00"
+| 2.3 Meta-Intel rc1
+| 05/11/2017
+| 05/16/2017
+| DONE
+|- bgcolor="00FF00"
+| 2.3 Meta-Intel rc4
+| 05/22/2017
+| 05/25/2017
+| DONE
+|- bgcolor="00FF00"
+| 2.3.1 rc2
+| 07/12/2017
+| 07/18/2017
+| DONE
+|- bgcolor="00FF00"
+| 2.3.2 rc1
+| 09/12/2017
+| 09/20/2017
+| DONE
+|- bgcolor="00FF00"
+| 2.3.3 rc1
+| 12/15/2017
+| 12/20/2017
+| DONE
+|- bgcolor="00FF00"
+| 2.3.4 rc1
+| 6/13/2018
+| 7/10/2018
+| DONE
 |}
@@ Line 354: / Line 389: @@
 ! REPORT
 |-
-| M1
+| M1 rc1
 |  [[WW52 - 2016-12-19 - Full Test Cycle 2.3 M1 rc1]]
 |-bgcoilor="green"
-| M2
+| M2 rc1
-|
+|  [[WW05 - 2017-01-24 - Full Test Cycle 2.3_M2 rc1]]
+|-bgcoilor="green"
+| M2 rc3
+|  [[WW06 - 2017-02-03 - Full Test Cycle 2.3_M2 rc3]]
+|-
+| M3 rc1
+| [[WW11 - 2017-03-15 - Full Test Cycle 2.3 M3 rc1]]
+|-
+| M3 rc2
+| [[WW14 - 2017-03-28 - Full Test Cycle 2.3 M3 rc2]]
+|-
+| M3 rc1 Meta-Intel
+| [[WW15 - 2017-04-05 - Full Test Cycle - 2.3 M3 - meta-intel]]
+|-
+| M4 rc2 (2.3 release - pyro)
+| [[WW17 - 2017-04-21 - Full Test Cycle 2.3 rc2]]
+|-
+| M4 rc4 (2.3 release - pyro)
+| [[WW19 - 2017-05-02 - Full Test Cycle 2.3 rc4]]
+|-
+| Meta-Intel  rc1 (2.3 release - pyro)
+| [[WW20 - 2017-05-11 - Full Test Cycle - 2.3 rc1 - meta-intel]]
+|-
+| Meta-Intel rc4 (2.3 release - pyro)
+| [[WW21 - 2017-05-22 - Full Test Cycle - 2.3 rc4 - meta-intel]]
 |-
-| M3
+| 2.3.1 rc2
-|
+| [[WW29 - 2017-07-18- Full Test Cycle - 2.3.1 rc2]]
 |-
-| M4
+| 2.3.2 rc1
-|
+| [[WW38_-_2017-09-20-_Full_Test_Cycle_-_2.3.2_rc1]]
 |-
-|
+| 2.3.3 rc1
-|
+| [[WW51_-_2017-12-20-_Full_Test_Cycle_-_2.3.3_rc1]]
 |-
-|
+| 2.3.4 rc1
-|
+| [[WW25_-_2018-06-22-_Full_Test_Cycle_-_2.3.4_rc1]]
 |}

Yocto Project 2.3 Release Test Plan: Difference between revisions