Latest revision as of 07:07, 14 December 2018

{{#vardefine:ADT_MAUAL | 6539}} {{#vardefine:ADT_AUTO | 6540}} {{#vardefine:ANYQEMU_SATO_MANUAL | 6563}} {{#vardefine:ANYQEMU_SATO_AUTO | 6564}} {{#vardefine:ANYBSP_SATO_AUTO | 6558}} {{#vardefine:ANYBSP_LSB_AUTO | 6561}} {{#vardefine:ANYBSP_WIC_AUTO | 6560}} {{#vardefine:MMAX_CORE2_MANUAL | 6554}} {{#vardefine:MMAX_I7_MANUAL | 6557}} {{#vardefine:NUC_I7_MANUAL | 6556}} {{#vardefine:NUC_GEX86-64_MANUAL | 6555}} {{#vardefine:MMAX_GEX86-32_MANUAL | 6553}} {{#vardefine:GAL_SATO_AUTO | 6565}} {{#vardefine:CROPS_AUTO | 6541}} {{#vardefine:CROPS_MANUAL | 6542}} {{#vardefine:ECLIPSE_MANUAL | 6543}} {{#vardefine:KERNEL_MANUAL | 6714}} {{#vardefine:METAYOCTO_MANUAL | 6544}} {{#vardefine:TOASTER_AUTO | 6545}} {{#vardefine:TOASTER_MANUAL | 6546}} {{#vardefine:ESDK_MANUAL | 6547}} {{#vardefine:ESDK_AUTO | 6548}} {{#vardefine:OE-CORE_AUTO | 6549}} {{#vardefine:BITBAKE_AUTO | 6550}} {{#vardefine:RUNTIME_AUTO | 6551}} {{#vardefine:BUILD-APPLIANCE_AUTO | 6552}}

Reversion history

Version	Modifier	Comments
1.0	Jose Perez Carranza	First draft

References

Documents that support this test plan

Introduction

This is the the overall test plan for version 2.5 of the Yocto Project, for all the generic information this document will be pointing to QA Master Test Plan

Objectives

This test plan for Yocto Project 2.5 release supports the following objectives:

To define the the tools to be used throughout the testing process.
To communicate to the responsible parties the items to be tested, set expectations around schedule, and define environmental needs.
To define how the tests will be conducted.
Release Criteria
Risks and contingencies

QA_Master_Test_Plan#Objectives_and_Tasks

Risk Analysis & Contingency Plan

This is the document for Risk Analysis https://docs.google.com/spreadsheets/d/1K-MZYo5sfabwI2UozGp6Uuh2sw5y309bXkyEZRPEEgk/edit#gid=0

Test Strategy

Is followed the general strategy defined on QA_Master_Test_Plan#Testing_Strategy and below is showed specific information applicable only for 2.5 Release

Bugs Tracking

Bugs that need to be implemented by QA Team
- 2.4 QA Assigned Bugs

Bugs that need to be verified by QA Team
- 2.5_QA_Bugs_To_Verify

Old Bugs that need to be Verified
- Old resolved bugs and features

Test Items

The Yocto Project QA team verifies each release against recent versions of the most popular Linux distributions that provide stable releases.

* centos 7 
* Debian 9
* Fedora 26
* Fedora 27
* OpenSuse42.3
* Tumbleweed (dropped)
* ubuntu 16.04 
* ubuntu 17.10

Test Cases

Yocto 2.5 Test Cases

Test Matrix

The components to be tested for version 2.5 of the Yocto Project against the different milestones are included in this section:

Code of Colors
Active Tracked on Testpoia
Active Non Tracked on Testopia
In Design Will be tracked on Tesopia
Inactive

ADT

Component	Environment	Auto	Manual	Owner
ADT	Fedora 25 i686	Y	Y	PG
ADT	Ubuntu 16.04 x86_64	Y	Y	PG

Eclipse Plugin

Component	Environment	Auto	Manual	Owner
Neon + Any Distro			Y	PG
Oxygen + Any Distro			Y	PG

Kernel

Component	Environment	Auto	Manual	Owner
Kernel	Any Distro		Y	PG

Meta-Yocto

Component	Environment	Auto	Manual	Owner
Meta-Yocto	Any Distro		Y	PG

Toaster

Component	Environment	Auto	Manual	Owner
Toaster	Any Distro	Y	Y	PG

eSDK

Component	Environment	Auto	Manual	Owner
eSDK	Any Distro + runtime	Y		PG
eSDK	Any Distro + selftest	Y		PG

Selftest

Component	Environment	Auto	Owner
OE-CORE	Ubuntu 16.10	Y	PG
	Ubuntu 16.04	Y	PG
	OpenSuse 42.2	Y	PG
	Opensuse 13.2	Y	PG
	Debian 8	Y	PG
	Centos 7	Y	PG
	Fedora 24	Y	PG
	Fedora 25	Y	PG

Bitbake

Component	Environment	Auto	Manual	Owner
Bitbake	Any Distro		Y	PG

Build - Apliance

Component	Environment	Auto	Manual	Owner
Build - Apliance	VM - Any Distro		Y	PG

Package Manager

Component	Environment	Auto	Manual	Owner
Package Manager	Any Distro		Y	PG

Performance

Component	Environment	Auto	Manual	Owner
Performance	Fedora 23	Y		GDC
Performance	Ubuntu 15	Y		GDC

BSP/QEMU

Component	Arch	HW	Environment	Image	Auto	Manual	pTest	Compliance
BSP	x86	MinwMax 32bit	genericx86	core-image-sato-sdk	Y	Y
		MinwMax 32bit	genericx86	core-image-lsb-sdk	Y
		MinwMax 64bit	genericx86-64	core-image-sato-sdk	Y
				core-image-lsb-sdk	Y
				core-image-tiny-initramfs	Y
		NUC 64	genericx86-64	core-image-sato-sdk	Y	Y	Y
		NUC 64		core-image-lsb-sdk	Y			Y
		QEMU	qemux86	core-image-sato-sdk	Y	Y
			qemux86	core-image-lsb-sdk	Y
			qemux86-64	core-image-sato-sdk	Y	Y
			qemux86-64	core-image-lsb-sdk	Y
	non x86	EdgeRouter	EdgeRouter	core-image-sato-sdk	Y	Y
		MPC8315e-rdb	MPC8315e-rdb	core-image-sato-sdk	Y	Y
		Beaglebone Black	Beaglebone	core-image-sato-sdk	Y	Y
		QEMU	qemuarm	core-image-sato-sdk	Y	Y
			qemuarm-64	core-image-sato-sdk	Y	Y
			qemuppc	core-image-sato-sdk	Y	Y
			qemumips	core-image-sato-sdk	Y	Y
			qemumips64	core-image-sato-sdk	Y	Y
	Meta-Intel	MinnowMax 32bit	core2-32	core-image-sato-sdk	Y	Y
		MinnowMax 32bit	core2-32	core-image-lsb-sdk	Y
		MinnowMax 64bit	corei7-64	core-image-sato-sdk	Y
		MinnowMax 64bit	corei7-64	core-image-lsb-sdk	Y
		NUC	corei7-64	core-image-sato-sdk	Y	Y
		NUC	corei7-64	core-image-lsb-sdk	Y
		CherryHill	corei7-64	core-image-sato-sdk	Y
		CherryHill	corei7-64	core-image-lsb-sdk	Y
		Joule	corei7-64	core-image-sato-sdk	Y
		meta-intel-quark 32bit	intel-quark	Core-image-sato-sdk	Y

All component to be tested on 2.5 are in #Test Items a detailed list of features per component can be found in Test Plan of the components QA_Master_Test_Plan#Component

Testopia Templates

COMPONENT	TEMPLATE
BitBake	TRTEMP_2.4_MANUAL_BitBake
BSP/QEMU	TRTEMP_2.4_MANUAL_BSP/QEMU_core-image-sato-sdk_Genericx86_MMAX32
	TRTEMP_2.4_MANUAL_BSP/QEMU_core-image-sato-sdk_core2-32_MMAX32
	TRTEMP_2.4_MANUAL_BSP/QEMU_core-image-sato-sdk_Genericx86-64_NUC
	TRTEMP_2.4_MANUAL_BSP/QEMU_core-image-sato-sdk_corei7-64_NUC
	TRTEMP_2.4_MANUAL_BSP/QEMU_core-image-sato-sdk_corei7-64_MMAX64
	TRTEMP_2.4_AUTO_BSP/QEMU_core-image-sato-sdk_ANYBSP
	TRTEMP_2.4_AUTO_BSP/QEMU_core-image-sato-sdk_wic_ANYBSP
	TRTEMP_2.4_AUTO_BSP/QEMU_core-image-lsb-sdk_ANYBSP
	TRTEMP_2.4_MANUAL_BSP/QEMU_core-image-sato-sdk_ANYQEMU
	TRTEMP_2.4_AUTO_BSP/QEMU_core-image-sato-sdk_ANYQEMU
	TRTEMP_2.4_AUTO_BSP/QEMU_core-image-sato-sdk_intel-quark
	TRTEMP_2.4_MANUAL_BSP/QEMU_core-image-sato-sdk_Beaglebone-Black
	TRTEMP_2.4_MANUAL_BSP/QEMU_core-image-sato-sdk_EdgeRouter
	TRTEMP_2.4_MANUAL_BSP/QEMU_core-image-sato-sdk_MPC8315e-rdb
	TRTEMP_2.4_AUTO_BSP/QEMU_core-image-lsb-sdk_ANYQEMU
Meta-Yocto	TRTEMP_2.4_MANUAL_Meta-yocto
OE-CORE	TRTEMP_2.4_AUTO_OE-Core
Build-Appliance	TRTEMP_2.4_MANUAL_Build-Appliance
Runtime	TRTEMP_2.4_AUTO_Runtime
Toaster	TRTEMP_2.4_AUTO_Toaster_UI
Toaster	TRTEMP_2.4_MANUAL_Toaster_UI-managed-mode
ADT	TRTEMP_2.4_MANUAL_ADT
ADT	TRTEMP_2.4_AUTO_ADT
CROPS	TRTEMP_2.4_AUTO_CROPS_Poky
CROPS	TRTEMP_2.4_MANUAL_CROPS_eSDK
Eclipse	TRTEMP_2.4_MANUAL_Eclipse
eSDK	TRTEMP_2.4_AUTO_eSDK_sdkext
eSDK	TRTEMP_2.4_AUTO_eSDK_oeselftest
Kernel	TRTEMP_2.4_MANUAL_Kernel
Package Management	TRTEMP_2.4_MANUAL_Package-Management-Updates

Features pending to be tested

Here is a list of areas that are not yet covered and need some work, implement test cases and add it to a component that belongs or create a new test plan for new component.

ADD FEATURES !!!!!!!!!!

The complete List of QA Implementations for YP 2.4 can be found at Features Implementation by QA Team

Features Not to be tested

Approach

Tests will be conducted per the documented test cases stored in Testopia. The test manager will create test runs for each tester. The tester will execute the tests in Testopia and mark each case as Pass / Fail / Skip /Blocked. The tester should leave notes on actual results and any other relevant details when possible.

Once complete, the test manager should review the test run reports in Testopia and report back to the team by mail and in the wiki as a test report.

Pass / Fail Criteria

This criteria is defined in Yocto_Project_v2.4_Status#Yocto_Project_v2.4_Release_Criteria

Exit Criteria

As defined in QA_Master_Test_Plan#Entry and Exit Criteria

Test Deliverables

Live Status 2.5_QA_Status
Final execution Report #Execution History
Mail sent to the mailing list

Test Environment

There are different environments and tools for every type of testing refer to QA_Master_Test_Plan#Test_Areas for more detailed information.

Schedule

M1 (May 8, 2017 to Jun. 23, 2017

M1 Feature/Tasks Development / Cutoff is Jun. 12, 2017

ID	Requester	Assignee	E	Summary (4 tasks) ⇒	Severity	P	Status
11244	Jair Gonzalez	Jair Gonzalez	2	QA: Validate Reference kit has BT audio support on gateway profile	enhancement	Medium	RESOLVED
11246	Jair Gonzalez	Jair Gonzalez	5	QA: Validate Reference kit supports image installer	enhancement	Medium	RESOLVED
10817	Paul Eggleton	Jose Perez C	4	oe-selftest: add a test for devtool modify on the kernel	enhancement	Medium+	RESOLVED
11550	Jose Perez C	Juan Ramos		poky-tiny: Add tests for poky-tiny images	enhancement	Medium	VERIFIED
			11

M1 Bugs

ID	Requester	Assignee	E	Summary (4 tasks) ⇒	Severity	P	Status
11436	Mariano Lopez	Humberto Ibarra	3	selftest: runtime-test install test is testing two different features	normal	Medium+	RESOLVED
10848	Jose Perez C	Jose Perez C	2	Add support for testing wic images on qemux86*	normal	Medium+	RESOLVED
11300	Jose Perez C	Jose Perez C	2	selftest: do_populate_sdk_ext task failing on eSDK test on GDC ABs	normal	Medium+	VERIFIED
10947	Libertad	Libertad	3	Reintegrate fully functional and corrected toaster automation testcases from toaster_automation_tests.py	normal	Medium	RESOLVED
			10

The planed schedule for 2.4 can be found at Yocto_2.4_Schedule

Live Schedule & Execution History

MILESTONE	STARTED	FINISHED	STATUS	REPORT
M1 rc1	12/27/2017	1/8/2018	DONE	https://wiki.yoctoproject.org/wiki/WW02_-_2018-1-8-_Full_Test_Cycle_-_2.5_M1_rc1
M1 rc3	1/8/2018	1/18/2018	DONE	https://wiki.yoctoproject.org/wiki/WW03_-_2018-1-18-_Full_Test_Cycle_-_2.5_M1_rc3
M1 rc1 meta-intel	2/1/2018	2/7/2018	DONE	https://wiki.yoctoproject.org/wiki/WW06_-_2018-02-07-_Full_Test_Cycle_-_2.5_M1_rc1_-_meta-intel
M2 rc1	2/7/2018	2/14/2018	DONE	https://wiki.yoctoproject.org/wiki/WW07_-_2018-02-14-_Full_Test_Cycle_-_2.5_M2_rc1
M3 rc1	3/22/2018	4/9/2018	DONE	https://wiki.yoctoproject.org/wiki/WW15_-_2018-04-09-_Full_Test_Cycle_-_2.5_M3_rc1
M4 rc1	4/25/2018	5/10/2018	DONE	https://wiki.yoctoproject.org/wiki/WW19_-_2018-05-10-_Full_Test_Cycle_-_2.5_M4_rc1
M4 rc1 meta-intel	5/18/2018	5/31/2018	DONE	https://wiki.yoctoproject.org/wiki/WW22_-_2018-05-31-_Full_Test_Cycle_-_2.5_M4_rc1_-_meta-intel
2.5.1.rc1	8/7/2018	8/16/2018	DONE	https://wiki.yoctoproject.org/wiki/WW33_-_2018-08-16-_Full_Test_Cycle_-_2.5.1_rc1
2.5.1.rc1 meta-intel	9/25/2018	9/28/2018	DONE	https://wiki.yoctoproject.org/wiki/WW39_-_2018-09-28-_Full_Test_Cycle_-_2.5.1_rc1_-_meta-intel
2.5.2.rc1	12/06/2018	12/14/2018	DONE	https://wiki.yoctoproject.org/wiki/WW49_-_2018-12-06-_Full_Test_Cycle_-_2.5.2_rc1

@@ Line 91: / Line 91: @@
   * Fedora 27
   * OpenSuse42.3
-  * Tumbleweed
+  * Tumbleweed (dropped)
   * ubuntu 16.04
   * ubuntu 17.10
@@ Line 152: / Line 152: @@
 | Neon + Any Distro
 |
+|
 | Y
 | PG
 |-
 | Oxygen + Any Distro
+|
 |
 | Y
@@ Line 724: / Line 726: @@
 The planed schedule for 2.4 can be found at [[Yocto_2.4_Schedule]]
-== Live Schedule ==
+== Live Schedule & Execution History ==
 {| class="wikitable"
@@ Line 731: / Line 733: @@
 ! FINISHED
 ! STATUS
-|- bgcolor="FFA500"
+! REPORT
+|- bgcolor="93DB70"
 | M1 rc1
 | 12/27/2017
-|
+| 1/8/2018
-| STARTED
+| DONE
-|- bgcolor="grey"
+| https://wiki.yoctoproject.org/wiki/WW02_-_2018-1-8-_Full_Test_Cycle_-_2.5_M1_rc1
-|
+|- bgcolor="93DB70"
-|
+| M1 rc3
-|
+| 1/8/2018
-|
+| 1/18/2018
+| DONE
+| https://wiki.yoctoproject.org/wiki/WW03_-_2018-1-18-_Full_Test_Cycle_-_2.5_M1_rc3
+|- bgcolor="93DB70"
+| M1 rc1 meta-intel
+| 2/1/2018
+| 2/7/2018
+| DONE
+| https://wiki.yoctoproject.org/wiki/WW06_-_2018-02-07-_Full_Test_Cycle_-_2.5_M1_rc1_-_meta-intel
+|- bgcolor="93DB70"
+| M2 rc1
+| 2/7/2018
+| 2/14/2018
+| DONE
+| https://wiki.yoctoproject.org/wiki/WW07_-_2018-02-14-_Full_Test_Cycle_-_2.5_M2_rc1
+|- bgcolor="93DB70"
+| M3 rc1
+| 3/22/2018
+| 4/9/2018
+| DONE
+| https://wiki.yoctoproject.org/wiki/WW15_-_2018-04-09-_Full_Test_Cycle_-_2.5_M3_rc1
+|- bgcolor="93DB70"
+| M4 rc1
+| 4/25/2018
+| 5/10/2018
+| DONE
+| https://wiki.yoctoproject.org/wiki/WW19_-_2018-05-10-_Full_Test_Cycle_-_2.5_M4_rc1
+|- bgcolor="93DB70"
+| M4 rc1 meta-intel
+| 5/18/2018
+| 5/31/2018
+| DONE
+| https://wiki.yoctoproject.org/wiki/WW22_-_2018-05-31-_Full_Test_Cycle_-_2.5_M4_rc1_-_meta-intel
+|- bgcolor="93DB70"
+| 2.5.1.rc1
+| 8/7/2018
+| 8/16/2018
+| DONE
+| https://wiki.yoctoproject.org/wiki/WW33_-_2018-08-16-_Full_Test_Cycle_-_2.5.1_rc1
+|- bgcolor="93DB70"
+| 2.5.1.rc1 meta-intel
+| 9/25/2018
+| 9/28/2018
+| DONE
+| https://wiki.yoctoproject.org/wiki/WW39_-_2018-09-28-_Full_Test_Cycle_-_2.5.1_rc1_-_meta-intel
+|- bgcolor="93DB70"
+| 2.5.2.rc1
+| 12/06/2018
+| 12/14/2018
+| DONE
+| https://wiki.yoctoproject.org/wiki/WW49_-_2018-12-06-_Full_Test_Cycle_-_2.5.2_rc1
 |- bgcolor="grey"
-|
-|
-|
-|
-|}
-= Execution History =
-{| class="wikitable"
-! MILESTONE
-! REPORT
-|-
-| M1
-|
-|-
-|
-|
-|-
-|
-|
 |}

Yocto Project 2.5 Release Test Plan: Difference between revisions