Latest revision as of 06:23, 8 April 2019

{{#vardefine:ADT_MAUAL | 6539}} {{#vardefine:ADT_AUTO | 6540}} {{#vardefine:ANYQEMU_SATO_MANUAL | 6563}} {{#vardefine:ANYQEMU_SATO_AUTO | 6564}} {{#vardefine:ANYBSP_SATO_AUTO | 6558}} {{#vardefine:ANYBSP_LSB_AUTO | 6561}} {{#vardefine:ANYBSP_WIC_AUTO | 6560}} {{#vardefine:MMAX_CORE2_MANUAL | 6554}} {{#vardefine:MMAX_I7_MANUAL | 6557}} {{#vardefine:NUC_I7_MANUAL | 6556}} {{#vardefine:NUC_GEX86-64_MANUAL | 6555}} {{#vardefine:MMAX_GEX86-32_MANUAL | 6553}} {{#vardefine:GAL_SATO_AUTO | 6565}} {{#vardefine:CROPS_AUTO | 6541}} {{#vardefine:CROPS_MANUAL | 6542}} {{#vardefine:ECLIPSE_MANUAL | 6543}} {{#vardefine:KERNEL_MANUAL | 6714}} {{#vardefine:METAYOCTO_MANUAL | 6544}} {{#vardefine:TOASTER_AUTO | 6545}} {{#vardefine:TOASTER_MANUAL | 6546}} {{#vardefine:ESDK_MANUAL | 6547}} {{#vardefine:ESDK_AUTO | 6548}} {{#vardefine:OE-CORE_AUTO | 6549}} {{#vardefine:BITBAKE_AUTO | 6550}} {{#vardefine:RUNTIME_AUTO | 6551}} {{#vardefine:BUILD-APPLIANCE_AUTO | 6552}}

Reversion history

Version	Modifier	Comments
1.0	Yeoh Ee Peng	First draft

References

Documents that support this test plan

Introduction

This is the the overall test plan for version 2.6 of the Yocto Project, for all the generic information this document will be pointing to QA Master Test Plan

Objectives

This test plan for Yocto Project 2.6 release supports the following objectives:

To define the tools to be used throughout the testing process.
To communicate to the responsible parties the items to be tested, set expectations around schedule, and define environmental needs.
To define how the tests will be conducted.
Release Criteria
Risks and contingencies

QA_Master_Test_Plan#Objectives_and_Tasks

Risk Analysis & Contingency Plan

This is the document for Risk Analysis https://docs.google.com/spreadsheets/d/1K-MZYo5sfabwI2UozGp6Uuh2sw5y309bXkyEZRPEEgk/edit#gid=0

Test Strategy

Is followed the general strategy defined on QA_Master_Test_Plan#Testing_Strategy and below is showed specific information applicable only for 2.6 Release

Bugs Tracking

Bugs that need to be implemented by QA Team
- 2.6 QA Assigned Bugs

Bugs that need to be verified by QA Team
- 2.6_QA_Bugs_To_Verify

Old Bugs that need to be Verified
- Old resolved bugs and features

Test Items

The Yocto Project QA team verifies each release against recent versions of the most popular Linux distributions that provide stable releases.

* centos 7 
* Debian 9
* Fedora 26
* Fedora 27
* OpenSuse42.3
* ubuntu 16.04 
* ubuntu 17.10

Test Cases

Yocto 2.6 Test Cases

Test Matrix

The components to be tested for version 2.6 of the Yocto Project against the different milestones are included in this section:

Code of Colors
Active Tracked on Testpoia
Active Non Tracked on Testopia
In Design Will be tracked on Tesopia
Inactive

ADT

Component	Environment	Auto	Manual	Owner
ADT	Fedora 25 i686	Y	Y	PG
ADT	Ubuntu 16.04 x86_64	Y	Y	PG

Eclipse Plugin

Component	Environment	Auto	Manual	Owner
Neon + Any Distro			Y	PG
Oxygen + Any Distro			Y	PG

Kernel

Component	Environment	Auto	Manual	Owner
Kernel	Any Distro		Y	PG

Meta-Yocto

Component	Environment	Auto	Manual	Owner
Meta-Yocto	Any Distro		Y	PG

Toaster

Component	Environment	Auto	Manual	Owner
Toaster	Any Distro		Y	PG

eSDK

Component	Environment	Auto	Manual	Owner
eSDK	Any Distro + runtime	Y		PG
eSDK	Any Distro + selftest	Y		PG

Selftest

Component	Environment	Auto	Owner
OE-CORE	Ubuntu 16.04	Y	PG
	Ubuntu 17.10	Y	PG
	OpenSuse 42.3	Y	PG
	Debian 9	Y	PG
	Centos 7	Y	PG
	Fedora 26	Y	PG
	Fedora 27	Y	PG

Bitbake

Component	Environment	Auto	Manual	Owner
Bitbake	Any Distro		Y	PG

Build - Apliance

Component	Environment	Auto	Manual	Owner
Build - Apliance	VM - Any Distro		Y	PG

Package Manager

Component	Environment	Auto	Manual	Owner
Package Manager	Any Distro	Y		PG

Performance

Component	Environment	Auto	Manual	Owner
Performance	Centos 7	Y		PG
Performance	Ubuntu 16.04	Y		PG

BSP/QEMU

Component	Arch	HW	Environment	Image	Auto	Manual	pTest	Compliance
BSP	x86	MinwTurbot 32bit	genericx86	core-image-sato-sdk	Y
		MinwTurbot 32bit	genericx86	core-image-lsb-sdk	Y
		MinwTurbot 64bit	genericx86-64	core-image-sato-sdk	Y	Y
		MinwTurbot 64bit	genericx86-64	core-image-lsb-sdk	Y
		NUC 6	genericx86-64	core-image-sato-sdk	Y		Y
		NUC 6	genericx86-64	core-image-lsb-sdk	Y			Y
		NUC 7	genericx86-64	core-image-sato-sdk	Y	Y
		NUC 7	genericx86-64	core-image-lsb-sdk	Y
		QEMU	qemux86	core-image-sato-sdk	Y	Y
			qemux86	core-image-lsb-sdk	Y
			qemux86-64	core-image-sato-sdk	Y	Y
			qemux86-64	core-image-lsb-sdk	Y
	non x86	EdgeRouter	EdgeRouter	core-image-sato-sdk	Y	Y
		MPC8315e-rdb	MPC8315e-rdb	core-image-sato-sdk	Y	Y
		Beaglebone Black	Beaglebone	core-image-sato-sdk	Y	Y
		QEMU	qemuarm	core-image-sato-sdk	Y	Y
			qemuarm-64	core-image-sato-sdk	Y	Y
			qemuppc	core-image-sato-sdk	Y	Y
			qemumips	core-image-sato-sdk	Y	Y
			qemumips64	core-image-sato-sdk	Y	Y
	Meta-Intel	MinwTurbot 32bit	core2-32	core-image-sato-sdk	Y
		MinwTurbot 32bit	core2-32	core-image-lsb-sdk	Y
		MinwTurbot 64bit	corei7-64	core-image-sato-sdk	Y	Y
		MinwTurbot 64bit	corei7-64	core-image-lsb-sdk	Y
		NUC 6	corei7-64	core-image-sato-sdk	Y
		NUC 6	corei7-64	core-image-lsb-sdk	Y
		NUC 7	corei7-64	core-image-sato-sdk	Y	Y
				core-image-lsb-sdk	Y
				poky-tiny		Y

All component to be tested on 2.6 are in #Test Items a detailed list of features per component can be found in Test Plan of the components QA_Master_Test_Plan#Component

Testopia Templates

COMPONENT	TEMPLATE
BitBake	TRTEMP_2.6_MANUAL_BitBake
BSP/QEMU	TRTEMP_2.6_MANUAL_BSP/QEMU_core-image-sato-sdk_Genericx86_MMAX32
	TRTEMP_2.6_MANUAL_BSP/QEMU_core-image-sato-sdk_core2-32_MMAX32
	TRTEMP_2.6_MANUAL_BSP/QEMU_core-image-sato-sdk_Genericx86-64_NUC
	TRTEMP_2.6_MANUAL_BSP/QEMU_core-image-sato-sdk_corei7-64_NUC
	TRTEMP_2.6_MANUAL_BSP/QEMU_core-image-sato-sdk_corei7-64_MMAX64
	TRTEMP_2.6_AUTO_BSP/QEMU_core-image-sato-sdk_ANYBSP
	TRTEMP_2.6_AUTO_BSP/QEMU_core-image-sato-sdk_wic_ANYBSP
	TRTEMP_2.6_AUTO_BSP/QEMU_core-image-lsb-sdk_ANYBSP
	TRTEMP_2.6_MANUAL_BSP/QEMU_core-image-sato-sdk_ANYQEMU
	TRTEMP_2.6_AUTO_BSP/QEMU_core-image-sato-sdk_ANYQEMU
	TRTEMP_2.6_AUTO_BSP/QEMU_core-image-sato-sdk_intel-quark
	TRTEMP_2.6_MANUAL_BSP/QEMU_core-image-sato-sdk_Beaglebone-Black
	TRTEMP_2.6_MANUAL_BSP/QEMU_core-image-sato-sdk_EdgeRouter
	TRTEMP_2.6_MANUAL_BSP/QEMU_core-image-sato-sdk_MPC8315e-rdb
	TRTEMP_2.6_AUTO_BSP/QEMU_core-image-lsb-sdk_ANYQEMU
Meta-Yocto	TRTEMP_2.6_MANUAL_Meta-yocto
OE-CORE	TRTEMP_2.6_AUTO_OE-Core
Build-Appliance	TRTEMP_2.6_MANUAL_Build-Appliance
Runtime	TRTEMP_2.6_AUTO_Runtime
Toaster	TRTEMP_2.6_AUTO_Toaster_UI
Toaster	TRTEMP_2.6_MANUAL_Toaster_UI-managed-mode
ADT	TRTEMP_2.6_MANUAL_ADT
ADT	TRTEMP_2.6_AUTO_ADT
CROPS	TRTEMP_2.6_AUTO_CROPS_Poky
CROPS	TRTEMP_2.6_MANUAL_CROPS_eSDK
Eclipse	TRTEMP_2.6_MANUAL_Eclipse
eSDK	TRTEMP_2.6_AUTO_eSDK_sdkext
eSDK	TRTEMP_2.6_AUTO_eSDK_oeselftest
Kernel	TRTEMP_2.6_MANUAL_Kernel
Package Management	TRTEMP_2.6_MANUAL_Package-Management-Updates

Features pending to be tested

Here is a list of areas that are not yet covered and need some work, implement test cases and add it to a component that belongs or create a new test plan for new component.

ADD FEATURES !!!!!!!!!!

The complete List of QA Implementations for YP 2.6 can be found at Features Implementation by QA Team

Features Not to be tested

Approach

Tests will be conducted per the documented test cases stored in Testopia. The test manager will create test runs for each tester. The tester will execute the tests in Testopia and mark each case as Pass / Fail / Skip /Blocked. The tester should leave notes on actual results and any other relevant details when possible.

Once complete, the test manager should review the test run reports in Testopia and report back to the team by mail and in the wiki as a test report.

Pass / Fail Criteria

This criteria is defined in Yocto_Project_v2.6_Status#Yocto_Project_v2.6_Release_Criteria

Exit Criteria

As defined in QA_Master_Test_Plan#Entry and Exit Criteria

Test Deliverables

Live Status 2.6_QA_Status
Final execution Report #Execution History
Mail sent to the mailing list

Test Environment

There are different environments and tools for every type of testing refer to QA_Master_Test_Plan#Test_Areas for more detailed information.

Schedule

M1 (Apr 30, 2018 to Jun 22, 2018)

M1 Feature/Tasks Development / Cutoff is Jun. 11, 2017

ID	Requester	Assignee	E	Summary (4 tasks) ⇒	Severity	P	Status
10681	Richard Purdie	Alejandro Hernandez	10	Support inter-multi-configuration dependencies	enhancement	Medium+	RESOLVED
12727	Ross Burton	Alexander Kanavin		Drop wireless-tools and use iw	enhancement	Medium+	RESOLVED
12689	Anuj Mittal	Anuj Mittal	1	wic should be able to use initrd value for grub	enhancement	High	RESOLVED
10744	Jose Perez C	Richard Purdie	5	Update test case of smartpm/createrepo under meta/lib/oeqa/runtime	enhancement	Medium+	RESOLVED
			16

M1 Bugs

ID	Requester	Assignee	E	Summary (35 tasks) ⇒	Severity	P	Whiteboard	Status
12224	brian avery	Anuj Mittal	2	wic image created for qemux86-64 stalls on undefined video mode number: 318	normal	Medium		RESOLVED
12767	Robert Berger	Anuj Mittal		bitbake-layers create-layer	normal	Medium	Backport 2.5.1	RESOLVED
12773	pankaj yadav	Anuj Mittal		[Meta-Yocto :2.4.3RC2] QA_WARN and QA_ERROR Mechanism checking Failed	normal	Medium+		RESOLVED
12666	pravin	Anuj Mittal	1	[2.5 M3 RC1] BSP-QEMU:core-image-sato-sdk-genericx86-64.hddimg unable to connect to console with serial on Minnowboard Turbot	normal	Medium+		RESOLVED
12776	Leonardo Sandoval Gonzalez	Anuj Mittal	1	hddimg installer hangs on sumo	normal	Medium+		RESOLVED
12832	pravin	Anuj Mittal	2	[ 2.6 M1 rc1 ][BSP][Test case 267]: audio and video does not play in media player[Mturbot x86-64 and NUC7]	normal	Medium+		RESOLVED
12803	pankaj yadav	Apoorv		[Yocto-2.6_M1.RC1] Building a kernel module on target image fails	normal	Medium+		RESOLVED
12805	pankaj yadav	Apoorv		[Test Case 1058] bitbake can use alternate download methods than git	normal	Medium+		RESOLVED
12742	Ross Burton	Paul Eggleton		buildhistory doesn't like files with whitespace	normal	Medium+		RESOLVED
12757	He Zhe	Bruce Ashfield		kernel do_configure fails to use tools in recipe-sysroot-native	critical	Medium+		RESOLVED
12794	Robert Berger	Chin Huat Ang		Eclipse Project SDK plugin Oxygen/2.5.0 can not be installed	normal	Medium+		RESOLVED
12796	Scott Rifenbark	Chin Huat Ang		Eclipse Oxygen section needs work-around - it is broke	normal	Undecided		RESOLVED
12741	Ross Burton	Hongxu Jia		distcc: upgrade to 3.3	normal	Medium+		RESOLVED
12808	Robert Berger	Joshua Watt		bitbake-layers create-layer	normal	Medium+		RESOLVED
12836	Manish Das	Manish Das		[2.6 M1 RC1] busybox ptest failed	normal	Medium+		RESOLVED
12656	Juro Bystricky	Maxin B. John		at-spi2-core_2.28.0.bb:do_compile error	normal	Medium+		RESOLVED
12708	Juro Bystricky	Mingli Yu		(reproducibility) boost-dbg contains build host references	normal	Medium+	reproducibility	RESOLVED
12711	Aaron Chan	Mohamad Noor Alim		[2.5 M3 rc1][nightly-arm64][fedora 27][autobuilder2] - Event handler runqueue_stats failed	major	Medium+		RESOLVED
12736	Mohamad Noor Alim	Mohamad Noor Alim		2.5 rc1: CentOS 7 oe-selftest fail at runtime_test.TestImage.test_testimage_dnf	normal	Medium+		RESOLVED
12792	Olof Johansson	Olof Johansson		RPM leaks pre-/postinstall scripts to /var/tmp	normal	Medium		RESOLVED
12768	Robert Berger	Robert Berger		Please place them into pkg_postinst_ontarget_${PN} ()	normal	Medium		RESOLVED
12729	Chen Qi	Chen Qi	1	testsdkext takes too long time to finish	normal	High		RESOLVED
12835	Chen Qi	Chen Qi		'core-image-weston + systemd' fails the 'parselogs' runtime oeqa test case	normal	Undecided		RESOLVED
12866	Mohamad Noor Alim	Richard Purdie		[2.6 M2 rc1] wic test_fix_size failed	normal	Medium		RESOLVED
12739	Matt Hoosier	Richard Purdie		Fetcher for Git submodules ('gitsm') bypasses BB_NO_NETWORK checks	normal	Medium+	Backport to YP 2.4 and 2.5	RESOLVED
12765	Hains	Richard Purdie		Race condition do_package_write_ipk fails due to empty package-split dir.	normal	Medium+		RESOLVED
12828	Maxin B. John	Richard Purdie		mesa build fails on edgerouter and beaglebone-yocto	normal	Medium+		RESOLVED
12756	Armin Kuster	Ross Burton		[ross/MUT] nightly-oecore failed	normal	Medium+		RESOLVED
12799	Armin Kuster	Ross Burton		Occasional mesa build race	normal	Medium+		RESOLVED
12815	Maxin B. John	Ross Burton		libsdl2 build failed	normal	Medium+		RESOLVED
12475	Amber Elliot	Stephen K Jolley	1	Bug tracking BKMS should be documented	normal	Medium+	10 May 2018: IN PROGRESS REVIEW	RESOLVED
12517	Stephano Cetola	Scott Rifenbark	1	Update documentation on gen-tapdevs script	normal	Medium+	15 May 2018: RESOLVED	RESOLVED
11567	Phlip Balister	Stephano Cetola	1	Trying to create an new layer with the yocto-layer script fails.	normal	Medium+	May 15 2018: RESOLVED	RESOLVED
12772	Scott Branden	Tim Orling		bitbake after devtool doesn't work properly with qemu	normal	Medium+		RESOLVED
12778	Uttam K	Uttam K		Unable to build yocto poky project	major	Medium+		RESOLVED
			10

The planed schedule for 2.6 can be found at Yocto_2.6_Schedule

Live Schedule & Execution History

MILESTONE	STARTED	FINISHED	STATUS	REPORT
M1 rc1	6/19/2018	7/2/2018	DONE	https://wiki.yoctoproject.org/wiki/WW27_-_2018-07-02_-_Full_Test_Cycle_2.6_M1_rc1
M1 rc1 meta-intel 10.0	7/13/2018	7/24/2018	DONE	https://wiki.yoctoproject.org/wiki/WW29_-_2018-07-19-_Full_Test_Cycle_-_2.6_M1_rc1_-_meta-intel
M2 rc1	7/19/2018	8/7/2018	DONE	https://wiki.yoctoproject.org/wiki/WW32_-_2018-08-07_-_Full_Test_Cycle_2.6_M2_rc1
M2 rc1 meta-intel 10.0	8/27/2018	9/4/2018	DONE	https://wiki.yoctoproject.org/wiki/WW36_-_2018-09-04-_Full_Test_Cycle_-_2.6_M2_rc1_-_meta-intel_10.0
M3 rc1	9/25/2018	10/09/2018	DONE	https://wiki.yoctoproject.org/wiki/WW41_-_2018-10-03_-_Full_Test_Cycle_2.6_M3_RC1
M4 rc1	10/30/2018	11/09/2018	DONE	https://wiki.yoctoproject.org/wiki/WW44_-_2018-10-30_-_Full_Test_Cycle_2.6_M4_RC1
M4 rc2 meta-intel 10.0	12/14/2018	12/21/2018	DONE	https://wiki.yoctoproject.org/wiki/WW51_-_2018-12-21-_Full_Test_Cycle_-_2.6_rc2_-_meta-intel_10.0
2.6.1 rc1	01/09/2019	01/22/2019	DONE	https://wiki.yoctoproject.org/wiki/WW02_-_2019-01-09_-_Full_Test_Cycle_2.6.1_RC1
2.6.1 rc1 meta-intel 10.1	02/12/2019	02/18/2019	DONE	https://wiki.yoctoproject.org/wiki/WW07_-_2019-02-12-_Full_Test_Cycle_-_2.6.1_rc1_-_meta-intel_10.1

@@ Line 735: / Line 735: @@
 | DONE
 | https://wiki.yoctoproject.org/wiki/WW29_-_2018-07-19-_Full_Test_Cycle_-_2.6_M1_rc1_-_meta-intel
-|- bgcolor="ffbf00"
+|- bgcolor="93DB70"
 | M2 rc1
 | 7/19/2018
-|
+| 8/7/2018
-| IN PROGRESS
+| DONE
 | https://wiki.yoctoproject.org/wiki/WW32_-_2018-08-07_-_Full_Test_Cycle_2.6_M2_rc1
-|- bgcolor="grey"
+|- bgcolor="93DB70"
+| M2 rc1 meta-intel 10.0
+| 8/27/2018
+| 9/4/2018
+| DONE
+| https://wiki.yoctoproject.org/wiki/WW36_-_2018-09-04-_Full_Test_Cycle_-_2.6_M2_rc1_-_meta-intel_10.0
+|- bgcolor="93DB70"
+| M3 rc1
+| 9/25/2018
+| 10/09/2018
+| DONE
+|https://wiki.yoctoproject.org/wiki/WW41_-_2018-10-03_-_Full_Test_Cycle_2.6_M3_RC1
+|- bgcolor="93DB70"
+| M4 rc1
+| 10/30/2018
+| 11/09/2018
+| DONE
+| https://wiki.yoctoproject.org/wiki/WW44_-_2018-10-30_-_Full_Test_Cycle_2.6_M4_RC1
+|- bgcolor="93DB70"
+| M4 rc2 meta-intel 10.0
+| 12/14/2018
+| 12/21/2018
+| DONE
+| https://wiki.yoctoproject.org/wiki/WW51_-_2018-12-21-_Full_Test_Cycle_-_2.6_rc2_-_meta-intel_10.0
+|- bgcolor="93DB70"
+| 2.6.1 rc1
+| 01/09/2019
+| 01/22/2019
+| DONE
+| https://wiki.yoctoproject.org/wiki/WW02_-_2019-01-09_-_Full_Test_Cycle_2.6.1_RC1
+|- bgcolor="93DB70"
+| 2.6.1 rc1 meta-intel 10.1
+| 02/12/2019
+| 02/18/2019
+| DONE
+| https://wiki.yoctoproject.org/wiki/WW07_-_2019-02-12-_Full_Test_Cycle_-_2.6.1_rc1_-_meta-intel_10.1
+|- bgcolor="grey"
 |}

Yocto Project 2.6 Release Test Plan: Difference between revisions