{{#vardefine:ADT_MAUAL | 6539}} {{#vardefine:ADT_AUTO | 6540}} {{#vardefine:ANYQEMU_SATO_MANUAL | 6563}} {{#vardefine:ANYQEMU_SATO_AUTO | 6564}} {{#vardefine:ANYBSP_SATO_AUTO | 6558}} {{#vardefine:ANYBSP_LSB_AUTO | 6561}} {{#vardefine:ANYBSP_WIC_AUTO | 6560}} {{#vardefine:MMAX_CORE2_MANUAL | 6554}} {{#vardefine:MMAX_I7_MANUAL | 6557}} {{#vardefine:NUC_I7_MANUAL | 6556}} {{#vardefine:NUC_GEX86-64_MANUAL | 6555}} {{#vardefine:MMAX_GEX86-32_MANUAL | 6553}} {{#vardefine:GAL_SATO_AUTO | 6565}} {{#vardefine:CROPS_AUTO | 6541}} {{#vardefine:CROPS_MANUAL | 6542}} {{#vardefine:ECLIPSE_MANUAL | 6543}} {{#vardefine:KERNEL_MANUAL | 6714}} {{#vardefine:METAYOCTO_MANUAL | 6544}} {{#vardefine:TOASTER_AUTO | 6545}} {{#vardefine:TOASTER_MANUAL | 6546}} {{#vardefine:ESDK_MANUAL | 6547}} {{#vardefine:ESDK_AUTO | 6548}} {{#vardefine:OE-CORE_AUTO | 6549}} {{#vardefine:BITBAKE_AUTO | 6550}} {{#vardefine:RUNTIME_AUTO | 6551}} {{#vardefine:BUILD-APPLIANCE_AUTO | 6552}}

Reversion history

Version	Modifier	Comments
1.0	Jain Sangeeta	First draft

References

Documents that support this test plan

Introduction

This is the the overall test plan for version 2.8 of the Yocto Project, for all the generic information this document will be pointing to QA Master Test Plan

Objectives

This test plan for Yocto Project 2.8 release supports the following objectives:

To define the tools to be used throughout the testing process.
To communicate to the responsible parties the items to be tested, set expectations around schedule, and define environmental needs.
To define how the tests will be conducted.
Release Criteria
Risks and contingencies

QA_Master_Test_Plan#Objectives_and_Tasks

Risk Analysis & Contingency Plan

This is the document for Risk Analysis https://docs.google.com/spreadsheets/d/1K-MZYo5sfabwI2UozGp6Uuh2sw5y309bXkyEZRPEEgk/edit#gid=0

Test Strategy

Is followed the general strategy defined on QA_Master_Test_Plan#Testing_Strategy and below is showed specific information applicable only for 2.8 Release

Bugs Tracking

Bugs that need to be implemented by QA Team
- 2.7 QA Assigned Bugs

Bugs that need to be verified by QA Team
- 2.7_QA_Bugs_To_Verify

Old Bugs that need to be Verified
- Old resolved bugs and features

Test Items

The Yocto Project QA team verifies each release against recent versions of the most popular Linux distributions that provide stable releases.

* centos 7 
* Debian 9
* Fedora 26
* Fedora 27
* OpenSuse42.3
* ubuntu 16.04 
* ubuntu 17.10

Test Cases

ADT

Component	Environment	Auto	Manual	Owner
ADT	Fedora 25 i686	Y		AB
ADT	Ubuntu 16.04 x86_64	Y		AB

Eclipse Plugin

Component	Environment	Auto	Manual	Owner
Neon + Any Distro			Y	PG
Oxygen + Any Distro			Y	PG

Kernel

Component	Environment	Auto	Manual	Owner
Kernel	Any Distro		Y	PG

Meta-Yocto

Component	Environment	Auto	Manual	Owner
Meta-Yocto	Any Distro		Y	PG

Toaster

Component	Environment	Auto	Manual	Owner
Toaster	Any Distro		Y	PG

eSDK

Component	Environment	Auto	Manual	Owner
eSDK	Any Distro + runtime	Y		PG
eSDK	Any Distro + selftest	Y		PG

Selftest

Component	Environment	Auto	Owner
OE-CORE	Ubuntu 16.04	Y	PG
	Ubuntu 17.10	Y	PG
	OpenSuse 42.3	Y	PG
	Debian 9	Y	PG
	Centos 7	Y	PG
	Fedora 26	Y	PG
	Fedora 27	Y	PG

Bitbake

Component	Environment	Auto	Manual	Owner
Bitbake	Any Distro		Y	PG

Build - Apliance

Component	Environment	Auto	Manual	Owner
Build - Apliance	VM - Any Distro		Y	PG

Package Manager

Component	Environment	Auto	Manual	Owner
Package Manager	Any Distro	Y		PG

Performance

Component	Environment	Auto	Manual	Owner
Performance	Centos 7	Y		PG
Performance	Ubuntu 16.04	Y		PG

BSP/QEMU

Component	Arch	HW	Environment	Image	Auto	Manual	pTest	Compliance
BSP	x86	MinwTurbot 32bit	genericx86	core-image-sato-sdk	Y
		MinwTurbot 32bit	genericx86	core-image-lsb-sdk	Y
		MinwTurbot 64bit	genericx86-64	core-image-sato-sdk	Y	Y
		MinwTurbot 64bit	genericx86-64	core-image-lsb-sdk	Y
		NUC 6	genericx86-64	core-image-sato-sdk	Y		Y
		NUC 6	genericx86-64	core-image-lsb-sdk	Y			Y
		NUC 7	genericx86-64	core-image-sato-sdk	Y	Y
		NUC 7	genericx86-64	core-image-lsb-sdk	Y
		QEMU	qemux86	core-image-sato-sdk	Y	Y
			qemux86	core-image-lsb-sdk	Y
			qemux86-64	core-image-sato-sdk	Y	Y
			qemux86-64	core-image-lsb-sdk	Y
	non x86	EdgeRouter	EdgeRouter	core-image-sato-sdk	Y	Y
		MPC8315e-rdb	MPC8315e-rdb	core-image-sato-sdk	Y	Y
		Beaglebone Black	Beaglebone	core-image-sato-sdk	Y	Y
		QEMU	qemuarm	core-image-sato-sdk	Y	Y
			qemuarm-64	core-image-sato-sdk	Y	Y
			qemuppc	core-image-sato-sdk	Y	Y
			qemumips	core-image-sato-sdk	Y	Y
			qemumips64	core-image-sato-sdk	Y	Y
	Meta-Intel	MinwTurbot 32bit	core2-32	core-image-sato-sdk	Y
		MinwTurbot 32bit	core2-32	core-image-lsb-sdk	Y
		MinwTurbot 64bit	corei7-64	core-image-sato-sdk	Y	Y
		MinwTurbot 64bit	corei7-64	core-image-lsb-sdk	Y
		NUC 6	corei7-64	core-image-sato-sdk	Y
		NUC 6	corei7-64	core-image-lsb-sdk	Y
		NUC 7	corei7-64	core-image-sato-sdk	Y	Y
				core-image-lsb-sdk	Y
				poky-tiny		Y

All component to be tested on 2.7 are in #Test Items a detailed list of features per component can be found in Test Plan of the components QA_Master_Test_Plan#Component

Features pending to be tested

Here is a list of areas that are not yet covered and need some work, implement test cases and add it to a component that belongs or create a new test plan for new component.

ADD FEATURES !!!!!!!!!!

The complete List of QA Implementations for YP 2.7 can be found at Features Implementation by QA Team

Features Not to be tested

Approach

Tests will be conducted per the documented test cases stored in Testopia. The test manager will create test runs for each tester. The tester will execute the tests in Testopia and mark each case as Pass / Fail / Skip /Blocked. The tester should leave notes on actual results and any other relevant details when possible.

Once complete, the test manager should review the test run reports in Testopia and report back to the team by mail and in the wiki as a test report.

Pass / Fail Criteria

This criteria is defined in Yocto_Project_v2.7_Status#Yocto_Project_v2.7_Release_Criteria

Exit Criteria

As defined in QA_Master_Test_Plan#Entry and Exit Criteria

Test Deliverables

Live Status 2.7_QA_Status
Final execution Report #Execution History
Mail sent to the mailing list

Test Environment

There are different environments and tools for every type of testing refer to QA_Master_Test_Plan#Test_Areas for more detailed information.

Schedule

M1 (Apr 30, 2018 to Jun 22, 2018)

M1 Feature/Tasks Development / Cutoff is Jun. 11, 2017

ID	Requester	Assignee	E	Summary (12 tasks) ⇒	Severity	P	Status
13058	Chen Qi	André Draszik	2	split util-linux automatically instead of hard coding	enhancement	Medium+	RESOLVED
13094	David Reyna	David Reyna	3	support custom extensions to reports	enhancement	Medium	RESOLVED
13110	David Reyna	David Reyna	3	Add "release" records to product records	enhancement	Medium	RESOLVED
11583	Aníbal Limón	yeoh ee peng	4	oeqa: core add support to publish results	enhancement	Medium+	RESOLVED
12774	Lei Yang	Hongzhi		oeqa: kernel config depenency for some kernel test cases	enhancement	Medium+	RESOLVED
12931	Joshua Watt	Joshua Watt	5	Change default PACKAGE_DEBUG_SPLIT_STYLE to "debug-with-srcpkg"	enhancement	Medium+	RESOLVED
13030	Joshua Watt	Joshua Watt		Add sstate hash equivalence support	enhancement	Medium+	RESOLVED
13088	Richard Purdie	Aníbal Limón	0.3	Add support for extracting individual ptest timing	enhancement	Medium+	RESOLVED
12332	Paul Eggleton	Michael Halstead	2	Regularly purge the error report database	enhancement	Medium+	RESOLVED
12421	Juro Bystricky	Mingli Yu		Host-compiled Python bytecode not reproducible	enhancement	Medium+	RESOLVED
13011	Armin Kuster	Richard Purdie		Start collecting ptest for arm and x86 on Vtargets.	enhancement	Medium+	RESOLVED
13114		Ross Burton		Split python3-ctypes tests package	enhancement	Medium+	RESOLVED
			19.3

M1 Bugs

ID	Requester	Assignee	E	Summary (49 tasks) ⇒	Severity	P	Whiteboard	Status
13295	Joshua Watt	Armin Kuster		openssh ptest results are inconsistent per image	normal	Medium+		RESOLVED
13391	Richard Purdie	Alexander Kanavin		acl ptest timeout due to perl update	normal	High		RESOLVED
13363	Sangeeta Jain	Anuj Mittal		LogrotateTest (lib/oeqa/runtime/cases/logrotate.py) failed.	minor	Medium+	Backport to 2.7	RESOLVED
13396	Sudhir Sharma	Anuj Mittal		[QA 2.8 M1 RC1][BSP HW] parselogs.ParseLogsTest.test_parselogs failure on coffeelake	normal	Medium+		RESOLVED
13441	Ross Burton	Anuj Mittal		Enable CONFIG_RANDOM_TRUST_CPU	normal	Medium+		RESOLVED
13248	Richard Purdie	Apoorv		New QA process issues	normal	High		RESOLVED
12923	Gary	Paul Eggleton		recipetool create error on empty setup.py	normal	Medium		RESOLVED
13083	Ross Burton	Paul Eggleton		recipetool create doesn't handle Python 3	normal	Medium+		RESOLVED
13168	Romuald Jeanne	Paul Eggleton		devtool: cannot add new recipe if reproducible_build class is inherited	normal	Medium+	Backport 2.5 and 2.6	RESOLVED
13404	Alexander Kanavin	Paul Eggleton		'devtool upgrade' tracebacks when a recipe includes another recipe with PV in the name	normal	Medium+		RESOLVED
13422	Ross Burton	Paul Eggleton		recipetool.RecipetoolTests.test_recipetool_load_plugin fails	normal	Medium+		RESOLVED
13284	Armin Kuster	Changqing Li		Patchtest seems to be off-line and not working	normal	High		RESOLVED
13416		Changqing Li		gcc 9.1 runtime do_install error	major	Medium+		RESOLVED
13356	Emil Melar	David Reyna	1	oe-core.xml in thud branch refers to wrong bitbake version	normal	Medium+	also backport to Thud (2.6) and Warrior (2.7)	RESOLVED
13382	Jaewon Lee	Jaewon Lee		devtool build should also deploy artifact	normal	Medium		RESOLVED
13377	Jaewon Lee	Jaewon Lee		gst-inspect on target is looking for a .so on host	normal	Medium+		RESOLVED
13379	Armin Kuster	Jon Mason		qemuarm64 runtime syslog test failure in testimage	normal	High		RESOLVED
12962	Ross Burton	Joshua Watt		[meta-mingw] update-alternative warnings	normal	Medium		RESOLVED
13301	Joshua Watt	Mariano Lopez	2	util-linux ptest results are inconsistent per image	normal	Medium+		RESOLVED
13309	Armin Kuster	Kevin Hao		[master] qemuppc fails to shutdown intermittently	normal	High	with backport	RESOLVED
13418	Ross Burton	Robert Yang		Multilib and update-alternatives conflict	normal	Medium+		RESOLVED
13133	Florian Kauer	Mark Hatle		Git submodule fetcher fails if last component of path is different than the URL	major	Medium+		RESOLVED
13340	Matthias Schoepfer	Matthias Schoepfer		python3 build fails when target is mips softfloat	normal	Medium		RESOLVED
13431	Ross Burton	Michael Halstead		Add new Debian Stable (Buster) autobuilder worker	normal	Medium+		RESOLVED
13290	Joshua Watt	Mingli Yu		elfutils ptest results are inconsistent per image	normal	Medium+		RESOLVED
13294	Joshua Watt	Mingli Yu		nettle ptest results are inconsistent per image	normal	Medium+		RESOLVED
13339	Nam Nguyen	Mingli Yu		Ruby 2.4.4 does not run with Apalis iMX6	normal	Medium+		RESOLVED
13243	Paul Eggleton	New Comer Bugs		devtool test failure in python scanning code	normal	Medium+	NEWCOMER	RESOLVED
13258	Kay Liu	Otavio Salvador		i.MX6 solox platform CPU hang in the case of using the Linux OS	normal	Medium		RESOLVED
11776	Patrick Ohly	Chen Qi	2	read-only rootfs + systemd: systemd-tmpfiles fails	normal	Medium+		RESOLVED
13061	Richard Purdie	Chen Qi		oe-selftest imagefeatures.ImageFeatures.test_image_fstypes (ubifs)	normal	Medium+		RESOLVED
13390	Richard Purdie	Chen Qi		oe-selftest qemu process handling issue	normal	Medium+		RESOLVED
12991	pravin	Richard Purdie		[2.6 M4 RC1][Build-Appliance] Bitbake build-appliance-image getting failed during building image due to webkitgtk package	normal	Medium+		RESOLVED
13208	Jaewon Lee	Richard Purdie		Python2.7 check in oe-buildenv-internal is erroring	major	Medium+		RESOLVED
13280	Denys Dmytriyenko	Richard Purdie		Race condition in wic to package IMAGE_BOOT_FILES before EXTRA_IMAGEDEPENDS are built	normal	Medium+		RESOLVED
13299	Joshua Watt	Richard Purdie		quilt ptest results are inconsistent per image	normal	Medium+	NEWCOMER	RESOLVED
12984	Joe Slater	Richard Purdie		Specifying invalid PRSERV_HOST causes bitbake to hang	normal	Medium+		RESOLVED
13057	Paul Eggleton	Ross Burton		No default network configuration for systemd-networkd	normal	Medium+		RESOLVED
13366	Richard Purdie	Ross Burton		bluez5 ptest hangs with gcc 9	normal	Medium+		RESOLVED
13072	Tim Orling	Ross Burton		vulkan: upstream repo has changed; latest version is 1.1.92.1	normal	Medium+		RESOLVED
13423	Richard Purdie	Ross Burton		efivar fails to build on opensuse423 with musl	normal	Medium+		RESOLVED
13291	Joshua Watt	Sakib Sajal		flex ptest results are inconsistent per image	normal	Medium+		RESOLVED
12857	Kai Ruhnau	Seebs		Running anything dotnet/CoreCLR based in a devshell doesn't work	normal	Medium+		RESOLVED
12873	Pau Espin Pedrol	Tim Orling	5	cpan_build.bbclass is broken (no dependency Module::Build)	normal	Medium+	Backport to 2.6.x, 2.7.x	RESOLVED
13200	Tim Orling	Tim Orling		perl-rdepends.txt missing many dependencies for perl-module-test-more	normal	Medium+	Backport to 2.7	RESOLVED
13381	Tim Orling	Tim Orling	1	SCons should migrate to Python 3	normal	Undecided		RESOLVED
13293	Joshua Watt	Unassigned		libxml-parser-perl ptest results are inconsistent per image	normal	Medium+	NEWCOMER	RESOLVED
13300	Joshua Watt	Unassigned		tcl ptest results are inconsistent per image	normal	Medium+	NEWCOMER	RESOLVED
12886	Ferry	He Zhe		Removal of VM_DEFAULT_KERNEL in Sumo breaks booting initramfs kernel	normal	Medium+	Move to YP 2.6.2, YP 2.5.3	RESOLVED
			11

The planed schedule for 2.8 can be found at Yocto_2.8_Schedule

Live Schedule & Execution History

MILESTONE	STARTED	FINISHED	STATUS	REPORT
M1 rc1	12/19/2018	1/2/2019	DONE	https://wiki.yoctoproject.org/wiki/WW01_-_2019-01-02_-_Full_Test_Cycle_2.7_M1_RC1
M2 rc1	01/28/2019	02/08/2019	DONE	https://wiki.yoctoproject.org/wiki/WW01_-_2019-01-28_-_Full_Test_Cycle_2.7_M2_RC1
M2 rc2	02/14/2019	02/25/2019	DONE	https://wiki.yoctoproject.org/wiki/WW07_-_2019-02-14_-_Full_Test_Cycle_2.7_M2_RC2
M3 rc1	03/21/2019	04/02/2019	DONE	https://wiki.yoctoproject.org/wiki/WW07_-_2019-04-02_-_Full_Test_Cycle_2.7_M3_RC1

Yocto Project 2.8 Release Test Plan