Revision as of 02:27, 18 February 2019

{{#vardefine:ADT_MAUAL | 6539}} {{#vardefine:ADT_AUTO | 6540}} {{#vardefine:ANYQEMU_SATO_MANUAL | 6563}} {{#vardefine:ANYQEMU_SATO_AUTO | 6564}} {{#vardefine:ANYBSP_SATO_AUTO | 6558}} {{#vardefine:ANYBSP_LSB_AUTO | 6561}} {{#vardefine:ANYBSP_WIC_AUTO | 6560}} {{#vardefine:MMAX_CORE2_MANUAL | 6554}} {{#vardefine:MMAX_I7_MANUAL | 6557}} {{#vardefine:NUC_I7_MANUAL | 6556}} {{#vardefine:NUC_GEX86-64_MANUAL | 6555}} {{#vardefine:MMAX_GEX86-32_MANUAL | 6553}} {{#vardefine:GAL_SATO_AUTO | 6565}} {{#vardefine:CROPS_AUTO | 6541}} {{#vardefine:CROPS_MANUAL | 6542}} {{#vardefine:ECLIPSE_MANUAL | 6543}} {{#vardefine:KERNEL_MANUAL | 6714}} {{#vardefine:METAYOCTO_MANUAL | 6544}} {{#vardefine:TOASTER_AUTO | 6545}} {{#vardefine:TOASTER_MANUAL | 6546}} {{#vardefine:ESDK_MANUAL | 6547}} {{#vardefine:ESDK_AUTO | 6548}} {{#vardefine:OE-CORE_AUTO | 6549}} {{#vardefine:BITBAKE_AUTO | 6550}} {{#vardefine:RUNTIME_AUTO | 6551}} {{#vardefine:BUILD-APPLIANCE_AUTO | 6552}}

Reversion history

Version	Modifier	Comments
1.0	Yeoh Ee Peng	First draft

References

Documents that support this test plan

Introduction

This is the the overall test plan for version 2.7 of the Yocto Project, for all the generic information this document will be pointing to QA Master Test Plan

Objectives

This test plan for Yocto Project 2.7 release supports the following objectives:

To define the tools to be used throughout the testing process.
To communicate to the responsible parties the items to be tested, set expectations around schedule, and define environmental needs.
To define how the tests will be conducted.
Release Criteria
Risks and contingencies

QA_Master_Test_Plan#Objectives_and_Tasks

Risk Analysis & Contingency Plan

This is the document for Risk Analysis https://docs.google.com/spreadsheets/d/1K-MZYo5sfabwI2UozGp6Uuh2sw5y309bXkyEZRPEEgk/edit#gid=0

Test Strategy

Is followed the general strategy defined on QA_Master_Test_Plan#Testing_Strategy and below is showed specific information applicable only for 2.7 Release

Bugs Tracking

Bugs that need to be implemented by QA Team
- 2.7 QA Assigned Bugs

Bugs that need to be verified by QA Team
- 2.7_QA_Bugs_To_Verify

Old Bugs that need to be Verified
- Old resolved bugs and features

Test Items

The Yocto Project QA team verifies each release against recent versions of the most popular Linux distributions that provide stable releases.

* centos 7 
* Debian 9
* Fedora 26
* Fedora 27
* OpenSuse42.3
* ubuntu 16.04 
* ubuntu 17.10

Test Cases

Yocto 2.7 Test Cases

Test Matrix

The components to be tested for version 2.7 of the Yocto Project against the different milestones are included in this section:

Code of Colors
Active Tracked on Testpoia
Active Non Tracked on Testopia
In Design Will be tracked on Tesopia
Inactive

ADT

Component	Environment	Auto	Manual	Owner
ADT	Fedora 25 i686	Y	Y	PG
ADT	Ubuntu 16.04 x86_64	Y	Y	PG

Eclipse Plugin

Component	Environment	Auto	Manual	Owner
Neon + Any Distro			Y	PG
Oxygen + Any Distro			Y	PG

Kernel

Component	Environment	Auto	Manual	Owner
Kernel	Any Distro		Y	PG

Meta-Yocto

Component	Environment	Auto	Manual	Owner
Meta-Yocto	Any Distro		Y	PG

Toaster

Component	Environment	Auto	Manual	Owner
Toaster	Any Distro		Y	PG

eSDK

Component	Environment	Auto	Manual	Owner
eSDK	Any Distro + runtime	Y		PG
eSDK	Any Distro + selftest	Y		PG

Selftest

Component	Environment	Auto	Owner
OE-CORE	Ubuntu 16.04	Y	PG
	Ubuntu 17.10	Y	PG
	OpenSuse 42.3	Y	PG
	Debian 9	Y	PG
	Centos 7	Y	PG
	Fedora 26	Y	PG
	Fedora 27	Y	PG

Bitbake

Component	Environment	Auto	Manual	Owner
Bitbake	Any Distro		Y	PG

Build - Apliance

Component	Environment	Auto	Manual	Owner
Build - Apliance	VM - Any Distro		Y	PG

Package Manager

Component	Environment	Auto	Manual	Owner
Package Manager	Any Distro	Y		PG

Performance

Component	Environment	Auto	Manual	Owner
Performance	Centos 7	Y		PG
Performance	Ubuntu 16.04	Y		PG

BSP/QEMU

Component	Arch	HW	Environment	Image	Auto	Manual	pTest	Compliance
BSP	x86	MinwTurbot 32bit	genericx86	core-image-sato-sdk	Y
		MinwTurbot 32bit	genericx86	core-image-lsb-sdk	Y
		MinwTurbot 64bit	genericx86-64	core-image-sato-sdk	Y	Y
		MinwTurbot 64bit	genericx86-64	core-image-lsb-sdk	Y
		NUC 6	genericx86-64	core-image-sato-sdk	Y		Y
		NUC 6	genericx86-64	core-image-lsb-sdk	Y			Y
		NUC 7	genericx86-64	core-image-sato-sdk	Y	Y
		NUC 7	genericx86-64	core-image-lsb-sdk	Y
		QEMU	qemux86	core-image-sato-sdk	Y	Y
			qemux86	core-image-lsb-sdk	Y
			qemux86-64	core-image-sato-sdk	Y	Y
			qemux86-64	core-image-lsb-sdk	Y
	non x86	EdgeRouter	EdgeRouter	core-image-sato-sdk	Y	Y
		MPC8315e-rdb	MPC8315e-rdb	core-image-sato-sdk	Y	Y
		Beaglebone Black	Beaglebone	core-image-sato-sdk	Y	Y
		QEMU	qemuarm	core-image-sato-sdk	Y	Y
			qemuarm-64	core-image-sato-sdk	Y	Y
			qemuppc	core-image-sato-sdk	Y	Y
			qemumips	core-image-sato-sdk	Y	Y
			qemumips64	core-image-sato-sdk	Y	Y
	Meta-Intel	MinwTurbot 32bit	core2-32	core-image-sato-sdk	Y
		MinwTurbot 32bit	core2-32	core-image-lsb-sdk	Y
		MinwTurbot 64bit	corei7-64	core-image-sato-sdk	Y	Y
		MinwTurbot 64bit	corei7-64	core-image-lsb-sdk	Y
		NUC 6	corei7-64	core-image-sato-sdk	Y
		NUC 6	corei7-64	core-image-lsb-sdk	Y
		NUC 7	corei7-64	core-image-sato-sdk	Y	Y
				core-image-lsb-sdk	Y
				poky-tiny		Y

All component to be tested on 2.7 are in #Test Items a detailed list of features per component can be found in Test Plan of the components QA_Master_Test_Plan#Component

Testopia Templates

COMPONENT	TEMPLATE
BitBake	TRTEMP_2.6_MANUAL_BitBake
BSP/QEMU	TRTEMP_2.6_MANUAL_BSP/QEMU_core-image-sato-sdk_Genericx86_MMAX32
	TRTEMP_2.6_MANUAL_BSP/QEMU_core-image-sato-sdk_core2-32_MMAX32
	TRTEMP_2.6_MANUAL_BSP/QEMU_core-image-sato-sdk_Genericx86-64_NUC
	TRTEMP_2.6_MANUAL_BSP/QEMU_core-image-sato-sdk_corei7-64_NUC
	TRTEMP_2.6_MANUAL_BSP/QEMU_core-image-sato-sdk_corei7-64_MMAX64
	TRTEMP_2.6_AUTO_BSP/QEMU_core-image-sato-sdk_ANYBSP
	TRTEMP_2.6_AUTO_BSP/QEMU_core-image-sato-sdk_wic_ANYBSP
	TRTEMP_2.6_AUTO_BSP/QEMU_core-image-lsb-sdk_ANYBSP
	TRTEMP_2.6_MANUAL_BSP/QEMU_core-image-sato-sdk_ANYQEMU
	TRTEMP_2.6_AUTO_BSP/QEMU_core-image-sato-sdk_ANYQEMU
	TRTEMP_2.6_AUTO_BSP/QEMU_core-image-sato-sdk_intel-quark
	TRTEMP_2.6_MANUAL_BSP/QEMU_core-image-sato-sdk_Beaglebone-Black
	TRTEMP_2.6_MANUAL_BSP/QEMU_core-image-sato-sdk_EdgeRouter
	TRTEMP_2.6_MANUAL_BSP/QEMU_core-image-sato-sdk_MPC8315e-rdb
	TRTEMP_2.6_AUTO_BSP/QEMU_core-image-lsb-sdk_ANYQEMU
Meta-Yocto	TRTEMP_2.6_MANUAL_Meta-yocto
OE-CORE	TRTEMP_2.6_AUTO_OE-Core
Build-Appliance	TRTEMP_2.6_MANUAL_Build-Appliance
Runtime	TRTEMP_2.6_AUTO_Runtime
Toaster	TRTEMP_2.6_AUTO_Toaster_UI
Toaster	TRTEMP_2.6_MANUAL_Toaster_UI-managed-mode
ADT	TRTEMP_2.6_MANUAL_ADT
ADT	TRTEMP_2.6_AUTO_ADT
CROPS	TRTEMP_2.6_AUTO_CROPS_Poky
CROPS	TRTEMP_2.6_MANUAL_CROPS_eSDK
Eclipse	TRTEMP_2.6_MANUAL_Eclipse
eSDK	TRTEMP_2.6_AUTO_eSDK_sdkext
eSDK	TRTEMP_2.6_AUTO_eSDK_oeselftest
Kernel	TRTEMP_2.6_MANUAL_Kernel
Package Management	TRTEMP_2.6_MANUAL_Package-Management-Updates

Features pending to be tested

Here is a list of areas that are not yet covered and need some work, implement test cases and add it to a component that belongs or create a new test plan for new component.

ADD FEATURES !!!!!!!!!!

The complete List of QA Implementations for YP 2.7 can be found at Features Implementation by QA Team

Features Not to be tested

Approach

Tests will be conducted per the documented test cases stored in Testopia. The test manager will create test runs for each tester. The tester will execute the tests in Testopia and mark each case as Pass / Fail / Skip /Blocked. The tester should leave notes on actual results and any other relevant details when possible.

Once complete, the test manager should review the test run reports in Testopia and report back to the team by mail and in the wiki as a test report.

Pass / Fail Criteria

This criteria is defined in Yocto_Project_v2.7_Status#Yocto_Project_v2.7_Release_Criteria

Exit Criteria

As defined in QA_Master_Test_Plan#Entry and Exit Criteria

Test Deliverables

Live Status 2.7_QA_Status
Final execution Report #Execution History
Mail sent to the mailing list

Test Environment

There are different environments and tools for every type of testing refer to QA_Master_Test_Plan#Test_Areas for more detailed information.

Schedule

M1 (Apr 30, 2018 to Jun 22, 2018)

M1 Feature/Tasks Development / Cutoff is Jun. 11, 2017

ID	Requester	Assignee	E	Summary (12 tasks) ⇒	Severity	P	Status
13058	Chen Qi	André Draszik	2	split util-linux automatically instead of hard coding	enhancement	Medium+	RESOLVED
13094	David Reyna	David Reyna	3	support custom extensions to reports	enhancement	Medium	RESOLVED
13110	David Reyna	David Reyna	3	Add "release" records to product records	enhancement	Medium	RESOLVED
11583	Aníbal Limón	yeoh ee peng	4	oeqa: core add support to publish results	enhancement	Medium+	RESOLVED
12774	Lei Yang	Hongzhi		oeqa: kernel config depenency for some kernel test cases	enhancement	Medium+	RESOLVED
12931	Joshua Watt	Joshua Watt	5	Change default PACKAGE_DEBUG_SPLIT_STYLE to "debug-with-srcpkg"	enhancement	Medium+	RESOLVED
13030	Joshua Watt	Joshua Watt		Add sstate hash equivalence support	enhancement	Medium+	RESOLVED
13088	Richard Purdie	Aníbal Limón	0.3	Add support for extracting individual ptest timing	enhancement	Medium+	RESOLVED
12332	Paul Eggleton	Michael Halstead	2	Regularly purge the error report database	enhancement	Medium+	RESOLVED
12421	Juro Bystricky	Mingli Yu		Host-compiled Python bytecode not reproducible	enhancement	Medium+	RESOLVED
13011	Armin Kuster	Richard Purdie		Start collecting ptest for arm and x86 on Vtargets.	enhancement	Medium+	RESOLVED
13114		Ross Burton		Split python3-ctypes tests package	enhancement	Medium+	RESOLVED
			19.3

M1 Bugs

ID	Requester	Assignee	E	Summary (62 tasks) ⇒	Severity	P	Whiteboard	Status
12985	Christian Andersen	Alejandro Hernandez		multiconfig in v2.6 no longer works with COMPATIBLE_MACHINE (e.g. in image recipes)	normal	Medium+	Backport to 2.6.1	RESOLVED
13028	Denys Dmytriyenko	Alexander Kanavin		ENABLE_BINARY_LOCALE_GENERATION = "0" is now broken by postinst refactoring	normal	Medium+		RESOLVED
5697	Chong Lu	Changqing Li	1	busybox: ptest cases failed	normal	Medium		RESOLVED
8832	Alexandru Georgescu	Changqing Li	1	pTest - bash tests fail for 2.1_M1.rc1	normal	Medium		RESOLVED
9042	Francisco Pedraza	Changqing Li	3	pTest - strace fails on NUC 5425WYK for YP 2.1_M2.rc1	normal	Medium		RESOLVED
12997	Ross Burton	David Reyna	1	MITRE import not working	normal	Medium		RESOLVED
12998	Ross Burton	David Reyna	2	Triage page should also have Investigate button	normal	Medium		RESOLVED
12994	Ross Burton	David Reyna	2	Make checkboxes easier to use in triage	normal	Medium		RESOLVED
12999	Ross Burton	David Reyna	4	Triage prediction isn't working	normal	Medium		RESOLVED
13093	David Reyna	David Reyna	1	Fix pylint reported issues	normal	Medium		RESOLVED
13099	Ross Burton	David Reyna	1	String/list confusion in user groups	normal	Medium		RESOLVED
13121	Karin Julenius	David Reyna	2	CVE Resource information not available in SRTool	normal	Medium		RESOLVED
13125	Ross Burton	David Reyna	1	CPEs for Yocto Project not correct	normal	Medium		RESOLVED
13126	Ross Burton	David Reyna	1	Don't try Py2 imports	normal	Medium		RESOLVED
13131	Ross Burton	David Reyna	3	update script fails	normal	Medium		RESOLVED
13134	Ross Burton	David Reyna	1	Doesn't fetch 2019 CVE data	normal	Medium		RESOLVED
13135	Ross Burton	David Reyna	2	New CVEs are Historical	normal	Medium		RESOLVED
13138	Ross Burton	David Reyna	1	srtool_update throws exception	normal	Medium		RESOLVED
13139	Ross Burton	David Reyna	2	Triage fails	normal	Medium		RESOLVED
13091	David Reyna	David Reyna	1	SRTool broken for Django-2.2	normal	Medium+		RESOLVED
10713	brian avery	yeoh ee peng	3	Allow testimage to work with slirp in addition to working with tun/tap	normal	Medium+		RESOLVED
2655	veera	Hongxu Jia		In /usr/bin/ldd shared library path mentioned multiple times ( RTLDLIST="/lib/ld-linux.so.2 /lib/ld-linux.so.2 )	normal	Low		RESOLVED
12480	Juro Bystricky	Hongxu Jia	3	libc6-dbg: non deterministic build (multilib)	normal	Medium		RESOLVED
12524	Juro Bystricky	Hongxu Jia	2	(reproducibility) autoconf-info contains build date	normal	Medium	reproducibility	RESOLVED
12525	Juro Bystricky	Hongxu Jia	2	(reproducibility) python3-iniparse contains build host references	normal	Medium	reproducibility	RESOLVED
12543	Juro Bystricky	Hongxu Jia	2	(reproducibility) python-xcbgen contains host references and timestamps	normal	Medium+	reproducibility	RESOLVED
12709	Juro Bystricky	Hongxu Jia	2	(reproducibility) strace-ptest contains build host reference	normal	Medium+	reproducibility	RESOLVED
13128	Ross Burton	Joshua Watt		nativesdk-icecc doesn't work for multilib SDKs	normal	Medium		RESOLVED
13020	Ross Burton	Joshua Watt		meta-mingw needs testing	normal	Medium+		RESOLVED
13108		Joshua Watt		log show many fetch error after commit: bitbake: persist_data: Retry database setup	normal	Medium+		RESOLVED
13124	Alejandro Hernandez	Joshua Watt		multiconfig builds break when preparing the task hash (runqueue)	normal	Medium+		RESOLVED
11969	Carlos Alberto Lopez Perez	Kai Kang	2	issues with network-manager handling of resolv.conf when systemd-resolved is not enabled	normal	Medium		RESOLVED
8586	Jussi Kukkonen	Kai Kang	1	multilib build includes 32bit connman without good reason	normal	Medium+		RESOLVED
12288	Robert Yang	Kai Kang	2	lib32 packages are wrongly installed	normal	Medium+	Backport 2.3.3 and 2.4.1?	RESOLVED
12298	Robert Yang	Kai Kang	2	bitbake lib32-meta-ide-support Manifest not found	normal	Medium+	Backport to 2.4.1 and 2.3.3 if needed.	RESOLVED
13074	Kai Kang	Kai Kang	2	It fails to build systemd on qemux86-64 with default tune x86-64-x32	normal	Medium+		RESOLVED
13081	Kai Kang	Kai Kang	1	lttng-tools fails to build for x32	normal	Medium+		RESOLVED
12987	Frederik Noe-Sdun	Christopher Larson		Remove variables containing shell substitutions from vardeps of shell function if it cannot be expanded	normal	Medium+		RESOLVED
12380	Jackie Huang	Robert Yang	2	Variable definition: leading spaces stripping behavior is not consistent	normal	Medium		RESOLVED
12112	Christian Andersen	Robert Yang	3	Public key for U-Boot verified boot is not inserted in DTB when rebuilding from sstate	major	Medium+	Backport 2.4.4, 2.5.1	RESOLVED
12898	Paul Eggleton	Robert Yang	2	Bitbake does not print server errors when there is no TTY	normal	Medium+		RESOLVED
12992	Mohamad Noor Alim	Mark Hatle		[2.6 M4 rc1] test_devtool_add_fetch_git failed on Fedora 27	normal	Medium+		RESOLVED
11020	brian avery	Michael Halstead	1	We need a patchwork instance pointing to the Yocto mailing list	normal	Medium+		RESOLVED
5057	Ilovetogetspamed	Mingli Yu	4	Conflict when building SDK for image containing dropbear and openssh-sftp-server	normal	Medium		RESOLVED
12639	Juro Bystricky	Mingli Yu		(reproducibility) nspr packages contain timestamps	normal	Medium+	reproducibility	RESOLVED
13043	Ross Burton	Richard Purdie		runqemu.RunqemuTests.* failing	normal	High		RESOLVED
13044	Ross Burton	Richard Purdie		Bitbake failing to start in wic test	normal	High		RESOLVED
13045	Ross Burton	Richard Purdie		gpg agent socket not found	normal	High		RESOLVED
13046	Ross Burton	Richard Purdie		ccache failure in oe-selftest	normal	High		RESOLVED
13068	Enrico Scholz	Richard Purdie		fetch: module 'os' has no attribute 'errno'	normal	High		RESOLVED
12586	Denys Dmytriyenko	Richard Purdie		yocto-check-layer fails signatures due to TUNES between ARM archs	normal	Medium+		RESOLVED
13003	Michael Halstead	Richard Purdie		Build hangs if expected bitbake.lock lock file is missing	normal	Medium+		RESOLVED
13007	Armin Kuster	Richard Purdie		oe-selftest on each supported distro	normal	Medium+		RESOLVED
13018	Armin Kuster	Richard Purdie		Separate logic on autobuilder for a lighter test run and a longer more comprehensive one	normal	Medium+		RESOLVED
13127	Jin	Richard Purdie		changes in bbappend do not trigger a rebuild	normal	Medium+		RESOLVED
13042	Ross Burton	Ross Burton		imagefeatures.ImageFeatures.test_image_fstypes failed in cpio	normal	High		RESOLVED
12966		Ross Burton		Mesa depends on xorgproto even when x11 isn't in DISTRO_FEATURES	normal	Medium+	Backport to 2.6.1	RESOLVED
12441	Libertad	Sudhir Sharma		ptest comparison script is not showing correct results	critical	Medium+		RESOLVED
13115	Richard Weinberger	Scott Rifenbark	1	Clarify IMAGE_FSTYPES note	normal	Medium	03 January 2019: RESOLVED	RESOLVED
12399	brian avery	Scott Rifenbark	5	add non conflicting parts of the bitbake manual to the mega manual	normal	Medium+	14 February 2019: RESOLVED	RESOLVED
13112	Richard Purdie	Scott Rifenbark	2	FILESPATH and FILESOVERRIDES documentation lacks key detail	normal	Medium+	18 February 2019: RESOLVED	RESOLVED
13101	Anuj Mittal	Tim Orling		eclipse oxygen-master build failing consistently	normal	Medium+		RESOLVED
			74

The planed schedule for 2.7 can be found at Yocto_2.7_Schedule

Live Schedule & Execution History

MILESTONE	STARTED	FINISHED	STATUS	REPORT
M1 rc1	12/19/2018	1/2/2019	DONE	https://wiki.yoctoproject.org/wiki/WW01_-_2019-01-02_-_Full_Test_Cycle_2.7_M1_RC1
M2 rc1	01/28/2019	02/08/2019	DONE	https://wiki.yoctoproject.org/wiki/WW01_-_2019-01-28_-_Full_Test_Cycle_2.7_M2_RC1
M2 rc2	02/14/2019		IN PROGRESS	https://wiki.yoctoproject.org/wiki/WW07_-_2019-02-14_-_Full_Test_Cycle_2.7_M2_RC2

@@ Line 742: / Line 742: @@
 |
 | IN PROGRESS
-| https://wiki.yoctoproject.org/wiki/WW07_-_2019-01-14_-_Full_Test_Cycle_2.7_M2_RC2
+| https://wiki.yoctoproject.org/wiki/WW07_-_2019-02-14_-_Full_Test_Cycle_2.7_M2_RC2
 |
 |}

Yocto Project 2.7 Release Test Plan: Difference between revisions