Yocto Project 3.4 Release Test Plan

{{#vardefine:ADT_MAUAL | 6539}} {{#vardefine:ADT_AUTO | 6540}} {{#vardefine:ANYQEMU_SATO_MANUAL | 6563}} {{#vardefine:ANYQEMU_SATO_AUTO | 6564}} {{#vardefine:ANYBSP_SATO_AUTO | 6558}} {{#vardefine:ANYBSP_LSB_AUTO | 6561}} {{#vardefine:ANYBSP_WIC_AUTO | 6560}} {{#vardefine:MMAX_CORE2_MANUAL | 6554}} {{#vardefine:MMAX_I7_MANUAL | 6557}} {{#vardefine:NUC_I7_MANUAL | 6556}} {{#vardefine:NUC_GEX86-64_MANUAL | 6555}} {{#vardefine:MMAX_GEX86-32_MANUAL | 6553}} {{#vardefine:GAL_SATO_AUTO | 6565}} {{#vardefine:CROPS_AUTO | 6541}} {{#vardefine:CROPS_MANUAL | 6542}} {{#vardefine:ECLIPSE_MANUAL | 6543}} {{#vardefine:KERNEL_MANUAL | 6714}} {{#vardefine:METAYOCTO_MANUAL | 6544}} {{#vardefine:TOASTER_AUTO | 6545}} {{#vardefine:TOASTER_MANUAL | 6546}} {{#vardefine:ESDK_MANUAL | 6547}} {{#vardefine:ESDK_AUTO | 6548}} {{#vardefine:OE-CORE_AUTO | 6549}} {{#vardefine:BITBAKE_AUTO | 6550}} {{#vardefine:RUNTIME_AUTO | 6551}} {{#vardefine:BUILD-APPLIANCE_AUTO | 6552}}

Revision history

Version	Modifier	Comments
1.0	Jain Sangeeta	First draft

Introduction

This is the the overall test plan for version 3.4 of the Yocto Project, for all the generic information this document will be pointing to QA Master Test Plan

Objectives

This test plan for Yocto Project 3.4 release supports the following objectives:

• To define the tools to be used throughout the testing process.
• To communicate to the responsible parties the items to be tested, set expectations around schedule, and define environmental needs.
• To define how the tests will be conducted.
• Release Criteria
• Risks and contingencies

QA_Master_Test_Plan#Objectives_and_Tasks

Risk Analysis & Contingency Plan

QA identified risks for this release are mentioned here.

Module	Risk Description	Risk Level	Contingency plan	Owner
BSP	31 manual test cases	medium	test cases to be automated	Intel
Build Appliance	3 manual test cases, not executed by any QA team	medium	test cases to be automated
Compliance	4 manual test cases, not executed by any QA team	medium	test cases to be automated
Kernel dev	9 manual test cases, not executed by any QA team	medium	test cases to be automated
OE-Core	4 manual test cases	medium	test cases to be automated	Intel
SDK	1 manual test case, not executed by any QA team	medium	test case to be automated
Toaster	93 manual test cases, not executed by any QA team	medium	test cases to be automated

Test Strategy

Is followed the general strategy defined on QA_Master_Test_Plan#Testing_Strategy and below is showed specific information applicable only for 3.4 Release

Test Items

The Yocto Project QA team verifies each release against recent versions of the most popular Linux distributions that provide stable releases.

* centos 7 
* Debian 8
* Debian 9
* Fedora 28
* Fedora 29
* Fedora 30
* OpenSuse 150
* OpenSuse 151
* OpenSuse 423
* OpenSuse Tumbleweed
* ubuntu 16.04 
* ubuntu 18.04

Test Cases

• http://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/meta/lib/oeqa/runtime
• http://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/meta/lib/oeqa/selftest
• http://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/meta/lib/oeqa/manual
• http://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/meta/lib/oeqa/sdk
• http://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/meta/lib/oeqa/sdkext

OE-CORE

Component	Environment	Auto	Manual	Owner
OE-CORE	Ubuntu 20.04		Y	Intel

BSP/QEMU

Component	Arch	HW	Environment	Image	Auto	Manual	pTest	Compliance	ltp
BSP	x86	MinwTurbot 32bit	genericx86	core-image-sato-sdk	Y
				core-image-lsb-sdk	Y
		Coffee Lake	genericx86-64	core-image-sato-sdk	Y
				core-image-lsb-sdk	Y
		NUC 6	genericx86-64	core-image-sato-sdk	Y		Y		Y
				core-image-lsb-sdk	Y			Y
		NUC 7	genericx86-64	core-image-sato-sdk	Y	Y
				core-image-lsb-sdk	Y
		QEMU	qemux86	core-image-sato-sdk	Y
				core-image-lsb-sdk	Y
			qemux86-64	core-image-sato-sdk	Y
				core-image-lsb-sdk	Y
	non x86	EdgeRouter	EdgeRouter	core-image-sato-sdk	Y	Y		Y
		Beaglebone Black	Beaglebone	core-image-sato-sdk	Y	Y		Y
		QEMU	qemuarm	core-image-sato-sdk	Y
			qemuarm-64	core-image-sato-sdk	Y		Y		Y
			qemuppc	core-image-sato-sdk	Y
			qemumips	core-image-sato-sdk	Y
			qemumips64	core-image-sato-sdk	Y
	Meta-Intel	MinwTurbot 32bit	core2-32	core-image-sato-sdk	Y
				core-image-lsb-sdk	Y
		Coffee Lake	corei7-64	core-image-sato-sdk	Y
				core-image-lsb-sdk	Y
		NUC 6	corei7-64	core-image-sato-sdk	Y
				core-image-lsb-sdk	Y
		NUC 7	corei7-64	core-image-sato-sdk	Y	Y
				core-image-lsb-sdk	Y
				poky-tiny		Y

All component to be tested on 3.4 are in #Test Items a detailed list of features per component can be found in Test Plan of the components QA_Master_Test_Plan#Component

Approach

Testing will be done as per QA process defined in main QA wiki page at https://wiki.yoctoproject.org/wiki/QA#QA_Process.

Pass / Fail Criteria

This criteria is defined in Yocto_Project_v3.4_Status#Yocto_Project_v3.4_Release_Criteria

Exit Criteria

As defined in QA_Master_Test_Plan#Entry and Exit Criteria

Test Deliverables

• Final Test results
• Final Test report
• Mail sent to the mailing list

Schedule

The planed schedule for 3.4 can be found at Yocto_3.4_Schedule

Live Schedule & Execution History

MILESTONE	STARTED	FINISHED	STATUS	REPORT
M1	06/12/2021	06/18/2021	DONE	https://git.yoctoproject.org/cgit/cgit.cgi/yocto-testresults-contrib/log/?h=intel-yocto-testresults
M2	07/26/2021	07/30/2021	DONE	https://git.yoctoproject.org/cgit/cgit.cgi/yocto-testresults-contrib/log/?h=intel-yocto-testresults
M3	09/06/2021		In Progress
M4