Yocto Project 2.3 Release Test Plan

{{#vardefine:ADT_MAUAL | 6539}} {{#vardefine:ADT_AUTO | 6540}} {{#vardefine:ANYQEMU_SATO_MANUAL | 6563}} {{#vardefine:ANYQEMU_SATO_AUTO | 6564}} {{#vardefine:ANYBSP_SATO_AUTO | 6558}} {{#vardefine:ANYBSP_LSB_AUTO | 6561}} {{#vardefine:ANYBSP_WIC_AUTO | 6560}} {{#vardefine:MMAX_CORE2_MANUAL | 6554}} {{#vardefine:MMAX_I7_MANUAL | 6557}} {{#vardefine:NUC_I7_MANUAL | 6556}} {{#vardefine:NUC_GEX86-64_MANUAL | 6555}} {{#vardefine:MMAX_GEX86-32_MANUAL | 6553}} {{#vardefine:GAL_SATO_AUTO | }} {{#vardefine:CROPS_AUTO | 6541}} {{#vardefine:CROPS_MANUAL | 6542}} {{#vardefine:ECLIPSE_MANUAL | 6543}} {{#vardefine:METAYOCTO_MANUAL | 6544}} {{#vardefine:TOASTER_AUTO | 6545}} {{#vardefine:TOASTER_MANUAL | 6546}} {{#vardefine:ESDK_MANUAL | 6547}} {{#vardefine:ESDK_AUTO | 6548}} {{#vardefine:OE-CORE_AUTO | 6549}} {{#vardefine:BITBAKE_AUTO | 6550}} {{#vardefine:RUNTIME_AUTO | 6551}} {{#vardefine:BUILD-APPLIANCE_AUTO | 6552}}

Reversion history

Version	Modifier	Comments
1.0	Jose Perez Carranza	First draft

References

Documents that support this test plan

Introduction

This is the the overall test plan for version 2.3 of the Yocto Project, for all the generic information this document will be pointing to QA Master Test Plan

Objectives

This test plan for Yocto Project 2.3 release supports the following objectives:

• To define the the tools to be used throughout the testing process.
• To communicate to the responsible parties the items to be tested, set expectations around schedule, and define environmental needs.
• To define how the tests will be conducted.
• Release Criteria
• Risks and contingencies

QA_Master_Test_Plan#Objectives_and_Tasks

Test Strategy

Is followed the general strategy defined on QA_Master_Test_Plan#Testing_Strategy and below is showed specific information applicable only for 2.3 Release

Bugs Tracking

• Bugs that need to be implemented by QA Team
  • 2.3 QA Assigned Bugs

• Bugs that need to be verified by QA Team
  • 2.3_QA_Bugs_To_Verify

• Old Bugs that need to be Verified
  • Old resolved bugs and features

Test Items

The Yocto Project QA team verifies each release against recent versions of the most popular Linux distributions that provide stable releases.

* centos 7 
* Debian 8
* Fedora 23 
* Fedora 24
* OpenSuse132 
* OpenSuse421 
* ubuntu 14.04 
* ubuntu 16.04

Test Cases

• Yocto 2.3 Test Cases

Features to be tested

The components to be tested for version 2.3 of the Yocto Project against the different milestones are included in this section:

Code of Colors
Active Tracked on Testpoia
Active Non Tracked on Testopia
In Design Will be tracked on Tesopia
Inactive

COMPONENT	TEMPLATE
ADT	TRTEMP_2.3_MANUAL_ADT
	TRTEMP_2.3_AUTO_ADT
BSP	TRTEMP_2.3_AUTO_BSP/QEMU_core-image-sato-sdk_ANYQEMU
	TRTEMP_2.3_AUTO_BSP/QEMU_core-image-sato-sdk_ANYBSP
	TRTEMP_2.3_AUTO_BSP/QEMU_core-image-lsb-sdk_ANYBSP
	TRTEMP_2.3_AUTO_BSP/QEMU_core-image-sato-sdk-wic_ANYBSP
	TRTEMP_2.3_AUTO_BSP/QEMU_core-image-sato-sdk_intel-quark
	TRTEMP_2.3_MANUAL_BSP/QEMU_core-image-sato-sdk_ANYQEMU
	TRTEMP_2.3_MANUAL_BSP/QEMU_core2-32_MinnowMax
	TRTEMP_2.3_MANUAL_BSP/QEMU_corei7-64_MinnowMax
	TRTEMP_2.3_MANUAL_BSP/QEMU_corei7-64_NUC
	}} TRTEMP_2.3_MANUAL_BSP/QEMU_GenericX86-64_NUC
	TRTEMP_2.3_MANUAL_BSP/QEMU_GenericX86_MMAX32
CROPS	TRTEMP_2.3_AUTO_CROPS
	TRTEMP_2.3_MANUAL_CROPS
Eclipse-Plugin	TRTEMP_2.3._MANUAL_Eclipse-Plugin
Meta-Yocto	TRTEMP_2.3_MANUAL_Meta-yocto
Toaster	TRTEMP_2.3_AUTO_TOASTER_UI
	TRTEMP_2.3_MANUAL_TOASTER_UI-managed-mode
eSDK	TRTEMP_2.3_AUTO_eSDK
	TRTEMP_2.3_MANUAL_eSDK
OE-CORE / BitBake	TRTEMP_2.3_AUTO_OE-Core
	TRTEMP_2.3_MANUAL_BitBake
Runtime (Compliance)	TRTEMP_2.3_AUTO_Runtime
Build Apliance	TRTEMP_2.3_MANUAL_Build-Appliance
pTest
Performance	Performance Test
WUpd (Software Updater)
Distributed Builds
MRAA
Meta-SwUpd

All component to be tested on 2.3 are in #Test Items a detailed list of features per component can be found in Test Plan of the components QA_Master_Test_Plan#Component

Features pending to be tested

Here is a list of areas that are not yet covered and need some work, implement test cases and add it to a component that belongs or create a new test plan for new component.

• build-appliance - can we run builds in the build-appliance image?
• Verify if -c menuconfig work?
• Verify if -c devshell work?
• Verify if BBPOSTCONF and BBPRECONF work in bitbake?
• Verify if oe-pkgdata-util print help when no parameters specified?
• Verify if parsing time is improved with a hot codeparser cache (build metrics did catch a regression here but we had to manually spot it).
• Verify Bitbake sane error if a layer doesn't exist? (no backtrace?) send-error-report - no unittesting, only caught python3 issues when it failed to send errors
• Verify if buildhistory_analsysis.py is tested
• Verify if compression image types is tested
• Verify combo-layer
• Verify yocto-bsp/yocto-layer
• Verify oe-selftest coverage
• Verify bitbake -u depexp -g XXX
• rpm to be replaced
• <ADD MORE>

Approach

Tests will be conducted per the documented test cases stored in Testopia. The test manager will create test runs for each tester. The tester will execute the tests in Testopia and mark each case as Pass / Fail / Skip /Blocked The tester should leave notes on actual results and any other relevant details when possible.

Once complete, the test manager should review the test run reports in Testopia and report back to the team by mail and in the wiki test report.

Pass / Fail Criteria

This criteria is defined in Yocto_Project_v2.3_Status#Yocto_Project_v2.3_Release_Criteria

Exit Criteria

As defined in QA_Master_Test_Plan#Entry and Exit Criteria

Test Deliverables

• Live Status 2.3_QA_Status
• Final execution Report #Execution History
• Mail sent to the mailing list

Test Environment

There are different environments and tools for every type of testing refer to QA_Master_Test_Plan#Test_Areas for more detailed information.

Schedule

The planed schedule for 2.3 can be found at Yocto_2.3_Schedule

Live Schedule

MILESTONE	STARTED	FINISHED	STATUS
M1
M2
M3
M4

Execution History

MILESTONE	REPORT
M1
M2
M3
M4