Yocto Project 2.3 Release Test Plan
Reversion history
| Version | Modifier | Comments |
|---|---|---|
| 1.0 | Jose Perez Carranza | First draft |
References
Documents that support this test plan
Introduction
This is the the overall test plan for version 2.3 of the Yocto Project, for all the generic information this document will be pointing to QA Master Test Plan
Objectives
This test plan for Yocto Project 2.3 release supports the following objectives:
- To define the the tools to be used throughout the testing process.
- To communicate to the responsible parties the items to be tested, set expectations around schedule, and define environmental needs.
- To define how the tests will be conducted.
- Release Criteria
- Risks and contingencies
QA_Master_Test_Plan#Objectives_and_Tasks
Test Strategy
Is followed the general strategy defined on QA_Master_Test_Plan#Testing_Strategy and below is showed specific information applicable only for 2.3 Release
Bugs Tracking
- Bugs that need to be implemented by QA Team
- Bugs that need to be verified by QA Team
- Old Bugs that need to be Verified
Test Items
The Yocto Project QA team verifies each release against recent versions of the most popular Linux distributions that provide stable releases.
* centos 7 * Debian 8 * Fedora 23 * Fedora 24 * OpenSuse132 * OpenSuse421 * ubuntu 14.04 * ubuntu 16.04
Test Cases
Components to be tested
The components to be tested for version 2.3 of the Yocto Project against the different milestones are included in this section:
| Code of Colors |
|---|
| Active
Tracked on Testpoia |
| Active
Non Tracked on Testopia |
| In Design
Will be tracked on Tesopia |
| Inactive |
All component to be tested on 2.3 are in #Test Items a detailed list of features per component can be found in Test Plan of the components QA_Master_Test_Plan#Component
Features pending to be tested
Here is a list of areas that are not yet covered and need some work, implement test cases and add it to a component that belongs or create a new test plan for new component.
| Features | |
|---|---|
| Bug ID | Description |
| Bug 10723 | Build-Appliance - Run builds in the build-appliance image |
| Bug 10724 | Verify if -c menuconfig work |
| Bug 10724 | Verify if -c devshell work |
| Bug 10725 | Verify if BBPOSTCONF and BBPRECONF work in bitbakek |
| Bug 10726 | Verify if oe-pkgdata-util print help when no parameters specified |
| Bug 10727 | Verify if buildhistory_analsysis.py is tested |
| Bug 10728 | Verify combo-layer |
| Bug 6926 | Verify yocto-bsp/yocto-layer |
| Bug 10724 | Verify bitbake -u depexp -g XXX |
The complete List of QA Implementations for YP 2.3 can be found at Features Implementation by QA Team
Features Not to be tested
Approach
Tests will be conducted per the documented test cases stored in Testopia. The test manager will create test runs for each tester. The tester will execute the tests in Testopia and mark each case as Pass / Fail / Skip /Blocked. The tester should leave notes on actual results and any other relevant details when possible.
Once complete, the test manager should review the test run reports in Testopia and report back to the team by mail and in the wiki as a test report.
Pass / Fail Criteria
This criteria is defined in Yocto_Project_v2.3_Status#Yocto_Project_v2.3_Release_Criteria
Exit Criteria
As defined in QA_Master_Test_Plan#Entry and Exit Criteria
Test Deliverables
- Live Status 2.3_QA_Status
- Final execution Report #Execution History
- Mail sent to the mailing list
Test Environment
There are different environments and tools for every type of testing refer to QA_Master_Test_Plan#Test_Areas for more detailed information.
Schedule
M1 (Oct. 31, 2016 to Dec. 23, 2016 -- 9 weeks: planning done in week 1, development week 2-6, stabilization week 7-8, release week 9)
M1 Feature/Tasks Development / Cutoff is Dec. 12, 2016
| ID | Requester | Assignee | E | Summary (6 tasks) ⇒ | Severity | P | Whiteboard | Status |
|---|---|---|---|---|---|---|---|---|
| 9088 | Aníbal Limón | Francisco Pedraza | 3 | Extensible SDK add tests for devtool | enhancement | Medium+ | RESOLVED | |
| 10753 | Francisco Pedraza | Francisco Pedraza | 5 | QA Kernel-dev test cases basic functionality | enhancement | Medium+ | VERIFIED | |
| 10394 | Jair Gonzalez | Jair Gonzalez | 2 | Creation of component to emulate keyboard with USB-KM232 | enhancement | High | RESOLVED | |
| 7376 | Ross Burton | Jose Perez C | 3 | oe-selftest doesn't run lib/oe/tests/* | enhancement | Medium+ | VERIFIED | |
| 8433 | Ross Burton | Jose Perez C | 4 | Add test for package postinsts | enhancement | Medium+ | RESOLVED | |
| 8584 | Ross Burton | Juan Ramos | 3 | Enable building API documentation | enhancement | Medium+ | 21 Nov 2016: RESOLVED | VERIFIED |
| 20 |
M1 Bugs
| ID | Requester | Assignee | E | Summary (4 tasks) ⇒ | Severity | P | Whiteboard | Status |
|---|---|---|---|---|---|---|---|---|
| 5319 | Mark Hatle | Francisco Pedraza | 3 | Need a test to verify firstboot postinst script order | normal | Medium+ | RESOLVED | |
| 10432 | Leonardo Sandoval Gonzalez | Francisco Pedraza | 2 | eSDK test test_install_libraries_headers fail when distro is nodistro | normal | Medium+ | VERIFIED | |
| 10481 | Jair Gonzalez | Jair Gonzalez | 1 | [Test Case 1059] Parselogs is failing on genericx86 (32 and 64 bits) WIC giving error: GPT: Use GNU Parted to correct GPT errors. | normal | Medium | RESOLVED | |
| 10594 | Jose Perez C | Jair Gonzalez | 2 | Ensure all options of WIC script are tested | normal | Medium+ | VERIFIED | |
| 8 |
M1 Stabilize schedule
Release Candidate and Milestone Release schedule
- RC1: 12/12/16
- M1 release: 12/23/16
The planed schedule for 2.3 can be found at Yocto_2.3_Schedule
Live Schedule
| MILESTONE | STARTED | FINISHED | STATUS |
|---|---|---|---|
| M1 rc1 | 12/19/2016 | 12/23/2016 | DONE |
| M2 rc1 | 01/24/2017 | 01/31/2017 | DONE |
| M2 rc3 | 02/03/2017 | 02/08/2017 | DONE |
| M3 rc1 | 03/15/2017 | 03/23/2017 | DONE |
| M3 rc2 | 03/28/2017 | 04/05/2017 | DONE |
| M3 rc1 Meta-Intel | 04/05/2017 | 04/12/2017 | DONE |
| M4 rc2 | 04/21/2017 | 04/29/2017 | DONE |
| M4 rc6 | 05/02/2017 | 05/08/2017 | DONE |
| 2.3 Meta-Intel rc1 | 05/11/2017 | 05/16/2017 | DONE |
| 2.3 Meta-Intel rc4 | 05/22/2017 | 05/25/2017 | DONE |
| 2.3.1 rc2 | 07/12/2017 | 07/18/2017 | DONE |
| 2.3.2 rc1 | 09/12/2017 | 09/20/2017 | DONE |
| 2.3.3 rc1 | 12/15/2017 | 12/20/2017 | DONE |
| 2.3.4 rc1 | 6/13/2018 | 7/10/2018 | DONE |
Execution History
| MILESTONE | REPORT |
|---|---|
| M1 rc1 | WW52 - 2016-12-19 - Full Test Cycle 2.3 M1 rc1 |
| M2 rc1 | WW05 - 2017-01-24 - Full Test Cycle 2.3_M2 rc1 |
| M2 rc3 | WW06 - 2017-02-03 - Full Test Cycle 2.3_M2 rc3 |
| M3 rc1 | WW11 - 2017-03-15 - Full Test Cycle 2.3 M3 rc1 |
| M3 rc2 | WW14 - 2017-03-28 - Full Test Cycle 2.3 M3 rc2 |
| M3 rc1 Meta-Intel | WW15 - 2017-04-05 - Full Test Cycle - 2.3 M3 - meta-intel |
| M4 rc2 (2.3 release - pyro) | WW17 - 2017-04-21 - Full Test Cycle 2.3 rc2 |
| M4 rc4 (2.3 release - pyro) | WW19 - 2017-05-02 - Full Test Cycle 2.3 rc4 |
| Meta-Intel rc1 (2.3 release - pyro) | WW20 - 2017-05-11 - Full Test Cycle - 2.3 rc1 - meta-intel |
| Meta-Intel rc4 (2.3 release - pyro) | WW21 - 2017-05-22 - Full Test Cycle - 2.3 rc4 - meta-intel |
| 2.3.1 rc2 | WW29 - 2017-07-18- Full Test Cycle - 2.3.1 rc2 |
| 2.3.2 rc1 | WW38_-_2017-09-20-_Full_Test_Cycle_-_2.3.2_rc1 |
| 2.3.3 rc1 | WW51_-_2017-12-20-_Full_Test_Cycle_-_2.3.3_rc1 |
| 2.3.4 rc1 | WW25_-_2018-06-22-_Full_Test_Cycle_-_2.3.4_rc1 |
