Yocto Project 2.6 Release Test Plan: Difference between revisions
(Created page with "<!-- Variables for Template --> <!-- ********************** --> {| {{#vardefine:ADT_MAUAL | 6539}} {{#vardefine:ADT_AUTO | 6540}} {{#vardefine:ANYQEMU_S...") |
No edit summary |
||
(43 intermediate revisions by 2 users not shown) | |||
Line 50: | Line 50: | ||
This test plan for Yocto Project 2.6 release supports the following objectives: | This test plan for Yocto Project 2.6 release supports the following objectives: | ||
*To define | *To define the tools to be used throughout the testing process. | ||
*To communicate to the responsible parties the items to be tested, set expectations around schedule, and define environmental needs. | *To communicate to the responsible parties the items to be tested, set expectations around schedule, and define environmental needs. | ||
*To define how the tests will be conducted. | *To define how the tests will be conducted. | ||
Line 69: | Line 69: | ||
*Bugs that need to be implemented by QA Team | *Bugs that need to be implemented by QA Team | ||
**[[2. | **[[2.6_QA_OWNED_BUGS | 2.6 QA Assigned Bugs]] | ||
*Bugs that need to be verified by QA Team | *Bugs that need to be verified by QA Team | ||
Line 205: | Line 205: | ||
| Toaster | | Toaster | ||
| Any Distro | | Any Distro | ||
| | | | ||
| Y | | Y | ||
| PG | | PG | ||
Line 241: | Line 241: | ||
|- | |- | ||
| rowspan="8" style="text-align: center;" | OE-CORE | | rowspan="8" style="text-align: center;" | OE-CORE | ||
| Ubuntu 16.04 | | Ubuntu 16.04 | ||
| Y | | Y | ||
Line 251: | Line 246: | ||
| PG | | PG | ||
|- | |- | ||
| | | Ubuntu 17.10 | ||
| Y | | Y | ||
| | | | ||
| PG | | PG | ||
|- | |- | ||
| | | OpenSuse 42.3 | ||
| Y | | Y | ||
| | | | ||
| PG | | PG | ||
|- | |- | ||
| Debian | | Debian 9 | ||
| Y | | Y | ||
| | | | ||
Line 271: | Line 266: | ||
| PG | | PG | ||
|- | |- | ||
| Fedora | | Fedora 26 | ||
| Y | | Y | ||
| | | | ||
| PG | | PG | ||
|- | |- | ||
| Fedora | | Fedora 27 | ||
| Y | | Y | ||
| | | | ||
Line 325: | Line 320: | ||
| Package Manager | | Package Manager | ||
| Any Distro | | Any Distro | ||
| Y | |||
| | | | ||
| PG | | PG | ||
|} | |} | ||
Line 340: | Line 335: | ||
|- | |- | ||
| rowspan="2" style="text-align: center;" | Performance | | rowspan="2" style="text-align: center;" | Performance | ||
| | | Centos 7 | ||
| Y | | Y | ||
| | | | ||
| | | PG | ||
|- | |- | ||
| Ubuntu | | Ubuntu 16.04 | ||
| Y | | Y | ||
| | | | ||
| | | PG | ||
|} | |} | ||
Line 364: | Line 359: | ||
! Compliance | ! Compliance | ||
|- | |- | ||
| rowspan=" | | rowspan="30" style="text-align: center;" | BSP | ||
| rowspan=" | | rowspan="12" style="text-align: center;" | x86 | ||
| rowspan="2" style="text-align: center;" | | | rowspan="2" style="text-align: center;" | MinwTurbot 32bit | ||
| rowspan="2" style="text-align: center;" | genericx86 | | rowspan="2" style="text-align: center;" | genericx86 | ||
| core-image-sato-sdk | | core-image-sato-sdk | ||
| Y | | Y | ||
| | | | ||
| | | | ||
| | | | ||
Line 380: | Line 375: | ||
| | | | ||
|- | |- | ||
| rowspan=" | | rowspan="2" style="text-align: center;" | MinwTurbot 64bit | ||
| rowspan=" | | rowspan="2" style="text-align: center;" | genericx86-64 | ||
| core-image-sato-sdk | | core-image-sato-sdk | ||
| Y | | Y | ||
| | | Y | ||
| | | | ||
| | | | ||
Line 394: | Line 389: | ||
| | | | ||
|- | |- | ||
| core-image- | | rowspan="2" style="text-align: center;" | NUC 6 | ||
| rowspan="2" style="text-align: center;" | genericx86-64 | |||
| core-image-sato-sdk | |||
| Y | |||
| | |||
| Y | | Y | ||
| | | | ||
|- | |||
| core-image-lsb-sdk | |||
| Y | |||
| | | | ||
| | | | ||
| Y | |||
|- | |- | ||
| rowspan="2" style="text-align: center;" | NUC | | rowspan="2" style="text-align: center;" | NUC 7 | ||
| style="text-align: center;" | genericx86-64 | | rowspan="2" style="text-align: center;" | genericx86-64 | ||
| core-image-sato-sdk | | core-image-sato-sdk | ||
| Y | | Y | ||
| Y | | Y | ||
| | | | ||
| | | | ||
|- | |- | ||
| core-image-lsb-sdk | | core-image-lsb-sdk | ||
| Y | | Y | ||
| | | | ||
| | | | ||
| | | | ||
|- | |- | ||
| rowspan="4" style="text-align: center;" | QEMU | | rowspan="4" style="text-align: center;" | QEMU | ||
Line 504: | Line 506: | ||
|- | |- | ||
| rowspan="10" style="text-align: center;" | Meta-Intel | | rowspan="10" style="text-align: center;" | Meta-Intel | ||
| rowspan="2" style="text-align: center;" | | | rowspan="2" style="text-align: center;" | MinwTurbot 32bit | ||
| rowspan="2" style="text-align: center;" | core2-32 | | rowspan="2" style="text-align: center;" | core2-32 | ||
| core-image-sato-sdk | | core-image-sato-sdk | ||
| Y | | Y | ||
| | | | ||
| | | | ||
| | | | ||
Line 518: | Line 520: | ||
| | | | ||
|- | |- | ||
| rowspan="2" style="text-align: center;" | | | rowspan="2" style="text-align: center;" | MinwTurbot 64bit | ||
| rowspan="2" style="text-align: center;" | corei7-64 | | rowspan="2" style="text-align: center;" | corei7-64 | ||
| core-image-sato-sdk | | core-image-sato-sdk | ||
| Y | | Y | ||
| | | Y | ||
| | | | ||
| | | | ||
Line 532: | Line 534: | ||
| | | | ||
|- | |- | ||
| rowspan="2" style="text-align: center;" | NUC | | rowspan="2" style="text-align: center;" | NUC 6 | ||
| rowspan="2" style="text-align: center;" | corei7-64 | | rowspan="2" style="text-align: center;" | corei7-64 | ||
| core-image-sato-sdk | | core-image-sato-sdk | ||
| Y | | Y | ||
| | | | ||
| | | | ||
| | | | ||
Line 546: | Line 548: | ||
| | | | ||
|- | |- | ||
| rowspan=" | | rowspan="3" style="text-align: center;" | NUC 7 | ||
| rowspan=" | | rowspan="3" style="text-align: center;" | corei7-64 | ||
| core-image-sato-sdk | | core-image-sato-sdk | ||
| Y | | Y | ||
| | | Y | ||
| | | | ||
| | | | ||
Line 560: | Line 562: | ||
| | | | ||
|- | |- | ||
| | | poky-tiny | ||
| | | | ||
| Y | | Y | ||
| | | | ||
| | | | ||
|} | |} | ||
All component to be tested on 2. | All component to be tested on 2.6 are in [[#Test Items]] a detailed list of features per component can be found in Test Plan of the components [[QA_Master_Test_Plan#Component]] | ||
= Testopia Templates = | = Testopia Templates = | ||
Line 586: | Line 578: | ||
|- | |- | ||
| BitBake | | BitBake | ||
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7403 TRTEMP_2. | | [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7403 TRTEMP_2.6_MANUAL_BitBake] | ||
|- | |- | ||
| rowspan="15" | BSP/QEMU | | rowspan="15" | BSP/QEMU | ||
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7404 TRTEMP_2. | | [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7404 TRTEMP_2.6_MANUAL_BSP/QEMU_core-image-sato-sdk_Genericx86_MMAX32] | ||
|- | |- | ||
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7405 TRTEMP_2. | | [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7405 TRTEMP_2.6_MANUAL_BSP/QEMU_core-image-sato-sdk_core2-32_MMAX32] | ||
|- | |- | ||
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7406 TRTEMP_2. | | [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7406 TRTEMP_2.6_MANUAL_BSP/QEMU_core-image-sato-sdk_Genericx86-64_NUC] | ||
|- | |- | ||
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7407 TRTEMP_2. | | [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7407 TRTEMP_2.6_MANUAL_BSP/QEMU_core-image-sato-sdk_corei7-64_NUC] | ||
|- | |- | ||
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7408 TRTEMP_2. | | [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7408 TRTEMP_2.6_MANUAL_BSP/QEMU_core-image-sato-sdk_corei7-64_MMAX64] | ||
|- | |- | ||
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7409 TRTEMP_2. | | [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7409 TRTEMP_2.6_AUTO_BSP/QEMU_core-image-sato-sdk_ANYBSP] | ||
|- | |- | ||
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7410 TRTEMP_2. | | [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7410 TRTEMP_2.6_AUTO_BSP/QEMU_core-image-sato-sdk_wic_ANYBSP] | ||
|- | |- | ||
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7411 TRTEMP_2. | | [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7411 TRTEMP_2.6_AUTO_BSP/QEMU_core-image-lsb-sdk_ANYBSP] | ||
|- | |- | ||
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7412 TRTEMP_2. | | [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7412 TRTEMP_2.6_MANUAL_BSP/QEMU_core-image-sato-sdk_ANYQEMU] | ||
|- | |- | ||
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7413 TRTEMP_2. | | [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7413 TRTEMP_2.6_AUTO_BSP/QEMU_core-image-sato-sdk_ANYQEMU] | ||
|- | |- | ||
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7414 TRTEMP_2. | | [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7414 TRTEMP_2.6_AUTO_BSP/QEMU_core-image-sato-sdk_intel-quark] | ||
|- | |- | ||
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7415 TRTEMP_2. | | [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7415 TRTEMP_2.6_MANUAL_BSP/QEMU_core-image-sato-sdk_Beaglebone-Black] | ||
|- | |- | ||
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7416 TRTEMP_2. | | [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7416 TRTEMP_2.6_MANUAL_BSP/QEMU_core-image-sato-sdk_EdgeRouter] | ||
|- | |- | ||
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7417 TRTEMP_2. | | [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7417 TRTEMP_2.6_MANUAL_BSP/QEMU_core-image-sato-sdk_MPC8315e-rdb] | ||
|- | |- | ||
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7418 TRTEMP_2. | | [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7418 TRTEMP_2.6_AUTO_BSP/QEMU_core-image-lsb-sdk_ANYQEMU] | ||
|- | |- | ||
| Meta-Yocto | | Meta-Yocto | ||
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7419 TRTEMP_2. | | [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7419 TRTEMP_2.6_MANUAL_Meta-yocto] | ||
|- | |- | ||
| OE-CORE | | OE-CORE | ||
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7420 TRTEMP_2. | | [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7420 TRTEMP_2.6_AUTO_OE-Core] | ||
|- | |- | ||
| Build-Appliance | | Build-Appliance | ||
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7421 TRTEMP_2. | | [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7421 TRTEMP_2.6_MANUAL_Build-Appliance] | ||
|- | |- | ||
| Runtime | | Runtime | ||
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7422 TRTEMP_2. | | [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7422 TRTEMP_2.6_AUTO_Runtime] | ||
|- | |- | ||
| rowspan="2" | Toaster | | rowspan="2" | Toaster | ||
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7423 TRTEMP_2. | | [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7423 TRTEMP_2.6_AUTO_Toaster_UI] | ||
|- | |- | ||
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7424 TRTEMP_2. | | [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7424 TRTEMP_2.6_MANUAL_Toaster_UI-managed-mode] | ||
|- | |- | ||
| rowspan="2" | ADT | | rowspan="2" | ADT | ||
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7425 TRTEMP_2. | | [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7425 TRTEMP_2.6_MANUAL_ADT] | ||
|- | |- | ||
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7426 TRTEMP_2. | | [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7426 TRTEMP_2.6_AUTO_ADT] | ||
|- | |- | ||
| rowspan="2" | CROPS | | rowspan="2" | CROPS | ||
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7430 TRTEMP_2. | | [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7430 TRTEMP_2.6_AUTO_CROPS_Poky] | ||
|- | |- | ||
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7431 TRTEMP_2. | | [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7431 TRTEMP_2.6_MANUAL_CROPS_eSDK] | ||
|- | |- | ||
| Eclipse | | Eclipse | ||
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7432 TRTEMP_2. | | [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7432 TRTEMP_2.6_MANUAL_Eclipse] | ||
|- | |- | ||
| rowspan="2" | eSDK | | rowspan="2" | eSDK | ||
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7433 TRTEMP_2. | | [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7433 TRTEMP_2.6_AUTO_eSDK_sdkext] | ||
|- | |- | ||
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7434 TRTEMP_2. | | [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7434 TRTEMP_2.6_AUTO_eSDK_oeselftest] | ||
|- | |- | ||
| Kernel | | Kernel | ||
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7435 TRTEMP_2. | | [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7435 TRTEMP_2.6_MANUAL_Kernel] | ||
|- | |- | ||
| Package Management | | Package Management | ||
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7489 TRTEMP_2. | | [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7489 TRTEMP_2.6_MANUAL_Package-Management-Updates] | ||
|} | |} | ||
Line 668: | Line 660: | ||
The complete List of QA Implementations for YP 2. | The complete List of QA Implementations for YP 2.6 can be found at [https://wiki.yoctoproject.org/wiki/2.6_QA_OWNED_BUGS Features Implementation by QA Team] | ||
= Features Not to be tested = | = Features Not to be tested = | ||
Line 681: | Line 673: | ||
= Pass / Fail Criteria = | = Pass / Fail Criteria = | ||
This criteria is defined in [[Yocto_Project_v2. | This criteria is defined in [[Yocto_Project_v2.6_Status#Yocto_Project_v2.6_Release_Criteria]] | ||
= Exit Criteria = | = Exit Criteria = | ||
Line 690: | Line 682: | ||
* Live Status [[2. | * Live Status [[2.6_QA_Status]] | ||
* Final execution Report [[#Execution History]] | * Final execution Report [[#Execution History]] | ||
* Mail sent to the mailing list | * Mail sent to the mailing list | ||
Line 699: | Line 691: | ||
= Schedule = | = Schedule = | ||
== M1 ( | == M1 (Apr 30, 2018 to Jun 22, 2018) == | ||
=== M1 Feature/Tasks Development / Cutoff is Jun. | === M1 Feature/Tasks Development / Cutoff is Jun. 11, 2017 === | ||
{{#bugzilla: | {{#bugzilla: | ||
|columns=id,from,to,estimated,summary,severity,priority,whiteboard,status | |columns=id,from,to,estimated,summary,severity,priority,whiteboard,status | ||
|total=estimated | |total=estimated | ||
|milestone=2. | |milestone=2.6 M1, 2.6 M2 | ||
|severity=enhancement | |severity=enhancement | ||
|sort=to,priority | |sort=to,priority | ||
}} | }} | ||
Line 716: | Line 707: | ||
|columns=id,from,to,estimated,summary,severity,priority,whiteboard,status | |columns=id,from,to,estimated,summary,severity,priority,whiteboard,status | ||
|total=estimated | |total=estimated | ||
|milestone=2. | |milestone=2.6 M1, 2.6 M2 | ||
|severity=!enhancement | |severity=!enhancement | ||
|sort=to,priority | |sort=to,priority | ||
}} | }} | ||
The planed schedule for 2. | The planed schedule for 2.6 can be found at [[Yocto_2.6_Schedule]] | ||
== Live Schedule & Execution History == | == Live Schedule & Execution History == | ||
Line 735: | Line 725: | ||
|- bgcolor="93DB70" | |- bgcolor="93DB70" | ||
| M1 rc1 | | M1 rc1 | ||
| | | 6/19/2018 | ||
| | | 7/2/2018 | ||
| DONE | |||
| https://wiki.yoctoproject.org/wiki/WW27_-_2018-07-02_-_Full_Test_Cycle_2.6_M1_rc1 | |||
|- bgcolor="93DB70" | |||
| M1 rc1 meta-intel 10.0 | |||
| 7/13/2018 | |||
| 7/24/2018 | |||
| DONE | |||
| https://wiki.yoctoproject.org/wiki/WW29_-_2018-07-19-_Full_Test_Cycle_-_2.6_M1_rc1_-_meta-intel | |||
|- bgcolor="93DB70" | |||
| M2 rc1 | |||
| 7/19/2018 | |||
| 8/7/2018 | |||
| DONE | | DONE | ||
| https://wiki.yoctoproject.org/wiki/ | | https://wiki.yoctoproject.org/wiki/WW32_-_2018-08-07_-_Full_Test_Cycle_2.6_M2_rc1 | ||
|- bgcolor="93DB70" | |- bgcolor="93DB70" | ||
| | | M2 rc1 meta-intel 10.0 | ||
| | | 8/27/2018 | ||
| | | 9/4/2018 | ||
| DONE | | DONE | ||
| https://wiki.yoctoproject.org/wiki/ | | https://wiki.yoctoproject.org/wiki/WW36_-_2018-09-04-_Full_Test_Cycle_-_2.6_M2_rc1_-_meta-intel_10.0 | ||
|- bgcolor="93DB70" | |- bgcolor="93DB70" | ||
| | | M3 rc1 | ||
| | | 9/25/2018 | ||
| | | 10/09/2018 | ||
| DONE | | DONE | ||
| https://wiki.yoctoproject.org/wiki/ | |https://wiki.yoctoproject.org/wiki/WW41_-_2018-10-03_-_Full_Test_Cycle_2.6_M3_RC1 | ||
|- bgcolor="93DB70" | |- bgcolor="93DB70" | ||
| | | M4 rc1 | ||
| | | 10/30/2018 | ||
| | | 11/09/2018 | ||
| DONE | | DONE | ||
| https://wiki.yoctoproject.org/wiki/ | | https://wiki.yoctoproject.org/wiki/WW44_-_2018-10-30_-_Full_Test_Cycle_2.6_M4_RC1 | ||
|- bgcolor="93DB70" | |- bgcolor="93DB70" | ||
| | | M4 rc2 meta-intel 10.0 | ||
| | | 12/14/2018 | ||
| | | 12/21/2018 | ||
| DONE | | DONE | ||
| https://wiki.yoctoproject.org/wiki/ | | https://wiki.yoctoproject.org/wiki/WW51_-_2018-12-21-_Full_Test_Cycle_-_2.6_rc2_-_meta-intel_10.0 | ||
|- bgcolor="93DB70" | |- bgcolor="93DB70" | ||
| | | 2.6.1 rc1 | ||
| | | 01/09/2019 | ||
| | | 01/22/2019 | ||
| DONE | | DONE | ||
| https://wiki.yoctoproject.org/wiki/ | | https://wiki.yoctoproject.org/wiki/WW02_-_2019-01-09_-_Full_Test_Cycle_2.6.1_RC1 | ||
|- bgcolor="93DB70" | |- bgcolor="93DB70" | ||
| | | 2.6.1 rc1 meta-intel 10.1 | ||
| | | 02/12/2019 | ||
| | | 02/18/2019 | ||
| DONE | | DONE | ||
| https://wiki.yoctoproject.org/wiki/ | | https://wiki.yoctoproject.org/wiki/WW07_-_2019-02-12-_Full_Test_Cycle_-_2.6.1_rc1_-_meta-intel_10.1 | ||
|- bgcolor="grey" | |- bgcolor="grey" | ||
|} | |} |
Latest revision as of 06:23, 8 April 2019
Reversion history
Version | Modifier | Comments |
---|---|---|
1.0 | Yeoh Ee Peng | First draft |
References
Documents that support this test plan
Introduction
This is the the overall test plan for version 2.6 of the Yocto Project, for all the generic information this document will be pointing to QA Master Test Plan
Objectives
This test plan for Yocto Project 2.6 release supports the following objectives:
- To define the tools to be used throughout the testing process.
- To communicate to the responsible parties the items to be tested, set expectations around schedule, and define environmental needs.
- To define how the tests will be conducted.
- Release Criteria
- Risks and contingencies
QA_Master_Test_Plan#Objectives_and_Tasks
Risk Analysis & Contingency Plan
This is the document for Risk Analysis https://docs.google.com/spreadsheets/d/1K-MZYo5sfabwI2UozGp6Uuh2sw5y309bXkyEZRPEEgk/edit#gid=0
Test Strategy
Is followed the general strategy defined on QA_Master_Test_Plan#Testing_Strategy and below is showed specific information applicable only for 2.6 Release
Bugs Tracking
- Bugs that need to be implemented by QA Team
- Bugs that need to be verified by QA Team
- Old Bugs that need to be Verified
Test Items
The Yocto Project QA team verifies each release against recent versions of the most popular Linux distributions that provide stable releases.
* centos 7 * Debian 9 * Fedora 26 * Fedora 27 * OpenSuse42.3 * ubuntu 16.04 * ubuntu 17.10
Test Cases
Test Matrix
The components to be tested for version 2.6 of the Yocto Project against the different milestones are included in this section:
Code of Colors |
---|
Active
Tracked on Testpoia |
Active
Non Tracked on Testopia |
In Design
Will be tracked on Tesopia |
Inactive |
ADT
Component | Environment | Auto | Manual | Owner |
---|---|---|---|---|
ADT | Fedora 25 i686 | Y | Y | PG |
Ubuntu 16.04 x86_64 | Y | Y | PG |
Eclipse Plugin
Component | Environment | Auto | Manual | Owner |
---|---|---|---|---|
Neon + Any Distro | Y | PG | ||
Oxygen + Any Distro | Y | PG |
Kernel
Component | Environment | Auto | Manual | Owner |
---|---|---|---|---|
Kernel | Any Distro | Y | PG |
Meta-Yocto
Component | Environment | Auto | Manual | Owner |
---|---|---|---|---|
Meta-Yocto | Any Distro | Y | PG |
Toaster
Component | Environment | Auto | Manual | Owner |
---|---|---|---|---|
Toaster | Any Distro | Y | PG |
eSDK
Component | Environment | Auto | Manual | Owner |
---|---|---|---|---|
eSDK | Any Distro + runtime | Y | PG | |
Any Distro + selftest | Y | PG |
Selftest
Component | Environment | Auto | Manual | Owner |
---|---|---|---|---|
OE-CORE | Ubuntu 16.04 | Y | PG | |
Ubuntu 17.10 | Y | PG | ||
OpenSuse 42.3 | Y | PG | ||
Debian 9 | Y | PG | ||
Centos 7 | Y | PG | ||
Fedora 26 | Y | PG | ||
Fedora 27 | Y | PG |
Bitbake
Component | Environment | Auto | Manual | Owner |
---|---|---|---|---|
Bitbake | Any Distro | Y | PG |
Build - Apliance
Component | Environment | Auto | Manual | Owner |
---|---|---|---|---|
Build - Apliance | VM - Any Distro | Y | PG |
Package Manager
Component | Environment | Auto | Manual | Owner |
---|---|---|---|---|
Package Manager | Any Distro | Y | PG |
Performance
Component | Environment | Auto | Manual | Owner |
---|---|---|---|---|
Performance | Centos 7 | Y | PG | |
Ubuntu 16.04 | Y | PG |
BSP/QEMU
Component | Arch | HW | Environment | Image | Auto | Manual | pTest | Compliance |
---|---|---|---|---|---|---|---|---|
BSP | x86 | MinwTurbot 32bit | genericx86 | core-image-sato-sdk | Y | |||
core-image-lsb-sdk | Y | |||||||
MinwTurbot 64bit | genericx86-64 | core-image-sato-sdk | Y | Y | ||||
core-image-lsb-sdk | Y | |||||||
NUC 6 | genericx86-64 | core-image-sato-sdk | Y | Y | ||||
core-image-lsb-sdk | Y | Y | ||||||
NUC 7 | genericx86-64 | core-image-sato-sdk | Y | Y | ||||
core-image-lsb-sdk | Y | |||||||
QEMU | qemux86 | core-image-sato-sdk | Y | Y | ||||
core-image-lsb-sdk | Y | |||||||
qemux86-64 | core-image-sato-sdk | Y | Y | |||||
core-image-lsb-sdk | Y | |||||||
non x86 | EdgeRouter | EdgeRouter | core-image-sato-sdk | Y | Y | |||
MPC8315e-rdb | MPC8315e-rdb | core-image-sato-sdk | Y | Y | ||||
Beaglebone Black | Beaglebone | core-image-sato-sdk | Y | Y | ||||
QEMU | qemuarm | core-image-sato-sdk | Y | Y | ||||
qemuarm-64 | core-image-sato-sdk | Y | Y | |||||
qemuppc | core-image-sato-sdk | Y | Y | |||||
qemumips | core-image-sato-sdk | Y | Y | |||||
qemumips64 | core-image-sato-sdk | Y | Y | |||||
Meta-Intel | MinwTurbot 32bit | core2-32 | core-image-sato-sdk | Y | ||||
core-image-lsb-sdk | Y | |||||||
MinwTurbot 64bit | corei7-64 | core-image-sato-sdk | Y | Y | ||||
core-image-lsb-sdk | Y | |||||||
NUC 6 | corei7-64 | core-image-sato-sdk | Y | |||||
core-image-lsb-sdk | Y | |||||||
NUC 7 | corei7-64 | core-image-sato-sdk | Y | Y | ||||
core-image-lsb-sdk | Y | |||||||
poky-tiny | Y |
All component to be tested on 2.6 are in #Test Items a detailed list of features per component can be found in Test Plan of the components QA_Master_Test_Plan#Component
Testopia Templates
Features pending to be tested
Here is a list of areas that are not yet covered and need some work, implement test cases and add it to a component that belongs or create a new test plan for new component.
ADD FEATURES !!!!!!!!!!
The complete List of QA Implementations for YP 2.6 can be found at Features Implementation by QA Team
Features Not to be tested
Approach
Tests will be conducted per the documented test cases stored in Testopia. The test manager will create test runs for each tester. The tester will execute the tests in Testopia and mark each case as Pass / Fail / Skip /Blocked. The tester should leave notes on actual results and any other relevant details when possible.
Once complete, the test manager should review the test run reports in Testopia and report back to the team by mail and in the wiki as a test report.
Pass / Fail Criteria
This criteria is defined in Yocto_Project_v2.6_Status#Yocto_Project_v2.6_Release_Criteria
Exit Criteria
As defined in QA_Master_Test_Plan#Entry and Exit Criteria
Test Deliverables
- Live Status 2.6_QA_Status
- Final execution Report #Execution History
- Mail sent to the mailing list
Test Environment
There are different environments and tools for every type of testing refer to QA_Master_Test_Plan#Test_Areas for more detailed information.
Schedule
M1 (Apr 30, 2018 to Jun 22, 2018)
M1 Feature/Tasks Development / Cutoff is Jun. 11, 2017
ID | Requester | Assignee | E | Summary (4 tasks) ⇒ | Severity | P | Whiteboard | Status |
---|---|---|---|---|---|---|---|---|
10681 | Richard Purdie | Alejandro Hernandez | 10 | Support inter-multi-configuration dependencies | enhancement | Medium+ | RESOLVED | |
12727 | Ross Burton | Alexander Kanavin | Drop wireless-tools and use iw | enhancement | Medium+ | RESOLVED | ||
12689 | Anuj Mittal | Anuj Mittal | 1 | wic should be able to use initrd value for grub | enhancement | High | RESOLVED | |
10744 | Jose Perez C | Richard Purdie | 5 | Update test case of smartpm/createrepo under meta/lib/oeqa/runtime | enhancement | Medium+ | RESOLVED | |
16 |
M1 Bugs
ID | Requester | Assignee | E | Summary (35 tasks) ⇒ | Severity | P | Whiteboard | Status |
---|---|---|---|---|---|---|---|---|
12224 | brian avery | Anuj Mittal | 2 | wic image created for qemux86-64 stalls on undefined video mode number: 318 | normal | Medium | RESOLVED | |
12767 | Robert Berger | Anuj Mittal | bitbake-layers create-layer | normal | Medium | Backport 2.5.1 | RESOLVED | |
12773 | pankaj yadav | Anuj Mittal | [Meta-Yocto :2.4.3RC2] QA_WARN and QA_ERROR Mechanism checking Failed | normal | Medium+ | RESOLVED | ||
12666 | pravin | Anuj Mittal | 1 | [2.5 M3 RC1] BSP-QEMU:core-image-sato-sdk-genericx86-64.hddimg unable to connect to console with serial on Minnowboard Turbot | normal | Medium+ | RESOLVED | |
12776 | Leonardo Sandoval Gonzalez | Anuj Mittal | 1 | hddimg installer hangs on sumo | normal | Medium+ | RESOLVED | |
12832 | pravin | Anuj Mittal | 2 | [ 2.6 M1 rc1 ][BSP][Test case 267]: audio and video does not play in media player[Mturbot x86-64 and NUC7] | normal | Medium+ | RESOLVED | |
12803 | pankaj yadav | Apoorv | [Yocto-2.6_M1.RC1] Building a kernel module on target image fails | normal | Medium+ | RESOLVED | ||
12805 | pankaj yadav | Apoorv | [Test Case 1058] bitbake can use alternate download methods than git | normal | Medium+ | RESOLVED | ||
12742 | Ross Burton | Paul Eggleton | buildhistory doesn't like files with whitespace | normal | Medium+ | RESOLVED | ||
12757 | He Zhe | Bruce Ashfield | kernel do_configure fails to use tools in recipe-sysroot-native | critical | Medium+ | RESOLVED | ||
12794 | Robert Berger | Chin Huat Ang | Eclipse Project SDK plugin Oxygen/2.5.0 can not be installed | normal | Medium+ | RESOLVED | ||
12796 | Scott Rifenbark | Chin Huat Ang | Eclipse Oxygen section needs work-around - it is broke | normal | Undecided | RESOLVED | ||
12741 | Ross Burton | Hongxu Jia | distcc: upgrade to 3.3 | normal | Medium+ | RESOLVED | ||
12808 | Robert Berger | Joshua Watt | bitbake-layers create-layer | normal | Medium+ | RESOLVED | ||
12836 | Manish Das | Manish Das | [2.6 M1 RC1] busybox ptest failed | normal | Medium+ | RESOLVED | ||
12656 | Juro Bystricky | Maxin B. John | at-spi2-core_2.28.0.bb:do_compile error | normal | Medium+ | RESOLVED | ||
12708 | Juro Bystricky | Mingli Yu | (reproducibility) boost-dbg contains build host references | normal | Medium+ | reproducibility | RESOLVED | |
12711 | Aaron Chan | Mohamad Noor Alim | [2.5 M3 rc1][nightly-arm64][fedora 27][autobuilder2] - Event handler runqueue_stats failed | major | Medium+ | RESOLVED | ||
12736 | Mohamad Noor Alim | Mohamad Noor Alim | 2.5 rc1: CentOS 7 oe-selftest fail at runtime_test.TestImage.test_testimage_dnf | normal | Medium+ | RESOLVED | ||
12792 | Olof Johansson | Olof Johansson | RPM leaks pre-/postinstall scripts to /var/tmp | normal | Medium | RESOLVED | ||
12768 | Robert Berger | Robert Berger | Please place them into pkg_postinst_ontarget_${PN} () | normal | Medium | RESOLVED | ||
12729 | Chen Qi | Chen Qi | 1 | testsdkext takes too long time to finish | normal | High | RESOLVED | |
12835 | Chen Qi | Chen Qi | 'core-image-weston + systemd' fails the 'parselogs' runtime oeqa test case | normal | Undecided | RESOLVED | ||
12866 | Mohamad Noor Alim | Richard Purdie | [2.6 M2 rc1] wic test_fix_size failed | normal | Medium | RESOLVED | ||
12739 | Matt Hoosier | Richard Purdie | Fetcher for Git submodules ('gitsm') bypasses BB_NO_NETWORK checks | normal | Medium+ | Backport to YP 2.4 and 2.5 | RESOLVED | |
12765 | Hains | Richard Purdie | Race condition do_package_write_ipk fails due to empty package-split dir. | normal | Medium+ | RESOLVED | ||
12828 | Maxin B. John | Richard Purdie | mesa build fails on edgerouter and beaglebone-yocto | normal | Medium+ | RESOLVED | ||
12756 | Armin Kuster | Ross Burton | [ross/MUT] nightly-oecore failed | normal | Medium+ | RESOLVED | ||
12799 | Armin Kuster | Ross Burton | Occasional mesa build race | normal | Medium+ | RESOLVED | ||
12815 | Maxin B. John | Ross Burton | libsdl2 build failed | normal | Medium+ | RESOLVED | ||
12475 | Amber Elliot | Stephen K Jolley | 1 | Bug tracking BKMS should be documented | normal | Medium+ | 10 May 2018: IN PROGRESS REVIEW | RESOLVED |
12517 | Stephano Cetola | Scott Rifenbark | 1 | Update documentation on gen-tapdevs script | normal | Medium+ | 15 May 2018: RESOLVED | RESOLVED |
11567 | Phlip Balister | Stephano Cetola | 1 | Trying to create an new layer with the yocto-layer script fails. | normal | Medium+ | May 15 2018: RESOLVED | RESOLVED |
12772 | Scott Branden | Tim Orling | bitbake after devtool doesn't work properly with qemu | normal | Medium+ | RESOLVED | ||
12778 | Uttam K | Uttam K | Unable to build yocto poky project | major | Medium+ | RESOLVED | ||
10 |
The planed schedule for 2.6 can be found at Yocto_2.6_Schedule