Yocto Project 2.8 Release Test Plan: Difference between revisions
(47 intermediate revisions by 2 users not shown) | |||
Line 38: | Line 38: | ||
|- | |- | ||
|} | |} | ||
= Introduction = | = Introduction = | ||
Line 60: | Line 57: | ||
= Risk Analysis & Contingency Plan = | = Risk Analysis & Contingency Plan = | ||
QA identified risks for this release are mentioned here. | |||
{| class="wikitable" | {| class="wikitable" | ||
! style="text-align: center; font-weight: bold;" | | ! style="text-align: center; font-weight: bold;" | Module | ||
! style="text-align: center; font-weight: bold;" | | ! style="text-align: center; font-weight: bold;" | Risk Description | ||
! style="text-align: center; font-weight: bold;" | | ! style="text-align: center; font-weight: bold;" | Risk Level | ||
! style="text-align: center; font-weight: bold;" | | ! style="text-align: center; font-weight: bold;" | Contingency plan | ||
! style="text-align: center; font-weight: bold;" | Owner | ! style="text-align: center; font-weight: bold;" | Owner | ||
|- | |- | ||
| | | style="text-align: center;" | BSP | ||
| | | 54 manual test cases | ||
| | | medium | ||
| test cases to be automated | |||
| Intel | |||
|- | |||
| style="text-align: center;" | Build Appliance | |||
| 3 manual test cases, not executed by any QA team | |||
| medium | |||
| test cases to be automated | |||
| | | | ||
|- | |- | ||
| | | style="text-align: center;" | Compliance | ||
| | | 4 manual test cases, not executed by any QA team | ||
| medium | |||
| test cases to be automated | |||
| | | | ||
|- | |- | ||
| | | style="text-align: center;" | Kernel dev | ||
| 9 manual test cases, not executed by any QA team | |||
| medium | |||
| test cases to be automated | |||
| | | | ||
|- | |- | ||
| | | style="text-align: center;" | OE-Core | ||
| | | 6 manual test cases | ||
| medium | |||
| test cases to be automated | |||
| Intel | |||
|- | |||
| style="text-align: center;" | SDK | |||
| 1 manual test case, not executed by any QA team | |||
| medium | |||
| test case to be automated | |||
| | | | ||
|- | |- | ||
| | | style="text-align: center;" | Toaster | ||
| | | 93 manual test cases, not executed by any QA team | ||
| medium | |||
| test cases to be automated | |||
| | | | ||
|} | |} | ||
= | = Test Strategy = | ||
Is followed the general strategy defined on [[QA_Master_Test_Plan#Testing_Strategy]] and below is showed specific information applicable only for 2.8 Release | |||
== | = Test Items = | ||
The Yocto Project QA team verifies each release against recent versions of the most popular Linux distributions that provide stable releases. | |||
* centos 7 | |||
* Debian 8 | |||
* Debian 9 | |||
* Fedora 28 | |||
* Fedora 29 | |||
* Fedora 30 | |||
* OpenSuse 150 | |||
* OpenSuse 151 | |||
* OpenSuse 423 | |||
* OpenSuse Tumbleweed | |||
* ubuntu 16.04 | |||
* ubuntu 18.04 | |||
= Test Cases = | |||
* http://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/meta/lib/oeqa/runtime | |||
* http://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/meta/lib/oeqa/selftest | |||
* http://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/meta/lib/oeqa/manual | |||
* http://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/meta/lib/oeqa/sdk | |||
* http://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/meta/lib/oeqa/sdkext | |||
== OE-CORE == | |||
{| class="wikitable" | {| class="wikitable" | ||
Line 225: | Line 147: | ||
| rowspan="8" style="text-align: center;" | OE-CORE | | rowspan="8" style="text-align: center;" | OE-CORE | ||
| Ubuntu 16.04 | | Ubuntu 16.04 | ||
| | | | ||
| Y | | Y | ||
| PG | | PG | ||
|- | |- | ||
|} | |} | ||
Line 341: | Line 165: | ||
! pTest | ! pTest | ||
! Compliance | ! Compliance | ||
! ltp | |||
|- | |- | ||
| rowspan="30" style="text-align: center;" | BSP | | rowspan="30" style="text-align: center;" | BSP | ||
Line 351: | Line 176: | ||
| | | | ||
| | | | ||
| | |||
|- | |- | ||
| core-image-lsb-sdk | | core-image-lsb-sdk | ||
Line 357: | Line 183: | ||
| | | | ||
| | | | ||
| | |||
|- | |- | ||
| rowspan="2" style="text-align: center;" | | | rowspan="2" style="text-align: center;" | Coffee Lake | ||
| rowspan="2" style="text-align: center;" | genericx86-64 | | rowspan="2" style="text-align: center;" | genericx86-64 | ||
| core-image-sato-sdk | | core-image-sato-sdk | ||
Line 365: | Line 192: | ||
| | | | ||
| | | | ||
| | |||
|- | |- | ||
| core-image-lsb-sdk | | core-image-lsb-sdk | ||
Line 371: | Line 199: | ||
| | | | ||
| | | | ||
| | |||
|- | |- | ||
| rowspan="2" style="text-align: center;" | NUC 6 | | rowspan="2" style="text-align: center;" | NUC 6 | ||
Line 379: | Line 208: | ||
| Y | | Y | ||
| | | | ||
| Y | |||
|- | |- | ||
| core-image-lsb-sdk | | core-image-lsb-sdk | ||
Line 385: | Line 215: | ||
| | | | ||
| Y | | Y | ||
| | |||
|- | |- | ||
| rowspan="2" style="text-align: center;" | NUC 7 | | rowspan="2" style="text-align: center;" | NUC 7 | ||
Line 393: | Line 224: | ||
| | | | ||
| | | | ||
| | |||
|- | |- | ||
| core-image-lsb-sdk | | core-image-lsb-sdk | ||
Line 399: | Line 231: | ||
| | | | ||
| | | | ||
| | |||
|- | |- | ||
| rowspan="4" style="text-align: center;" | QEMU | | rowspan="4" style="text-align: center;" | QEMU | ||
Line 407: | Line 240: | ||
| | | | ||
| | | | ||
| | |||
|- | |- | ||
| core-image-lsb-sdk | | core-image-lsb-sdk | ||
Line 413: | Line 247: | ||
| | | | ||
| | | | ||
| | |||
|- | |- | ||
| rowspan="2" style="text-align: center;" | qemux86-64 | | rowspan="2" style="text-align: center;" | qemux86-64 | ||
Line 420: | Line 255: | ||
| | | | ||
| | | | ||
| | |||
|- | |- | ||
| core-image-lsb-sdk | | core-image-lsb-sdk | ||
Line 426: | Line 262: | ||
| | | | ||
| | | | ||
| | |||
|- | |- | ||
| rowspan="8" style="text-align: center;" | non x86 | | rowspan="8" style="text-align: center;" | non x86 | ||
Line 434: | Line 271: | ||
| Y | | Y | ||
| | | | ||
| | | Y | ||
| | |||
|- | |- | ||
| style="text-align: center;" | MPC8315e-rdb | | style="text-align: center;" | MPC8315e-rdb | ||
Line 442: | Line 280: | ||
| Y | | Y | ||
| | | | ||
| | | Y | ||
| | |||
|- | |- | ||
| style="text-align: center;" | Beaglebone Black | | style="text-align: center;" | Beaglebone Black | ||
Line 450: | Line 289: | ||
| Y | | Y | ||
| | | | ||
| | | Y | ||
| | |||
|- | |- | ||
| rowspan="5" style="text-align: center;" | QEMU | | rowspan="5" style="text-align: center;" | QEMU | ||
Line 459: | Line 299: | ||
| | | | ||
| | | | ||
| | |||
|- | |- | ||
| qemuarm-64 | | qemuarm-64 | ||
| core-image-sato-sdk | | core-image-sato-sdk | ||
| Y | |||
| Y | | Y | ||
| Y | | Y | ||
| | | | ||
| | | Y | ||
|- | |- | ||
| qemuppc | | qemuppc | ||
Line 473: | Line 315: | ||
| | | | ||
| | | | ||
| | |||
|- | |- | ||
| qemumips | | qemumips | ||
Line 480: | Line 323: | ||
| | | | ||
| | | | ||
| | |||
|- | |- | ||
| qemumips64 | | qemumips64 | ||
Line 487: | Line 331: | ||
| | | | ||
| | | | ||
| | |||
|- | |- | ||
| rowspan="10" style="text-align: center;" | Meta-Intel | | rowspan="10" style="text-align: center;" | Meta-Intel | ||
Line 496: | Line 341: | ||
| | | | ||
| | | | ||
| | |||
|- | |- | ||
| core-image-lsb-sdk | | core-image-lsb-sdk | ||
Line 502: | Line 348: | ||
| | | | ||
| | | | ||
| | |||
|- | |- | ||
| rowspan="2" style="text-align: center;" | | | rowspan="2" style="text-align: center;" | Coffee Lake | ||
| rowspan="2" style="text-align: center;" | corei7-64 | | rowspan="2" style="text-align: center;" | corei7-64 | ||
| core-image-sato-sdk | | core-image-sato-sdk | ||
Line 510: | Line 357: | ||
| | | | ||
| | | | ||
| | |||
|- | |- | ||
| core-image-lsb-sdk | | core-image-lsb-sdk | ||
Line 516: | Line 364: | ||
| | | | ||
| | | | ||
| | |||
|- | |- | ||
| rowspan="2" style="text-align: center;" | NUC 6 | | rowspan="2" style="text-align: center;" | NUC 6 | ||
Line 524: | Line 373: | ||
| | | | ||
| | | | ||
| | |||
|- | |- | ||
| core-image-lsb-sdk | | core-image-lsb-sdk | ||
Line 530: | Line 380: | ||
| | | | ||
| | | | ||
| | |||
|- | |- | ||
| rowspan="3" style="text-align: center;" | NUC 7 | | rowspan="3" style="text-align: center;" | NUC 7 | ||
Line 538: | Line 389: | ||
| | | | ||
| | | | ||
| | |||
|- | |- | ||
| core-image-lsb-sdk | | core-image-lsb-sdk | ||
Line 544: | Line 396: | ||
| | | | ||
| | | | ||
| | |||
|- | |- | ||
| poky-tiny | | poky-tiny | ||
Line 550: | Line 403: | ||
| | | | ||
| | | | ||
| | |||
|} | |} | ||
All component to be tested on 2. | All component to be tested on 2.8 are in [[#Test Items]] a detailed list of features per component can be found in Test Plan of the components [[QA_Master_Test_Plan#Component]] | ||
<!--= Features pending to be tested = | |||
= Features pending to be tested = | |||
TBD | TBD | ||
Line 563: | Line 416: | ||
The complete List of QA Implementations for YP 2.8 can be found at [https://wiki.yoctoproject.org/wiki/2.8_QA_OWNED_BUGS Features Implementation by QA Team] | The complete List of QA Implementations for YP 2.8 can be found at [https://wiki.yoctoproject.org/wiki/2.8_QA_OWNED_BUGS Features Implementation by QA Team] | ||
--> | |||
= Features Not to be tested = | <!--= Features Not to be tested = | ||
* TBD | * TBD | ||
--> | |||
=Approach= | =Approach= | ||
Testing will be done as per QA process defined in main QA wiki page at https://wiki.yoctoproject.org/wiki/QA#QA_Process. | |||
= Pass / Fail Criteria = | = Pass / Fail Criteria = | ||
Line 587: | Line 439: | ||
* Final Test report | * Final Test report | ||
* Mail sent to the mailing list | * Mail sent to the mailing list | ||
= Schedule = | = Schedule = | ||
The planed schedule for 2.8 can be found at [[Yocto_2.8_Schedule]] | The planed schedule for 2.8 can be found at [[Yocto_2.8_Schedule]] | ||
Line 630: | Line 457: | ||
| 06/14/2019 | | 06/14/2019 | ||
| DONE | | DONE | ||
| https:// | | https://git.yoctoproject.org/cgit/cgit.cgi/yocto-testresults-contrib/log/?h=intel-yocto-testresults | ||
|- bgcolor="93DB70" | |- bgcolor="93DB70" | ||
| M2 | | M2 | ||
| 7/26/2019 | |||
| | | 7/31/2019 | ||
| | | DONE | ||
| | | https://git.yoctoproject.org/cgit/cgit.cgi/yocto-testresults-contrib/tree/?h=zeus&id=d550d40646e1ca78a4b5d106e1b20831208b4c37 | ||
| | |||
|- bgcolor="93DB70" | |- bgcolor="93DB70" | ||
| | | M3 | ||
| | | 9/23/2019 | ||
| | | 9/26/2019 | ||
| DONE | | DONE | ||
| https:// | | https://git.yoctoproject.org/cgit/cgit.cgi/yocto-testresults-contrib/log/?h=intel-yocto-testresults | ||
|- bgcolor="93DB70" | |- bgcolor="93DB70" | ||
| | | 3.0 | ||
| | | 10/10/2019 | ||
| | | 10/15/2019 | ||
| DONE | | DONE | ||
| https:// | | https://git.yoctoproject.org/cgit/cgit.cgi/yocto-testresults-contrib/log/?h=intel-yocto-testresults | ||
|} | |} |
Latest revision as of 07:16, 7 January 2020
Reversion history
Version | Modifier | Comments |
---|---|---|
1.0 | Jain Sangeeta | First draft |
Introduction
This is the the overall test plan for version 2.8 of the Yocto Project, for all the generic information this document will be pointing to QA Master Test Plan
Objectives
This test plan for Yocto Project 2.8 release supports the following objectives:
- To define the tools to be used throughout the testing process.
- To communicate to the responsible parties the items to be tested, set expectations around schedule, and define environmental needs.
- To define how the tests will be conducted.
- Release Criteria
- Risks and contingencies
QA_Master_Test_Plan#Objectives_and_Tasks
Risk Analysis & Contingency Plan
QA identified risks for this release are mentioned here.
Module | Risk Description | Risk Level | Contingency plan | Owner |
---|---|---|---|---|
BSP | 54 manual test cases | medium | test cases to be automated | Intel |
Build Appliance | 3 manual test cases, not executed by any QA team | medium | test cases to be automated | |
Compliance | 4 manual test cases, not executed by any QA team | medium | test cases to be automated | |
Kernel dev | 9 manual test cases, not executed by any QA team | medium | test cases to be automated | |
OE-Core | 6 manual test cases | medium | test cases to be automated | Intel |
SDK | 1 manual test case, not executed by any QA team | medium | test case to be automated | |
Toaster | 93 manual test cases, not executed by any QA team | medium | test cases to be automated |
Test Strategy
Is followed the general strategy defined on QA_Master_Test_Plan#Testing_Strategy and below is showed specific information applicable only for 2.8 Release
Test Items
The Yocto Project QA team verifies each release against recent versions of the most popular Linux distributions that provide stable releases.
* centos 7 * Debian 8 * Debian 9 * Fedora 28 * Fedora 29 * Fedora 30 * OpenSuse 150 * OpenSuse 151 * OpenSuse 423 * OpenSuse Tumbleweed * ubuntu 16.04 * ubuntu 18.04
Test Cases
- http://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/meta/lib/oeqa/runtime
- http://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/meta/lib/oeqa/selftest
- http://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/meta/lib/oeqa/manual
- http://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/meta/lib/oeqa/sdk
- http://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/meta/lib/oeqa/sdkext
OE-CORE
Component | Environment | Auto | Manual | Owner |
---|---|---|---|---|
OE-CORE | Ubuntu 16.04 | Y | PG |
BSP/QEMU
Component | Arch | HW | Environment | Image | Auto | Manual | pTest | Compliance | ltp |
---|---|---|---|---|---|---|---|---|---|
BSP | x86 | MinwTurbot 32bit | genericx86 | core-image-sato-sdk | Y | ||||
core-image-lsb-sdk | Y | ||||||||
Coffee Lake | genericx86-64 | core-image-sato-sdk | Y | Y | |||||
core-image-lsb-sdk | Y | ||||||||
NUC 6 | genericx86-64 | core-image-sato-sdk | Y | Y | Y | ||||
core-image-lsb-sdk | Y | Y | |||||||
NUC 7 | genericx86-64 | core-image-sato-sdk | Y | Y | |||||
core-image-lsb-sdk | Y | ||||||||
QEMU | qemux86 | core-image-sato-sdk | Y | Y | |||||
core-image-lsb-sdk | Y | ||||||||
qemux86-64 | core-image-sato-sdk | Y | Y | ||||||
core-image-lsb-sdk | Y | ||||||||
non x86 | EdgeRouter | EdgeRouter | core-image-sato-sdk | Y | Y | Y | |||
MPC8315e-rdb | MPC8315e-rdb | core-image-sato-sdk | Y | Y | Y | ||||
Beaglebone Black | Beaglebone | core-image-sato-sdk | Y | Y | Y | ||||
QEMU | qemuarm | core-image-sato-sdk | Y | Y | |||||
qemuarm-64 | core-image-sato-sdk | Y | Y | Y | Y | ||||
qemuppc | core-image-sato-sdk | Y | Y | ||||||
qemumips | core-image-sato-sdk | Y | Y | ||||||
qemumips64 | core-image-sato-sdk | Y | Y | ||||||
Meta-Intel | MinwTurbot 32bit | core2-32 | core-image-sato-sdk | Y | |||||
core-image-lsb-sdk | Y | ||||||||
Coffee Lake | corei7-64 | core-image-sato-sdk | Y | Y | |||||
core-image-lsb-sdk | Y | ||||||||
NUC 6 | corei7-64 | core-image-sato-sdk | Y | ||||||
core-image-lsb-sdk | Y | ||||||||
NUC 7 | corei7-64 | core-image-sato-sdk | Y | Y | |||||
core-image-lsb-sdk | Y | ||||||||
poky-tiny | Y |
All component to be tested on 2.8 are in #Test Items a detailed list of features per component can be found in Test Plan of the components QA_Master_Test_Plan#Component
Approach
Testing will be done as per QA process defined in main QA wiki page at https://wiki.yoctoproject.org/wiki/QA#QA_Process.
Pass / Fail Criteria
This criteria is defined in Yocto_Project_v2.8_Status#Yocto_Project_v2.8_Release_Criteria
Exit Criteria
As defined in QA_Master_Test_Plan#Entry and Exit Criteria
Test Deliverables
- Final Test results
- Final Test report
- Mail sent to the mailing list
Schedule
The planed schedule for 2.8 can be found at Yocto_2.8_Schedule
Live Schedule & Execution History
MILESTONE | STARTED | FINISHED | STATUS | REPORT |
---|---|---|---|---|
M1 rc2 | 06/11/2019 | 06/14/2019 | DONE | https://git.yoctoproject.org/cgit/cgit.cgi/yocto-testresults-contrib/log/?h=intel-yocto-testresults |
M2 | 7/26/2019 | 7/31/2019 | DONE | https://git.yoctoproject.org/cgit/cgit.cgi/yocto-testresults-contrib/tree/?h=zeus&id=d550d40646e1ca78a4b5d106e1b20831208b4c37 |
M3 | 9/23/2019 | 9/26/2019 | DONE | https://git.yoctoproject.org/cgit/cgit.cgi/yocto-testresults-contrib/log/?h=intel-yocto-testresults |
3.0 | 10/10/2019 | 10/15/2019 | DONE | https://git.yoctoproject.org/cgit/cgit.cgi/yocto-testresults-contrib/log/?h=intel-yocto-testresults |