Yocto Project 2.8 Release Test Plan: Difference between revisions

From Yocto Project
Jump to navigationJump to search
(Created page with "<!-- Variables for Template --> <!-- ********************** --> {| {{#vardefine:ADT_MAUAL | 6539}} {{#vardefine:ADT_AUTO | 6540}} {{#vardefine:ANYQEMU_S...")
 
 
(67 intermediate revisions by 2 users not shown)
Line 35: Line 35:
! Version || Modifier || Comments
! Version || Modifier || Comments
|-
|-
| 1.0 || Yeoh Ee Peng ||  First draft
| 1.0 || Jain Sangeeta ||  First draft
|-
|-
|}
|}
=References=
Documents that support this test plan


= Introduction =
= Introduction =




This is the the overall test plan for version 2.7 of the {{ns:4}}, for all the generic information this document will be pointing to [[QA Master Test Plan]]
This is the the overall test plan for version 2.8 of the {{ns:4}}, for all the generic information this document will be pointing to [[QA Master Test Plan]]


= Objectives =
= Objectives =


This test plan for Yocto Project 2.7 release supports the following objectives:
This test plan for Yocto Project 2.8 release supports the following objectives:
*To define the tools to be used throughout the testing process.
*To define the tools to be used throughout the testing process.
*To communicate to the responsible parties the items to be tested, set expectations around schedule, and define environmental needs.
*To communicate to the responsible parties the items to be tested, set expectations around schedule, and define environmental needs.
Line 60: Line 57:
= Risk Analysis & Contingency Plan =
= Risk Analysis & Contingency Plan =


This is the document for Risk Analysis https://docs.google.com/spreadsheets/d/1K-MZYo5sfabwI2UozGp6Uuh2sw5y309bXkyEZRPEEgk/edit#gid=0
QA identified risks for this release are mentioned here.
 
= Test Strategy =
 
Is followed the general strategy defined on [[QA_Master_Test_Plan#Testing_Strategy]] and below is showed specific information applicable only for 2.7 Release
 
== Bugs Tracking ==
 
*Bugs that need to be implemented by QA Team
**[[2.7_QA_OWNED_BUGS | 2.7 QA Assigned Bugs]]
 
*Bugs that need to be verified by QA Team
**[[2.7_QA_Bugs_To_Verify]]
<!--
*Features to verify
**[[2.2 qa_owned features to verify]]
 
*Features to implement
**[[2.2 qa owned features]]
-->
 
*Old Bugs that need to be Verified
**[[Old resolved bugs and features]]
 
= Test Items =
The Yocto Project QA team verifies each release against recent versions of the most popular Linux distributions that provide stable releases.
* centos 7
* Debian 9
* Fedora 26
* Fedora 27
* OpenSuse42.3
* ubuntu 16.04
* ubuntu 17.10
 
= Test Cases =
 
*[[Yocto 2.7 Test Cases]]
 
 
= Test Matrix =
 
The components to be tested  for version 2.7  of the {{ns:4}} against the different milestones  are included in this section:


{| class="wikitable"
{| class="wikitable"
! style="text-align: center;" | Code of Colors
! style="text-align: center; font-weight: bold;" | Module
|-bgcolor="93DB70"
! style="text-align: center; font-weight: bold;" | Risk Description
| style="text-align: center;" | Active
! style="text-align: center; font-weight: bold;" | Risk Level
Tracked on Testpoia
! style="text-align: center; font-weight: bold;" | Contingency plan
|-bgcolor="67E6EC"
| style="text-align: center;" | Active
Non Tracked on Testopia
|-bgcolor="ffbf00"
| style="text-align: center;" | In Design
Will be tracked on Tesopia
|- bgcolor="gray"
| style="text-align: center;" | Inactive
|}
 
 
== ADT ==
 
{| class="wikitable"
! style="text-align: center; font-weight: bold;" | Component
! style="text-align: center; font-weight: bold;" | Environment
! style="text-align: center; font-weight: bold;" | Auto
! style="text-align: center; font-weight: bold;" | Manual
! style="text-align: center; font-weight: bold;" | Owner
! style="text-align: center; font-weight: bold;" | Owner
|-
|-
| rowspan="2" style="text-align: center;" | ADT
| style="text-align: center;" | BSP
| Fedora 25 i686
| 54 manual test cases
| Y
| medium
| Y
| test cases to be automated
| PG
| Intel
|-
| Ubuntu 16.04 x86_64
| Y
| Y
| PG
|}
 
== Eclipse Plugin ==
 
{| class="wikitable"
! style="text-align: center;" | Component
! style="text-align: center;" | Environment
! style="text-align: center;" | Auto
! style="text-align: center;" | Manual
! style="text-align: center;" | Owner
|-
|-
| Neon + Any Distro
| style="text-align: center;" | Build Appliance
| 3 manual test cases, not executed by any QA team
| medium
| test cases to be automated
|  
|  
|
| Y
| PG
|-
|-
| Oxygen + Any Distro
| style="text-align: center;" | Compliance
|
| 4 manual test cases, not executed by any QA team
| medium
| test cases to be automated
|  
|  
| Y
| PG
|}
== Kernel ==
{| class="wikitable"
! style="text-align: center;" | Component
! style="text-align: center;" | Environment
! style="text-align: center;" | Auto
! style="text-align: center;" | Manual
! style="text-align: center;" | Owner
|-
|-
| Kernel
| style="text-align: center;" | Kernel dev
| Any Distro
| 9 manual test cases, not executed by any QA team
| medium
| test cases to be automated
|  
|  
| Y
| PG
|}
== Meta-Yocto ==
{| class="wikitable"
! style="text-align: center;" | Component
! style="text-align: center;" | Environment
! style="text-align: center;" | Auto
! style="text-align: center;" | Manual
! style="text-align: center;" | Owner
|-
|-
| Meta-Yocto
| style="text-align: center;" | OE-Core
| Any Distro
| 6 manual test cases
|
| medium
| Y
| test cases to be automated
| PG
| Intel
|}
 
== Toaster ==
 
{| class="wikitable"
! style="text-align: center;" | Component
! style="text-align: center;" | Environment
! style="text-align: center;" | Auto
! style="text-align: center;" | Manual
! style="text-align: center;" | Owner
|-
| Toaster
| Any Distro
|
| Y
| PG
|}
 
== eSDK ==
 
{| class="wikitable"
! style="text-align: center;" | Component
! style="text-align: center;" | Environment
! style="text-align: center;" | Auto
! style="text-align: center;" | Manual
! style="text-align: center;" | Owner
|-
|-
| rowspan="2" style="text-align: center;" | eSDK
| style="text-align: center;" | SDK
| Any Distro + runtime
| 1 manual test case, not executed by any QA team
| Y
| medium
| test case to be automated
|  
|  
| PG
|-
|-
| Any Distro + selftest
| style="text-align: center;" | Toaster
| Y
| 93 manual test cases, not executed by any QA team
| medium
| test cases to be automated
|  
|  
| PG
|}
|}


== Selftest ==
= Test Strategy =
 
{| class="wikitable"
! style="text-align: center;" | Component
! style="text-align: center;" | Environment
! style="text-align: center;" | Auto
! style="text-align: center;" | Manual
! style="text-align: center;" | Owner
|-
| rowspan="8" style="text-align: center;" | OE-CORE
| Ubuntu 16.04
| Y
|
| PG
|-
| Ubuntu 17.10
| Y
|
| PG
|-
| OpenSuse 42.3
| Y
|
| PG
|-
| Debian 9
| Y
|
| PG
|-
| Centos 7
| Y
|
| PG
|-
| Fedora 26
| Y
|
| PG
|-
| Fedora 27
| Y
|
| PG
|}


== Bitbake ==
Is followed the general strategy defined on [[QA_Master_Test_Plan#Testing_Strategy]] and below is showed specific information applicable only for 2.8 Release


{| class="wikitable"
= Test Items =
! style="text-align: center;" | Component
The Yocto Project QA team verifies each release against recent versions of the most popular Linux distributions that provide stable releases.
! style="text-align: center;" | Environment
* centos 7
! style="text-align: center;" | Auto
* Debian 8
! style="text-align: center;" | Manual
* Debian 9
! style="text-align: center;" | Owner
* Fedora 28
|-
* Fedora 29
| Bitbake
* Fedora 30
| Any Distro
* OpenSuse 150
|
* OpenSuse 151
| Y
* OpenSuse 423
| PG
* OpenSuse Tumbleweed
|}
* ubuntu 16.04
* ubuntu 18.04


== Build - Apliance ==
= Test Cases =


{| class="wikitable"
* http://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/meta/lib/oeqa/runtime
! style="text-align: center;" | Component
* http://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/meta/lib/oeqa/selftest
! style="text-align: center;" | Environment
* http://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/meta/lib/oeqa/manual
! style="text-align: center;" | Auto
* http://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/meta/lib/oeqa/sdk
! style="text-align: center;" | Manual
* http://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/meta/lib/oeqa/sdkext
! style="text-align: center;" | Owner
|-
| Build - Apliance
| VM - Any Distro
|
| Y
| PG
|}


== Package Manager ==
== OE-CORE ==


{| class="wikitable"
{| class="wikitable"
Line 318: Line 145:
! style="text-align: center;" | Owner
! style="text-align: center;" | Owner
|-
|-
| Package Manager
| rowspan="8" style="text-align: center;" | OE-CORE
| Any Distro
| Ubuntu 16.04
| Y
|  
|  
| PG
|}
== Performance ==
{| class="wikitable"
! style="text-align: center;" | Component
! style="text-align: center;" | Environment
! style="text-align: center;" | Auto
! style="text-align: center;" | Manual
! style="text-align: center;" | Owner
|-
| rowspan="2" style="text-align: center;" | Performance
| Centos 7
| Y
| Y
|
| PG
| PG
|-
|-
| Ubuntu 16.04
| Y
|
| PG
|}
|}


Line 358: Line 165:
! pTest
! pTest
! Compliance
! Compliance
!  ltp
|-
|-
| rowspan="30" style="text-align: center;" | BSP
| rowspan="30" style="text-align: center;" | BSP
Line 368: Line 176:
|  
|  
|  
|  
|
|-
|-
| core-image-lsb-sdk
| core-image-lsb-sdk
Line 374: Line 183:
|  
|  
|  
|  
|
|-
|-
| rowspan="2" style="text-align: center;" | MinwTurbot 64bit
| rowspan="2" style="text-align: center;" | Coffee Lake
| rowspan="2" style="text-align: center;" | genericx86-64
| rowspan="2" style="text-align: center;" | genericx86-64
| core-image-sato-sdk
| core-image-sato-sdk
Line 382: Line 192:
|  
|  
|  
|  
|
|-
|-
| core-image-lsb-sdk
| core-image-lsb-sdk
Line 388: Line 199:
|  
|  
|  
|  
|
|-
|-
| rowspan="2" style="text-align: center;" | NUC 6
| rowspan="2" style="text-align: center;" | NUC 6
Line 396: Line 208:
| Y
| Y
|  
|  
| Y
|-
|-
| core-image-lsb-sdk
| core-image-lsb-sdk
Line 402: Line 215:
|  
|  
| Y
| Y
|
|-
|-
| rowspan="2" style="text-align: center;" | NUC 7
| rowspan="2" style="text-align: center;" | NUC 7
Line 410: Line 224:
|  
|  
|  
|  
|
|-
|-
| core-image-lsb-sdk
| core-image-lsb-sdk
Line 416: Line 231:
|  
|  
|  
|  
|
|-
|-
| rowspan="4" style="text-align: center;" | QEMU
| rowspan="4" style="text-align: center;" | QEMU
Line 424: Line 240:
|  
|  
|  
|  
|
|-
|-
| core-image-lsb-sdk
| core-image-lsb-sdk
Line 430: Line 247:
|  
|  
|  
|  
|
|-
|-
| rowspan="2" style="text-align: center;" | qemux86-64
| rowspan="2" style="text-align: center;" | qemux86-64
Line 437: Line 255:
|  
|  
|  
|  
|
|-
|-
| core-image-lsb-sdk
| core-image-lsb-sdk
Line 443: Line 262:
|  
|  
|  
|  
|
|-
|-
| rowspan="8" style="text-align: center;" | non x86
| rowspan="8" style="text-align: center;" | non x86
Line 451: Line 271:
| Y
| Y
|  
|  
|  
| Y
|
|-
|-
| style="text-align: center;" | MPC8315e-rdb
| style="text-align: center;" | MPC8315e-rdb
Line 459: Line 280:
| Y
| Y
|  
|  
|  
| Y
|
|-
|-
| style="text-align: center;" | Beaglebone Black
| style="text-align: center;" | Beaglebone Black
Line 467: Line 289:
| Y
| Y
|  
|  
|  
| Y
|
|-
|-
| rowspan="5" style="text-align: center;" | QEMU
| rowspan="5" style="text-align: center;" | QEMU
Line 476: Line 299:
|  
|  
|  
|  
|
|-
|-
| qemuarm-64
| qemuarm-64
| core-image-sato-sdk
| core-image-sato-sdk
| Y
| Y
| Y
| Y
| Y
|  
|  
|  
| Y
|-
|-
| qemuppc
| qemuppc
Line 490: Line 315:
|  
|  
|  
|  
|
|-
|-
| qemumips
| qemumips
Line 497: Line 323:
|  
|  
|  
|  
|
|-
|-
| qemumips64
| qemumips64
Line 504: Line 331:
|  
|  
|  
|  
|
|-
|-
| rowspan="10" style="text-align: center;" | Meta-Intel
| rowspan="10" style="text-align: center;" | Meta-Intel
Line 513: Line 341:
|  
|  
|  
|  
|
|-
|-
| core-image-lsb-sdk
| core-image-lsb-sdk
Line 519: Line 348:
|  
|  
|  
|  
|
|-
|-
| rowspan="2" style="text-align: center;" | MinwTurbot 64bit
| rowspan="2" style="text-align: center;" | Coffee Lake
| rowspan="2" style="text-align: center;" | corei7-64
| rowspan="2" style="text-align: center;" | corei7-64
| core-image-sato-sdk
| core-image-sato-sdk
Line 527: Line 357:
|  
|  
|  
|  
|
|-
|-
| core-image-lsb-sdk
| core-image-lsb-sdk
Line 533: Line 364:
|  
|  
|  
|  
|
|-
|-
| rowspan="2" style="text-align: center;" | NUC 6
| rowspan="2" style="text-align: center;" | NUC 6
Line 541: Line 373:
|  
|  
|  
|  
|
|-
|-
| core-image-lsb-sdk
| core-image-lsb-sdk
Line 547: Line 380:
|  
|  
|  
|  
|
|-
|-
| rowspan="3" style="text-align: center;" | NUC 7
| rowspan="3" style="text-align: center;" | NUC 7
Line 555: Line 389:
|  
|  
|  
|  
|
|-
|-
| core-image-lsb-sdk
| core-image-lsb-sdk
Line 561: Line 396:
|  
|  
|  
|  
|
|-
|-
| poky-tiny
| poky-tiny
Line 567: Line 403:
|  
|  
|  
|  
|
|}
|}


All component to be tested on 2.7 are in [[#Test Items]] a detailed list of features per component can be found in Test Plan of the components [[QA_Master_Test_Plan#Component]]
All component to be tested on 2.8 are in [[#Test Items]] a detailed list of features per component can be found in Test Plan of the components [[QA_Master_Test_Plan#Component]]
 
<!--= Features pending to be tested =
= Testopia Templates =
 
{| class="wikitable"
! style="font-weight: bold;" | COMPONENT
! style="font-weight: bold;" | TEMPLATE
|-
| BitBake
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7403 TRTEMP_2.6_MANUAL_BitBake]
|-
| rowspan="15" | BSP/QEMU
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7404 TRTEMP_2.6_MANUAL_BSP/QEMU_core-image-sato-sdk_Genericx86_MMAX32]
|-
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7405 TRTEMP_2.6_MANUAL_BSP/QEMU_core-image-sato-sdk_core2-32_MMAX32]
|-
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7406 TRTEMP_2.6_MANUAL_BSP/QEMU_core-image-sato-sdk_Genericx86-64_NUC]
|-
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7407 TRTEMP_2.6_MANUAL_BSP/QEMU_core-image-sato-sdk_corei7-64_NUC]
|-
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7408 TRTEMP_2.6_MANUAL_BSP/QEMU_core-image-sato-sdk_corei7-64_MMAX64]
|-
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7409 TRTEMP_2.6_AUTO_BSP/QEMU_core-image-sato-sdk_ANYBSP]
|-
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7410 TRTEMP_2.6_AUTO_BSP/QEMU_core-image-sato-sdk_wic_ANYBSP]
|-
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7411 TRTEMP_2.6_AUTO_BSP/QEMU_core-image-lsb-sdk_ANYBSP]
|-
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7412 TRTEMP_2.6_MANUAL_BSP/QEMU_core-image-sato-sdk_ANYQEMU]
|-
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7413 TRTEMP_2.6_AUTO_BSP/QEMU_core-image-sato-sdk_ANYQEMU]
|-
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7414 TRTEMP_2.6_AUTO_BSP/QEMU_core-image-sato-sdk_intel-quark]
|-
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7415 TRTEMP_2.6_MANUAL_BSP/QEMU_core-image-sato-sdk_Beaglebone-Black]
|-
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7416 TRTEMP_2.6_MANUAL_BSP/QEMU_core-image-sato-sdk_EdgeRouter]
|-
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7417 TRTEMP_2.6_MANUAL_BSP/QEMU_core-image-sato-sdk_MPC8315e-rdb]
|-
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7418 TRTEMP_2.6_AUTO_BSP/QEMU_core-image-lsb-sdk_ANYQEMU]
|-
| Meta-Yocto
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7419 TRTEMP_2.6_MANUAL_Meta-yocto]
|-
| OE-CORE
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7420 TRTEMP_2.6_AUTO_OE-Core]
|-
| Build-Appliance
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7421 TRTEMP_2.6_MANUAL_Build-Appliance]
|-
| Runtime
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7422 TRTEMP_2.6_AUTO_Runtime]
|-
| rowspan="2" | Toaster
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7423 TRTEMP_2.6_AUTO_Toaster_UI]
|-
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7424 TRTEMP_2.6_MANUAL_Toaster_UI-managed-mode]
|-
| rowspan="2" | ADT
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7425 TRTEMP_2.6_MANUAL_ADT]
|-
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7426 TRTEMP_2.6_AUTO_ADT]
|-
| rowspan="2" | CROPS
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7430 TRTEMP_2.6_AUTO_CROPS_Poky]
|-
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7431 TRTEMP_2.6_MANUAL_CROPS_eSDK]
|-
| Eclipse
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7432 TRTEMP_2.6_MANUAL_Eclipse]
|-
| rowspan="2" | eSDK
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7433 TRTEMP_2.6_AUTO_eSDK_sdkext]
|-
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7434 TRTEMP_2.6_AUTO_eSDK_oeselftest]
|-
| Kernel
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7435 TRTEMP_2.6_MANUAL_Kernel]
|-
| Package Management
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7489 TRTEMP_2.6_MANUAL_Package-Management-Updates]
|}
 
= Features pending to be tested =


TBD
Here is a list of areas that are not yet covered and need some work, implement test cases and add it to a component that belongs or create a new test plan for new component.
Here is a list of areas that are not yet covered and need some work, implement test cases and add it to a component that belongs or create a new test plan for new component.


Line 660: Line 415:




The complete List of QA Implementations for YP 2.7 can be found at [https://wiki.yoctoproject.org/wiki/2.7_QA_OWNED_BUGS Features Implementation by QA Team]
The complete List of QA Implementations for YP 2.8 can be found at [https://wiki.yoctoproject.org/wiki/2.8_QA_OWNED_BUGS Features Implementation by QA Team]
 
-->
= Features Not to be tested =
<!--= Features Not to be tested =
*
* TBD
-->


=Approach=
=Approach=
Tests will be conducted per the documented test cases stored in Testopia. The test manager will create test runs for each tester. The tester will execute the tests in Testopia and mark each case as Pass / Fail / Skip /Blocked.
The tester should leave notes on actual results and any other relevant details when possible.


Once complete, the test manager should review the test run reports in Testopia and report back to the team by mail and in the wiki as a test report.
Testing will be done as per QA process defined in main QA wiki page at https://wiki.yoctoproject.org/wiki/QA#QA_Process.


= Pass / Fail Criteria =
= Pass / Fail Criteria =


This criteria is defined in [[Yocto_Project_v2.7_Status#Yocto_Project_v2.7_Release_Criteria]]
This criteria is defined in [[Yocto_Project_v2.8_Status#Yocto_Project_v2.8_Release_Criteria]]


= Exit Criteria =
= Exit Criteria =
Line 682: Line 436:




* Live Status [[2.7_QA_Status]]
* Final Test results
* Final execution Report [[#Execution History]]
* Final Test report
* Mail sent to the mailing list
* Mail sent to the mailing list
= Test Environment =
There are different environments and tools for every type of testing refer to [[QA_Master_Test_Plan#Test_Areas]] for more detailed information.


= Schedule =
= Schedule =
== M1 (Apr 30, 2018 to Jun 22, 2018)  ==
=== M1 Feature/Tasks Development / Cutoff is Jun. 11, 2017 ===
{{#bugzilla:
  |columns=id,from,to,estimated,summary,severity,priority,whiteboard,status
  |total=estimated
  |milestone=2.7 M1, 2.7 M2
  |severity=enhancement
  |sort=to,priority
}}
=== M1 Bugs ===
{{#bugzilla:
  |columns=id,from,to,estimated,summary,severity,priority,whiteboard,status
  |total=estimated
  |milestone=2.7 M1, 2.7 M2
  |severity=!enhancement
  |sort=to,priority
}}


The planed schedule for 2.7 can be found at [[Yocto_2.7_Schedule]]
The planed schedule for 2.8 can be found at [[Yocto_2.8_Schedule]]


== Live Schedule & Execution History ==
== Live Schedule & Execution History ==
Line 724: Line 453:
! REPORT
! REPORT
|- bgcolor="93DB70"
|- bgcolor="93DB70"
| M1 rc1
| M1 rc2
| 12/19/2018
| 06/11/2019
| 1/2/2019
| 06/14/2019
| DONE
| DONE
| https://wiki.yoctoproject.org/wiki/WW01_-_2019-01-02_-_Full_Test_Cycle_2.7_M1_RC1
| https://git.yoctoproject.org/cgit/cgit.cgi/yocto-testresults-contrib/log/?h=intel-yocto-testresults
|
|- bgcolor="93DB70"
|- bgcolor="93DB70"
| M2 rc1
| M2  
| 01/28/2019
| 7/26/2019
| 02/08/2019
| 7/31/2019
| DONE
| DONE
| https://wiki.yoctoproject.org/wiki/WW01_-_2019-01-28_-_Full_Test_Cycle_2.7_M2_RC1
| https://git.yoctoproject.org/cgit/cgit.cgi/yocto-testresults-contrib/tree/?h=zeus&id=d550d40646e1ca78a4b5d106e1b20831208b4c37
|
|- bgcolor="93DB70"
|- bgcolor="93DB70"
| M2 rc2
| M3
| 02/14/2019
| 9/23/2019
| 02/25/2019
| 9/26/2019
| DONE
| DONE
| https://wiki.yoctoproject.org/wiki/WW07_-_2019-02-14_-_Full_Test_Cycle_2.7_M2_RC2
| https://git.yoctoproject.org/cgit/cgit.cgi/yocto-testresults-contrib/log/?h=intel-yocto-testresults
|
|- bgcolor="93DB70"
|- bgcolor="93DB70"
| M3 rc1
| 3.0
| 03/21/2019
| 10/10/2019
| 04/02/2019
| 10/15/2019
| DONE
| DONE
| https://wiki.yoctoproject.org/wiki/WW07_-_2019-04-02_-_Full_Test_Cycle_2.7_M3_RC1
| https://git.yoctoproject.org/cgit/cgit.cgi/yocto-testresults-contrib/log/?h=intel-yocto-testresults
|
|}
|}

Latest revision as of 07:16, 7 January 2020

{{#vardefine:ADT_MAUAL | 6539}} {{#vardefine:ADT_AUTO | 6540}} {{#vardefine:ANYQEMU_SATO_MANUAL | 6563}} {{#vardefine:ANYQEMU_SATO_AUTO | 6564}} {{#vardefine:ANYBSP_SATO_AUTO | 6558}} {{#vardefine:ANYBSP_LSB_AUTO | 6561}} {{#vardefine:ANYBSP_WIC_AUTO | 6560}} {{#vardefine:MMAX_CORE2_MANUAL | 6554}} {{#vardefine:MMAX_I7_MANUAL | 6557}} {{#vardefine:NUC_I7_MANUAL | 6556}} {{#vardefine:NUC_GEX86-64_MANUAL | 6555}} {{#vardefine:MMAX_GEX86-32_MANUAL | 6553}} {{#vardefine:GAL_SATO_AUTO | 6565}} {{#vardefine:CROPS_AUTO | 6541}} {{#vardefine:CROPS_MANUAL | 6542}} {{#vardefine:ECLIPSE_MANUAL | 6543}} {{#vardefine:KERNEL_MANUAL | 6714}} {{#vardefine:METAYOCTO_MANUAL | 6544}} {{#vardefine:TOASTER_AUTO | 6545}} {{#vardefine:TOASTER_MANUAL | 6546}} {{#vardefine:ESDK_MANUAL | 6547}} {{#vardefine:ESDK_AUTO | 6548}} {{#vardefine:OE-CORE_AUTO | 6549}} {{#vardefine:BITBAKE_AUTO | 6550}} {{#vardefine:RUNTIME_AUTO | 6551}} {{#vardefine:BUILD-APPLIANCE_AUTO | 6552}}

Reversion history

Version Modifier Comments
1.0 Jain Sangeeta First draft

Introduction

This is the the overall test plan for version 2.8 of the Yocto Project, for all the generic information this document will be pointing to QA Master Test Plan

Objectives

This test plan for Yocto Project 2.8 release supports the following objectives:

  • To define the tools to be used throughout the testing process.
  • To communicate to the responsible parties the items to be tested, set expectations around schedule, and define environmental needs.
  • To define how the tests will be conducted.
  • Release Criteria
  • Risks and contingencies

QA_Master_Test_Plan#Objectives_and_Tasks

Risk Analysis & Contingency Plan

QA identified risks for this release are mentioned here.

Module Risk Description Risk Level Contingency plan Owner
BSP 54 manual test cases medium test cases to be automated Intel
Build Appliance 3 manual test cases, not executed by any QA team medium test cases to be automated
Compliance 4 manual test cases, not executed by any QA team medium test cases to be automated
Kernel dev 9 manual test cases, not executed by any QA team medium test cases to be automated
OE-Core 6 manual test cases medium test cases to be automated Intel
SDK 1 manual test case, not executed by any QA team medium test case to be automated
Toaster 93 manual test cases, not executed by any QA team medium test cases to be automated

Test Strategy

Is followed the general strategy defined on QA_Master_Test_Plan#Testing_Strategy and below is showed specific information applicable only for 2.8 Release

Test Items

The Yocto Project QA team verifies each release against recent versions of the most popular Linux distributions that provide stable releases. 

* centos 7 
* Debian 8
* Debian 9
* Fedora 28
* Fedora 29
* Fedora 30
* OpenSuse 150
* OpenSuse 151
* OpenSuse 423
* OpenSuse Tumbleweed
* ubuntu 16.04 
* ubuntu 18.04

Test Cases

OE-CORE

Component Environment Auto Manual Owner
OE-CORE Ubuntu 16.04 Y PG

BSP/QEMU

Component Arch HW Environment Image Auto Manual pTest Compliance ltp
BSP x86 MinwTurbot 32bit genericx86 core-image-sato-sdk Y
core-image-lsb-sdk Y
Coffee Lake genericx86-64 core-image-sato-sdk Y Y
core-image-lsb-sdk Y
NUC 6 genericx86-64 core-image-sato-sdk Y Y Y
core-image-lsb-sdk Y Y
NUC 7 genericx86-64 core-image-sato-sdk Y Y
core-image-lsb-sdk Y
QEMU qemux86 core-image-sato-sdk Y Y
core-image-lsb-sdk Y
qemux86-64 core-image-sato-sdk Y Y
core-image-lsb-sdk Y
non x86 EdgeRouter EdgeRouter core-image-sato-sdk Y Y Y
MPC8315e-rdb MPC8315e-rdb core-image-sato-sdk Y Y Y
Beaglebone Black Beaglebone core-image-sato-sdk Y Y Y
QEMU qemuarm core-image-sato-sdk Y Y
qemuarm-64 core-image-sato-sdk Y Y Y Y
qemuppc core-image-sato-sdk Y Y
qemumips core-image-sato-sdk Y Y
qemumips64 core-image-sato-sdk Y Y
Meta-Intel MinwTurbot 32bit core2-32 core-image-sato-sdk Y
core-image-lsb-sdk Y
Coffee Lake corei7-64 core-image-sato-sdk Y Y
core-image-lsb-sdk Y
NUC 6 corei7-64 core-image-sato-sdk Y
core-image-lsb-sdk Y
NUC 7 corei7-64 core-image-sato-sdk Y Y
core-image-lsb-sdk Y
poky-tiny Y

All component to be tested on 2.8 are in #Test Items a detailed list of features per component can be found in Test Plan of the components QA_Master_Test_Plan#Component

Approach

Testing will be done as per QA process defined in main QA wiki page at https://wiki.yoctoproject.org/wiki/QA#QA_Process.

Pass / Fail Criteria

This criteria is defined in Yocto_Project_v2.8_Status#Yocto_Project_v2.8_Release_Criteria

Exit Criteria

As defined in QA_Master_Test_Plan#Entry and Exit Criteria

Test Deliverables

  • Final Test results
  • Final Test report
  • Mail sent to the mailing list

Schedule

The planed schedule for 2.8 can be found at Yocto_2.8_Schedule

Live Schedule & Execution History

MILESTONE STARTED FINISHED STATUS REPORT
M1 rc2 06/11/2019 06/14/2019 DONE https://git.yoctoproject.org/cgit/cgit.cgi/yocto-testresults-contrib/log/?h=intel-yocto-testresults
M2 7/26/2019 7/31/2019 DONE https://git.yoctoproject.org/cgit/cgit.cgi/yocto-testresults-contrib/tree/?h=zeus&id=d550d40646e1ca78a4b5d106e1b20831208b4c37
M3 9/23/2019 9/26/2019 DONE https://git.yoctoproject.org/cgit/cgit.cgi/yocto-testresults-contrib/log/?h=intel-yocto-testresults
3.0 10/10/2019 10/15/2019 DONE https://git.yoctoproject.org/cgit/cgit.cgi/yocto-testresults-contrib/log/?h=intel-yocto-testresults
Retrieved from "https://wiki.yoctoproject.org/wiki/index.php?title=Yocto_Project_2.8_Release_Test_Plan&oldid=66808"