Yocto Project 2.6 Release Test Plan: Difference between revisions

From Yocto Project
Jump to navigationJump to search
No edit summary
No edit summary
 
(40 intermediate revisions by 2 users not shown)
Line 69: Line 69:


*Bugs that need to be implemented by QA Team
*Bugs that need to be implemented by QA Team
**[[2.5_QA_OWNED_BUGS | 2.5 QA Assigned Bugs]]
**[[2.6_QA_OWNED_BUGS | 2.6 QA Assigned Bugs]]


*Bugs that need to be verified by QA Team  
*Bugs that need to be verified by QA Team  
Line 205: Line 205:
| Toaster
| Toaster
| Any Distro
| Any Distro
| Y
|  
| Y
| Y
| PG
| PG
Line 241: Line 241:
|-
|-
| rowspan="8" style="text-align: center;" | OE-CORE
| rowspan="8" style="text-align: center;" | OE-CORE
| Ubuntu 16.10
| Y
|
| PG
|-
| Ubuntu 16.04
| Ubuntu 16.04
| Y
| Y
Line 251: Line 246:
| PG
| PG
|-
|-
| OpenSuse 42.2
| Ubuntu 17.10
| Y
| Y
|  
|  
| PG
| PG
|-
|-
| Opensuse 13.2
| OpenSuse 42.3
| Y
| Y
|  
|  
| PG
| PG
|-
|-
| Debian 8
| Debian 9
| Y
| Y
|  
|  
Line 271: Line 266:
| PG
| PG
|-
|-
| Fedora 24
| Fedora 26
| Y
| Y
|  
|  
| PG
| PG
|-
|-
| Fedora 25
| Fedora 27
| Y
| Y
|  
|  
Line 325: Line 320:
| Package Manager
| Package Manager
| Any Distro
| Any Distro
| Y
|  
|  
| Y
| PG
| PG
|}
|}
Line 340: Line 335:
|-
|-
| rowspan="2" style="text-align: center;" | Performance
| rowspan="2" style="text-align: center;" | Performance
| Fedora 23
| Centos 7
| Y
| Y
|  
|  
| GDC
| PG
|-
|-
| Ubuntu 15
| Ubuntu 16.04
| Y
| Y
|  
|  
| GDC
| PG
|}
|}


Line 364: Line 359:
! Compliance
! Compliance
|-
|-
| rowspan="29" style="text-align: center;" | BSP
| rowspan="30" style="text-align: center;" | BSP
| rowspan="11" style="text-align: center;" | x86
| rowspan="12" style="text-align: center;" | x86
| rowspan="2" style="text-align: center;" | MinwMax 32bit
| rowspan="2" style="text-align: center;" | MinwTurbot 32bit
| rowspan="2" style="text-align: center;" | genericx86
| rowspan="2" style="text-align: center;" | genericx86
| core-image-sato-sdk
| core-image-sato-sdk
| Y
| Y
| Y
|  
|  
|  
|  
|  
Line 380: Line 375:
|  
|  
|-
|-
| rowspan="3" style="text-align: center;" | MinwMax 64bit
| rowspan="2" style="text-align: center;" | MinwTurbot 64bit
| rowspan="3" style="text-align: center;" | genericx86-64
| rowspan="2" style="text-align: center;" | genericx86-64
| core-image-sato-sdk
| core-image-sato-sdk
| Y
| Y
|  
| Y
|  
|  
|  
|  
Line 394: Line 389:
|  
|  
|-
|-
| core-image-tiny-initramfs
| rowspan="2" style="text-align: center;" | NUC 6
| rowspan="2" style="text-align: center;" | genericx86-64
| core-image-sato-sdk
| Y
| Y
|  
|  
| Y
|  
|  
|-
| core-image-lsb-sdk
| Y
|  
|  
|
| Y
|-
|-
| rowspan="2" style="text-align: center;" | NUC 64
| rowspan="2" style="text-align: center;" | NUC 7
| style="text-align: center;" | genericx86-64
| rowspan="2" style="text-align: center;" | genericx86-64
| core-image-sato-sdk
| core-image-sato-sdk
| Y
| Y
| Y
| Y
| Y
|  
|  
|  
|-
|-
|
| core-image-lsb-sdk
| core-image-lsb-sdk
| Y
| Y
|  
|  
|  
|  
| Y
|  
|-
|-
| rowspan="4" style="text-align: center;" | QEMU
| rowspan="4" style="text-align: center;" | QEMU
Line 504: Line 506:
|-
|-
| rowspan="10" style="text-align: center;" | Meta-Intel
| rowspan="10" style="text-align: center;" | Meta-Intel
| rowspan="2" style="text-align: center;" | MinnowMax 32bit
| rowspan="2" style="text-align: center;" | MinwTurbot 32bit
| rowspan="2" style="text-align: center;" | core2-32
| rowspan="2" style="text-align: center;" | core2-32
| core-image-sato-sdk
| core-image-sato-sdk
| Y
| Y
| Y
|  
|  
|  
|  
|  
Line 518: Line 520:
|  
|  
|-
|-
| rowspan="2" style="text-align: center;" | MinnowMax 64bit
| rowspan="2" style="text-align: center;" | MinwTurbot 64bit
| rowspan="2" style="text-align: center;" | corei7-64
| rowspan="2" style="text-align: center;" | corei7-64
| core-image-sato-sdk
| core-image-sato-sdk
| Y
| Y
|  
| Y
|  
|  
|  
|  
Line 532: Line 534:
|  
|  
|-
|-
| rowspan="2" style="text-align: center;" | NUC
| rowspan="2" style="text-align: center;" | NUC 6
| rowspan="2" style="text-align: center;" | corei7-64
| rowspan="2" style="text-align: center;" | corei7-64
| core-image-sato-sdk
| core-image-sato-sdk
| Y
| Y
| Y
|  
|  
|  
|  
|  
Line 546: Line 548:
|  
|  
|-
|-
| rowspan="2" style="text-align: center;" | CherryHill
| rowspan="3" style="text-align: center;" | NUC 7
| rowspan="2" style="text-align: center;" | corei7-64
| rowspan="3" style="text-align: center;" | corei7-64
| core-image-sato-sdk
| core-image-sato-sdk
| Y
| Y
|  
| Y
|  
|  
|  
|  
Line 560: Line 562:
|  
|  
|-
|-
| style="text-align: center;" | Joule
| poky-tiny
| style="text-align: center;" | corei7-64
| core-image-sato-sdk
| Y
|  
|  
|
|
|-
| style="text-align: center;" | meta-intel-quark 32bit
| style="text-align: center;" | intel-quark
| Core-image-sato-sdk
| Y
| Y
|
|  
|  
|  
|  
|}
|}


All component to be tested on 2.5 are in [[#Test Items]] a detailed list of features per component can be found in Test Plan of the components [[QA_Master_Test_Plan#Component]]
All component to be tested on 2.6 are in [[#Test Items]] a detailed list of features per component can be found in Test Plan of the components [[QA_Master_Test_Plan#Component]]


= Testopia Templates =
= Testopia Templates =
Line 586: Line 578:
|-
|-
| BitBake
| BitBake
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7403 TRTEMP_2.4_MANUAL_BitBake]
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7403 TRTEMP_2.6_MANUAL_BitBake]
|-
|-
| rowspan="15" | BSP/QEMU
| rowspan="15" | BSP/QEMU
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7404 TRTEMP_2.4_MANUAL_BSP/QEMU_core-image-sato-sdk_Genericx86_MMAX32]
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7404 TRTEMP_2.6_MANUAL_BSP/QEMU_core-image-sato-sdk_Genericx86_MMAX32]
|-
|-
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7405 TRTEMP_2.4_MANUAL_BSP/QEMU_core-image-sato-sdk_core2-32_MMAX32]
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7405 TRTEMP_2.6_MANUAL_BSP/QEMU_core-image-sato-sdk_core2-32_MMAX32]
|-
|-
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7406 TRTEMP_2.4_MANUAL_BSP/QEMU_core-image-sato-sdk_Genericx86-64_NUC]
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7406 TRTEMP_2.6_MANUAL_BSP/QEMU_core-image-sato-sdk_Genericx86-64_NUC]
|-
|-
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7407 TRTEMP_2.4_MANUAL_BSP/QEMU_core-image-sato-sdk_corei7-64_NUC]
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7407 TRTEMP_2.6_MANUAL_BSP/QEMU_core-image-sato-sdk_corei7-64_NUC]
|-
|-
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7408 TRTEMP_2.4_MANUAL_BSP/QEMU_core-image-sato-sdk_corei7-64_MMAX64]
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7408 TRTEMP_2.6_MANUAL_BSP/QEMU_core-image-sato-sdk_corei7-64_MMAX64]
|-
|-
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7409 TRTEMP_2.4_AUTO_BSP/QEMU_core-image-sato-sdk_ANYBSP]
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7409 TRTEMP_2.6_AUTO_BSP/QEMU_core-image-sato-sdk_ANYBSP]
|-
|-
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7410 TRTEMP_2.4_AUTO_BSP/QEMU_core-image-sato-sdk_wic_ANYBSP]
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7410 TRTEMP_2.6_AUTO_BSP/QEMU_core-image-sato-sdk_wic_ANYBSP]
|-
|-
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7411 TRTEMP_2.4_AUTO_BSP/QEMU_core-image-lsb-sdk_ANYBSP]
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7411 TRTEMP_2.6_AUTO_BSP/QEMU_core-image-lsb-sdk_ANYBSP]
|-
|-
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7412 TRTEMP_2.4_MANUAL_BSP/QEMU_core-image-sato-sdk_ANYQEMU]
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7412 TRTEMP_2.6_MANUAL_BSP/QEMU_core-image-sato-sdk_ANYQEMU]
|-
|-
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7413 TRTEMP_2.4_AUTO_BSP/QEMU_core-image-sato-sdk_ANYQEMU]
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7413 TRTEMP_2.6_AUTO_BSP/QEMU_core-image-sato-sdk_ANYQEMU]
|-
|-
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7414 TRTEMP_2.4_AUTO_BSP/QEMU_core-image-sato-sdk_intel-quark]
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7414 TRTEMP_2.6_AUTO_BSP/QEMU_core-image-sato-sdk_intel-quark]
|-
|-
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7415 TRTEMP_2.4_MANUAL_BSP/QEMU_core-image-sato-sdk_Beaglebone-Black]
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7415 TRTEMP_2.6_MANUAL_BSP/QEMU_core-image-sato-sdk_Beaglebone-Black]
|-
|-
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7416 TRTEMP_2.4_MANUAL_BSP/QEMU_core-image-sato-sdk_EdgeRouter]
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7416 TRTEMP_2.6_MANUAL_BSP/QEMU_core-image-sato-sdk_EdgeRouter]
|-
|-
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7417 TRTEMP_2.4_MANUAL_BSP/QEMU_core-image-sato-sdk_MPC8315e-rdb]
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7417 TRTEMP_2.6_MANUAL_BSP/QEMU_core-image-sato-sdk_MPC8315e-rdb]
|-
|-
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7418 TRTEMP_2.4_AUTO_BSP/QEMU_core-image-lsb-sdk_ANYQEMU]
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7418 TRTEMP_2.6_AUTO_BSP/QEMU_core-image-lsb-sdk_ANYQEMU]
|-
|-
| Meta-Yocto
| Meta-Yocto
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7419 TRTEMP_2.4_MANUAL_Meta-yocto]
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7419 TRTEMP_2.6_MANUAL_Meta-yocto]
|-
|-
| OE-CORE
| OE-CORE
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7420 TRTEMP_2.4_AUTO_OE-Core]
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7420 TRTEMP_2.6_AUTO_OE-Core]
|-
|-
| Build-Appliance
| Build-Appliance
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7421 TRTEMP_2.4_MANUAL_Build-Appliance]
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7421 TRTEMP_2.6_MANUAL_Build-Appliance]
|-
|-
| Runtime
| Runtime
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7422 TRTEMP_2.4_AUTO_Runtime]
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7422 TRTEMP_2.6_AUTO_Runtime]
|-
|-
| rowspan="2" | Toaster
| rowspan="2" | Toaster
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7423 TRTEMP_2.4_AUTO_Toaster_UI]
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7423 TRTEMP_2.6_AUTO_Toaster_UI]
|-
|-
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7424 TRTEMP_2.4_MANUAL_Toaster_UI-managed-mode]
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7424 TRTEMP_2.6_MANUAL_Toaster_UI-managed-mode]
|-
|-
| rowspan="2" | ADT
| rowspan="2" | ADT
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7425 TRTEMP_2.4_MANUAL_ADT]
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7425 TRTEMP_2.6_MANUAL_ADT]
|-
|-
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7426 TRTEMP_2.4_AUTO_ADT]
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7426 TRTEMP_2.6_AUTO_ADT]
|-
|-
| rowspan="2" | CROPS
| rowspan="2" | CROPS
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7430 TRTEMP_2.4_AUTO_CROPS_Poky]
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7430 TRTEMP_2.6_AUTO_CROPS_Poky]
|-
|-
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7431 TRTEMP_2.4_MANUAL_CROPS_eSDK]
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7431 TRTEMP_2.6_MANUAL_CROPS_eSDK]
|-
|-
| Eclipse
| Eclipse
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7432 TRTEMP_2.4_MANUAL_Eclipse]
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7432 TRTEMP_2.6_MANUAL_Eclipse]
|-
|-
| rowspan="2" | eSDK
| rowspan="2" | eSDK
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7433 TRTEMP_2.4_AUTO_eSDK_sdkext]
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7433 TRTEMP_2.6_AUTO_eSDK_sdkext]
|-
|-
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7434 TRTEMP_2.4_AUTO_eSDK_oeselftest]
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7434 TRTEMP_2.6_AUTO_eSDK_oeselftest]
|-
|-
| Kernel
| Kernel
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7435 TRTEMP_2.4_MANUAL_Kernel]
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7435 TRTEMP_2.6_MANUAL_Kernel]
|-
|-
| Package Management
| Package Management
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7489 TRTEMP_2.4_MANUAL_Package-Management-Updates]
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7489 TRTEMP_2.6_MANUAL_Package-Management-Updates]
|}
|}


Line 668: Line 660:




The complete List of QA Implementations for YP 2.4 can be found at [https://wiki.yoctoproject.org/wiki/2.4_QA_OWNED_BUGS Features Implementation by QA Team]
The complete List of QA Implementations for YP 2.6 can be found at [https://wiki.yoctoproject.org/wiki/2.6_QA_OWNED_BUGS Features Implementation by QA Team]


= Features Not to be tested =
= Features Not to be tested =
Line 681: Line 673:
= Pass / Fail Criteria =
= Pass / Fail Criteria =


This criteria is defined in [[Yocto_Project_v2.4_Status#Yocto_Project_v2.4_Release_Criteria]]
This criteria is defined in [[Yocto_Project_v2.6_Status#Yocto_Project_v2.6_Release_Criteria]]


= Exit Criteria =
= Exit Criteria =
Line 690: Line 682:




* Live Status [[2.5_QA_Status]]
* Live Status [[2.6_QA_Status]]
* Final execution Report [[#Execution History]]
* Final execution Report [[#Execution History]]
* Mail sent to the mailing list
* Mail sent to the mailing list
Line 699: Line 691:


= Schedule =
= Schedule =
== M1 (May 8, 2017 to Jun. 23, 2017 ==
== M1 (Apr 30, 2018 to Jun 22, 2018) ==
=== M1 Feature/Tasks Development / Cutoff is Jun. 12, 2017 ===
=== M1 Feature/Tasks Development / Cutoff is Jun. 11, 2017 ===


{{#bugzilla:
{{#bugzilla:
   |columns=id,from,to,estimated,summary,severity,priority,whiteboard,status
   |columns=id,from,to,estimated,summary,severity,priority,whiteboard,status
   |total=estimated
   |total=estimated
   |milestone=2.4 M1
   |milestone=2.6 M1, 2.6 M2
   |severity=enhancement
   |severity=enhancement
   |sort=to,priority
   |sort=to,priority
|to=corneliux.stoicescu@intel.com,alexandru.c.georgescu@intel.com,ionutx.chisanovici@intel.com,georgex.l.musat@intel.com,stanciux.mihail@intel.com,andrax.g.necula@intel.com,costin.c.constantin@intel.com,daniel.alexandrux.istrate@intel.com,alexandru.costinx.roman@intel.com,valentinx.hangan@intel.com,libertad.gonzalez.de.la.cruz@intel.com,francisco.j.pedraza.gonzalez@intel.com,jose.perez.carranza@intel.com,humberto.ibarra.lopez@intel.com,davidx.lopez.barriba@intel.com,juan.fernandox.ramos.frayle@intel.com,jair.de.jesus.gonzalez.plascencia@intel.com,sonia.leon@intel.com
}}
}}


Line 716: Line 707:
   |columns=id,from,to,estimated,summary,severity,priority,whiteboard,status
   |columns=id,from,to,estimated,summary,severity,priority,whiteboard,status
   |total=estimated
   |total=estimated
   |milestone=2.4 M1
   |milestone=2.6 M1, 2.6 M2
   |severity=!enhancement
   |severity=!enhancement
   |sort=to,priority
   |sort=to,priority
|to=corneliux.stoicescu@intel.com,alexandru.c.georgescu@intel.com,ionutx.chisanovici@intel.com,georgex.l.musat@intel.com,stanciux.mihail@intel.com,andrax.g.necula@intel.com,costin.c.constantin@intel.com,daniel.alexandrux.istrate@intel.com,alexandru.costinx.roman@intel.com,valentinx.hangan@intel.com,libertad.gonzalez.de.la.cruz@intel.com,francisco.j.pedraza.gonzalez@intel.com,jose.perez.carranza@intel.com,humberto.ibarra.lopez@intel.com,david.israelx.rodriguez.castellanos@intel.com,davidx.lopez.barriba@intel.com,juan.fernandox.ramos.frayle@intel.com,sonia.delgado@intel.com,jair.de.jesus.gonzalez.plascencia@intel.com
}}
}}




The planed schedule for 2.4 can be found at [[Yocto_2.4_Schedule]]
The planed schedule for 2.6 can be found at [[Yocto_2.6_Schedule]]


== Live Schedule & Execution History ==
== Live Schedule & Execution History ==
Line 733: Line 723:
! STATUS
! STATUS
! REPORT
! REPORT
|- bgcolor="ffbf00"
|- bgcolor="93DB70"
| M1 rc1
| M1 rc1
| 6/19/2018
| 6/19/2018
|  
| 7/2/2018
| IN PROGRESS
| DONE
|  
| https://wiki.yoctoproject.org/wiki/WW27_-_2018-07-02_-_Full_Test_Cycle_2.6_M1_rc1
|- bgcolor="grey"
|- bgcolor="93DB70"
| M1 rc1 meta-intel 10.0
| 7/13/2018
| 7/24/2018
| DONE
| https://wiki.yoctoproject.org/wiki/WW29_-_2018-07-19-_Full_Test_Cycle_-_2.6_M1_rc1_-_meta-intel
|- bgcolor="93DB70"
| M2 rc1
| 7/19/2018
| 8/7/2018
| DONE
| https://wiki.yoctoproject.org/wiki/WW32_-_2018-08-07_-_Full_Test_Cycle_2.6_M2_rc1
|- bgcolor="93DB70"
| M2 rc1 meta-intel 10.0
| 8/27/2018
| 9/4/2018
| DONE
| https://wiki.yoctoproject.org/wiki/WW36_-_2018-09-04-_Full_Test_Cycle_-_2.6_M2_rc1_-_meta-intel_10.0
|- bgcolor="93DB70"
| M3 rc1
| 9/25/2018
| 10/09/2018
| DONE
|https://wiki.yoctoproject.org/wiki/WW41_-_2018-10-03_-_Full_Test_Cycle_2.6_M3_RC1
|- bgcolor="93DB70"
| M4 rc1
| 10/30/2018
| 11/09/2018
| DONE
| https://wiki.yoctoproject.org/wiki/WW44_-_2018-10-30_-_Full_Test_Cycle_2.6_M4_RC1
|- bgcolor="93DB70"
| M4 rc2 meta-intel 10.0
| 12/14/2018
| 12/21/2018
| DONE
| https://wiki.yoctoproject.org/wiki/WW51_-_2018-12-21-_Full_Test_Cycle_-_2.6_rc2_-_meta-intel_10.0
|- bgcolor="93DB70"
| 2.6.1 rc1
| 01/09/2019
| 01/22/2019
| DONE
| https://wiki.yoctoproject.org/wiki/WW02_-_2019-01-09_-_Full_Test_Cycle_2.6.1_RC1
|- bgcolor="93DB70"
| 2.6.1 rc1 meta-intel 10.1
| 02/12/2019
| 02/18/2019
| DONE
| https://wiki.yoctoproject.org/wiki/WW07_-_2019-02-12-_Full_Test_Cycle_-_2.6.1_rc1_-_meta-intel_10.1
|- bgcolor="grey"  
|}
|}

Latest revision as of 06:23, 8 April 2019

{{#vardefine:ADT_MAUAL | 6539}} {{#vardefine:ADT_AUTO | 6540}} {{#vardefine:ANYQEMU_SATO_MANUAL | 6563}} {{#vardefine:ANYQEMU_SATO_AUTO | 6564}} {{#vardefine:ANYBSP_SATO_AUTO | 6558}} {{#vardefine:ANYBSP_LSB_AUTO | 6561}} {{#vardefine:ANYBSP_WIC_AUTO | 6560}} {{#vardefine:MMAX_CORE2_MANUAL | 6554}} {{#vardefine:MMAX_I7_MANUAL | 6557}} {{#vardefine:NUC_I7_MANUAL | 6556}} {{#vardefine:NUC_GEX86-64_MANUAL | 6555}} {{#vardefine:MMAX_GEX86-32_MANUAL | 6553}} {{#vardefine:GAL_SATO_AUTO | 6565}} {{#vardefine:CROPS_AUTO | 6541}} {{#vardefine:CROPS_MANUAL | 6542}} {{#vardefine:ECLIPSE_MANUAL | 6543}} {{#vardefine:KERNEL_MANUAL | 6714}} {{#vardefine:METAYOCTO_MANUAL | 6544}} {{#vardefine:TOASTER_AUTO | 6545}} {{#vardefine:TOASTER_MANUAL | 6546}} {{#vardefine:ESDK_MANUAL | 6547}} {{#vardefine:ESDK_AUTO | 6548}} {{#vardefine:OE-CORE_AUTO | 6549}} {{#vardefine:BITBAKE_AUTO | 6550}} {{#vardefine:RUNTIME_AUTO | 6551}} {{#vardefine:BUILD-APPLIANCE_AUTO | 6552}}

Reversion history

Version Modifier Comments
1.0 Yeoh Ee Peng First draft

References

Documents that support this test plan

Introduction

This is the the overall test plan for version 2.6 of the Yocto Project, for all the generic information this document will be pointing to QA Master Test Plan

Objectives

This test plan for Yocto Project 2.6 release supports the following objectives:

  • To define the tools to be used throughout the testing process.
  • To communicate to the responsible parties the items to be tested, set expectations around schedule, and define environmental needs.
  • To define how the tests will be conducted.
  • Release Criteria
  • Risks and contingencies

QA_Master_Test_Plan#Objectives_and_Tasks

Risk Analysis & Contingency Plan

This is the document for Risk Analysis https://docs.google.com/spreadsheets/d/1K-MZYo5sfabwI2UozGp6Uuh2sw5y309bXkyEZRPEEgk/edit#gid=0

Test Strategy

Is followed the general strategy defined on QA_Master_Test_Plan#Testing_Strategy and below is showed specific information applicable only for 2.6 Release

Bugs Tracking

Test Items

The Yocto Project QA team verifies each release against recent versions of the most popular Linux distributions that provide stable releases. 

* centos 7 
* Debian 9
* Fedora 26
* Fedora 27
* OpenSuse42.3
* ubuntu 16.04 
* ubuntu 17.10

Test Cases


Test Matrix

The components to be tested for version 2.6 of the Yocto Project against the different milestones are included in this section:

Code of Colors
Active

Tracked on Testpoia

Active

Non Tracked on Testopia

In Design

Will be tracked on Tesopia

Inactive


ADT

Component Environment Auto Manual Owner
ADT Fedora 25 i686 Y Y PG
Ubuntu 16.04 x86_64 Y Y PG

Eclipse Plugin

Component Environment Auto Manual Owner
Neon + Any Distro Y PG
Oxygen + Any Distro Y PG

Kernel

Component Environment Auto Manual Owner
Kernel Any Distro Y PG

Meta-Yocto

Component Environment Auto Manual Owner
Meta-Yocto Any Distro Y PG

Toaster

Component Environment Auto Manual Owner
Toaster Any Distro Y PG

eSDK

Component Environment Auto Manual Owner
eSDK Any Distro + runtime Y PG
Any Distro + selftest Y PG

Selftest

Component Environment Auto Manual Owner
OE-CORE Ubuntu 16.04 Y PG
Ubuntu 17.10 Y PG
OpenSuse 42.3 Y PG
Debian 9 Y PG
Centos 7 Y PG
Fedora 26 Y PG
Fedora 27 Y PG

Bitbake

Component Environment Auto Manual Owner
Bitbake Any Distro Y PG

Build - Apliance

Component Environment Auto Manual Owner
Build - Apliance VM - Any Distro Y PG

Package Manager

Component Environment Auto Manual Owner
Package Manager Any Distro Y PG

Performance

Component Environment Auto Manual Owner
Performance Centos 7 Y PG
Ubuntu 16.04 Y PG

BSP/QEMU

Component Arch HW Environment Image Auto Manual pTest Compliance
BSP x86 MinwTurbot 32bit genericx86 core-image-sato-sdk Y
core-image-lsb-sdk Y
MinwTurbot 64bit genericx86-64 core-image-sato-sdk Y Y
core-image-lsb-sdk Y
NUC 6 genericx86-64 core-image-sato-sdk Y Y
core-image-lsb-sdk Y Y
NUC 7 genericx86-64 core-image-sato-sdk Y Y
core-image-lsb-sdk Y
QEMU qemux86 core-image-sato-sdk Y Y
core-image-lsb-sdk Y
qemux86-64 core-image-sato-sdk Y Y
core-image-lsb-sdk Y
non x86 EdgeRouter EdgeRouter core-image-sato-sdk Y Y
MPC8315e-rdb MPC8315e-rdb core-image-sato-sdk Y Y
Beaglebone Black Beaglebone core-image-sato-sdk Y Y
QEMU qemuarm core-image-sato-sdk Y Y
qemuarm-64 core-image-sato-sdk Y Y
qemuppc core-image-sato-sdk Y Y
qemumips core-image-sato-sdk Y Y
qemumips64 core-image-sato-sdk Y Y
Meta-Intel MinwTurbot 32bit core2-32 core-image-sato-sdk Y
core-image-lsb-sdk Y
MinwTurbot 64bit corei7-64 core-image-sato-sdk Y Y
core-image-lsb-sdk Y
NUC 6 corei7-64 core-image-sato-sdk Y
core-image-lsb-sdk Y
NUC 7 corei7-64 core-image-sato-sdk Y Y
core-image-lsb-sdk Y
poky-tiny Y

All component to be tested on 2.6 are in #Test Items a detailed list of features per component can be found in Test Plan of the components QA_Master_Test_Plan#Component

Testopia Templates

COMPONENT TEMPLATE
BitBake TRTEMP_2.6_MANUAL_BitBake
BSP/QEMU TRTEMP_2.6_MANUAL_BSP/QEMU_core-image-sato-sdk_Genericx86_MMAX32
TRTEMP_2.6_MANUAL_BSP/QEMU_core-image-sato-sdk_core2-32_MMAX32
TRTEMP_2.6_MANUAL_BSP/QEMU_core-image-sato-sdk_Genericx86-64_NUC
TRTEMP_2.6_MANUAL_BSP/QEMU_core-image-sato-sdk_corei7-64_NUC
TRTEMP_2.6_MANUAL_BSP/QEMU_core-image-sato-sdk_corei7-64_MMAX64
TRTEMP_2.6_AUTO_BSP/QEMU_core-image-sato-sdk_ANYBSP
TRTEMP_2.6_AUTO_BSP/QEMU_core-image-sato-sdk_wic_ANYBSP
TRTEMP_2.6_AUTO_BSP/QEMU_core-image-lsb-sdk_ANYBSP
TRTEMP_2.6_MANUAL_BSP/QEMU_core-image-sato-sdk_ANYQEMU
TRTEMP_2.6_AUTO_BSP/QEMU_core-image-sato-sdk_ANYQEMU
TRTEMP_2.6_AUTO_BSP/QEMU_core-image-sato-sdk_intel-quark
TRTEMP_2.6_MANUAL_BSP/QEMU_core-image-sato-sdk_Beaglebone-Black
TRTEMP_2.6_MANUAL_BSP/QEMU_core-image-sato-sdk_EdgeRouter
TRTEMP_2.6_MANUAL_BSP/QEMU_core-image-sato-sdk_MPC8315e-rdb
TRTEMP_2.6_AUTO_BSP/QEMU_core-image-lsb-sdk_ANYQEMU
Meta-Yocto TRTEMP_2.6_MANUAL_Meta-yocto
OE-CORE TRTEMP_2.6_AUTO_OE-Core
Build-Appliance TRTEMP_2.6_MANUAL_Build-Appliance
Runtime TRTEMP_2.6_AUTO_Runtime
Toaster TRTEMP_2.6_AUTO_Toaster_UI
TRTEMP_2.6_MANUAL_Toaster_UI-managed-mode
ADT TRTEMP_2.6_MANUAL_ADT
TRTEMP_2.6_AUTO_ADT
CROPS TRTEMP_2.6_AUTO_CROPS_Poky
TRTEMP_2.6_MANUAL_CROPS_eSDK
Eclipse TRTEMP_2.6_MANUAL_Eclipse
eSDK TRTEMP_2.6_AUTO_eSDK_sdkext
TRTEMP_2.6_AUTO_eSDK_oeselftest
Kernel TRTEMP_2.6_MANUAL_Kernel
Package Management TRTEMP_2.6_MANUAL_Package-Management-Updates

Features pending to be tested

Here is a list of areas that are not yet covered and need some work, implement test cases and add it to a component that belongs or create a new test plan for new component.

ADD FEATURES !!!!!!!!!!


The complete List of QA Implementations for YP 2.6 can be found at Features Implementation by QA Team

Features Not to be tested

Approach

Tests will be conducted per the documented test cases stored in Testopia. The test manager will create test runs for each tester. The tester will execute the tests in Testopia and mark each case as Pass / Fail / Skip /Blocked. The tester should leave notes on actual results and any other relevant details when possible.

Once complete, the test manager should review the test run reports in Testopia and report back to the team by mail and in the wiki as a test report.

Pass / Fail Criteria

This criteria is defined in Yocto_Project_v2.6_Status#Yocto_Project_v2.6_Release_Criteria

Exit Criteria

As defined in QA_Master_Test_Plan#Entry and Exit Criteria

Test Deliverables

Test Environment

There are different environments and tools for every type of testing refer to QA_Master_Test_Plan#Test_Areas for more detailed information.

Schedule

M1 (Apr 30, 2018 to Jun 22, 2018)

M1 Feature/Tasks Development / Cutoff is Jun. 11, 2017

IDRequesterAssigneeESummary (4 tasks) ⇒SeverityPWhiteboardStatus
10681Richard PurdieAlejandro Hernandez10Support inter-multi-configuration dependenciesenhancementMedium+RESOLVED
12727Ross BurtonAlexander KanavinDrop wireless-tools and use iwenhancementMedium+RESOLVED
12689Anuj MittalAnuj Mittal1wic should be able to use initrd value for grubenhancementHighRESOLVED
10744Jose Perez CRichard Purdie5Update test case of smartpm/createrepo under meta/lib/oeqa/runtimeenhancementMedium+RESOLVED
   16     

M1 Bugs

IDRequesterAssigneeESummary (35 tasks) ⇒SeverityPWhiteboardStatus
12224brian averyAnuj Mittal2wic image created for qemux86-64 stalls on undefined video mode number: 318normalMediumRESOLVED
12767Robert BergerAnuj Mittalbitbake-layers create-layernormalMediumBackport 2.5.1RESOLVED
12773pankaj yadavAnuj Mittal[Meta-Yocto :2.4.3RC2] QA_WARN and QA_ERROR Mechanism checking FailednormalMedium+RESOLVED
12666pravinAnuj Mittal1[2.5 M3 RC1] BSP-QEMU:core-image-sato-sdk-genericx86-64.hddimg unable to connect to console with serial on Minnowboard TurbotnormalMedium+RESOLVED
12776Leonardo Sandoval GonzalezAnuj Mittal1hddimg installer hangs on sumonormalMedium+RESOLVED
12832pravinAnuj Mittal2[ 2.6 M1 rc1 ][BSP][Test case 267]: audio and video does not play in media player[Mturbot x86-64 and NUC7]normalMedium+RESOLVED
12803pankaj yadavApoorv[Yocto-2.6_M1.RC1] Building a kernel module on target image failsnormalMedium+RESOLVED
12805pankaj yadavApoorv[Test Case 1058] bitbake can use alternate download methods than gitnormalMedium+RESOLVED
12742Ross BurtonPaul Eggletonbuildhistory doesn't like files with whitespacenormalMedium+RESOLVED
12757He ZheBruce Ashfieldkernel do_configure fails to use tools in recipe-sysroot-nativecriticalMedium+RESOLVED
12794Robert BergerChin Huat AngEclipse Project SDK plugin Oxygen/2.5.0 can not be installednormalMedium+RESOLVED
12796Scott RifenbarkChin Huat AngEclipse Oxygen section needs work-around - it is brokenormalUndecidedRESOLVED
12741Ross BurtonHongxu Jiadistcc: upgrade to 3.3normalMedium+RESOLVED
12808Robert BergerJoshua Wattbitbake-layers create-layernormalMedium+RESOLVED
12836Manish DasManish Das[2.6 M1 RC1] busybox ptest failednormalMedium+RESOLVED
12656Juro BystrickyMaxin B. Johnat-spi2-core_2.28.0.bb:do_compile errornormalMedium+RESOLVED
12708Juro BystrickyMingli Yu(reproducibility) boost-dbg contains build host referencesnormalMedium+reproducibilityRESOLVED
12711Aaron ChanMohamad Noor Alim[2.5 M3 rc1][nightly-arm64][fedora 27][autobuilder2] - Event handler runqueue_stats failedmajorMedium+RESOLVED
12736Mohamad Noor AlimMohamad Noor Alim2.5 rc1: CentOS 7 oe-selftest fail at runtime_test.TestImage.test_testimage_dnfnormalMedium+RESOLVED
12792Olof JohanssonOlof JohanssonRPM leaks pre-/postinstall scripts to /var/tmpnormalMediumRESOLVED
12768Robert BergerRobert BergerPlease place them into pkg_postinst_ontarget_${PN} ()normalMediumRESOLVED
12729Chen QiChen Qi1testsdkext takes too long time to finishnormalHighRESOLVED
12835Chen QiChen Qi'core-image-weston + systemd' fails the 'parselogs' runtime oeqa test casenormalUndecidedRESOLVED
12866Mohamad Noor AlimRichard Purdie[2.6 M2 rc1] wic test_fix_size failednormalMediumRESOLVED
12739Matt HoosierRichard PurdieFetcher for Git submodules ('gitsm') bypasses BB_NO_NETWORK checksnormalMedium+Backport to YP 2.4 and 2.5RESOLVED
12765HainsRichard PurdieRace condition do_package_write_ipk fails due to empty package-split dir.normalMedium+RESOLVED
12828Maxin B. JohnRichard Purdiemesa build fails on edgerouter and beaglebone-yoctonormalMedium+RESOLVED
12756Armin KusterRoss Burton[ross/MUT] nightly-oecore failednormalMedium+RESOLVED
12799Armin KusterRoss BurtonOccasional mesa build racenormalMedium+RESOLVED
12815Maxin B. JohnRoss Burtonlibsdl2 build failednormalMedium+RESOLVED
12475Amber ElliotStephen K Jolley1Bug tracking BKMS should be documentednormalMedium+10 May 2018: IN PROGRESS REVIEWRESOLVED
12517Stephano CetolaScott Rifenbark1Update documentation on gen-tapdevs scriptnormalMedium+15 May 2018: RESOLVEDRESOLVED
11567Phlip BalisterStephano Cetola1Trying to create an new layer with the yocto-layer script fails.normalMedium+May 15 2018: RESOLVEDRESOLVED
12772Scott BrandenTim Orlingbitbake after devtool doesn't work properly with qemunormalMedium+RESOLVED
12778Uttam KUttam KUnable to build yocto poky projectmajorMedium+RESOLVED
   10     


The planed schedule for 2.6 can be found at Yocto_2.6_Schedule

Live Schedule & Execution History

MILESTONE STARTED FINISHED STATUS REPORT
M1 rc1 6/19/2018 7/2/2018 DONE https://wiki.yoctoproject.org/wiki/WW27_-_2018-07-02_-_Full_Test_Cycle_2.6_M1_rc1
M1 rc1 meta-intel 10.0 7/13/2018 7/24/2018 DONE https://wiki.yoctoproject.org/wiki/WW29_-_2018-07-19-_Full_Test_Cycle_-_2.6_M1_rc1_-_meta-intel
M2 rc1 7/19/2018 8/7/2018 DONE https://wiki.yoctoproject.org/wiki/WW32_-_2018-08-07_-_Full_Test_Cycle_2.6_M2_rc1
M2 rc1 meta-intel 10.0 8/27/2018 9/4/2018 DONE https://wiki.yoctoproject.org/wiki/WW36_-_2018-09-04-_Full_Test_Cycle_-_2.6_M2_rc1_-_meta-intel_10.0
M3 rc1 9/25/2018 10/09/2018 DONE https://wiki.yoctoproject.org/wiki/WW41_-_2018-10-03_-_Full_Test_Cycle_2.6_M3_RC1
M4 rc1 10/30/2018 11/09/2018 DONE https://wiki.yoctoproject.org/wiki/WW44_-_2018-10-30_-_Full_Test_Cycle_2.6_M4_RC1
M4 rc2 meta-intel 10.0 12/14/2018 12/21/2018 DONE https://wiki.yoctoproject.org/wiki/WW51_-_2018-12-21-_Full_Test_Cycle_-_2.6_rc2_-_meta-intel_10.0
2.6.1 rc1 01/09/2019 01/22/2019 DONE https://wiki.yoctoproject.org/wiki/WW02_-_2019-01-09_-_Full_Test_Cycle_2.6.1_RC1
2.6.1 rc1 meta-intel 10.1 02/12/2019 02/18/2019 DONE https://wiki.yoctoproject.org/wiki/WW07_-_2019-02-12-_Full_Test_Cycle_-_2.6.1_rc1_-_meta-intel_10.1
Retrieved from "https://wiki.yoctoproject.org/wiki/index.php?title=Yocto_Project_2.6_Release_Test_Plan&oldid=50658"