QA/sandbox: Difference between revisions

From Yocto Project
Jump to navigationJump to search
No edit summary
No edit summary
Line 1: Line 1:
[https://wiki.yoctoproject.org/wiki/Ptest/archive < Archive]
<!-- Variables for Template -->
<!-- ********************** -->
{|
{{#vardefine:ADT_MAUAL            | 6539}}
{{#vardefine:ADT_AUTO            | 6540}}
{{#vardefine:ANYQEMU_SATO_MANUAL  | 6563}}
{{#vardefine:ANYQEMU_SATO_AUTO    | 6564}}
{{#vardefine:ANYBSP_SATO_AUTO    | 6558}}
{{#vardefine:ANYBSP_LSB_AUTO      | 6561}}
{{#vardefine:ANYBSP_WIC_AUTO      | 6560}}
{{#vardefine:MMAX_CORE2_MANUAL    | 6554}}
{{#vardefine:MMAX_I7_MANUAL      | 6557}}
{{#vardefine:NUC_I7_MANUAL        | 6556}}
{{#vardefine:NUC_GEX86-64_MANUAL  | 6555}}
{{#vardefine:MMAX_GEX86-32_MANUAL | 6553}}
{{#vardefine:GAL_SATO_AUTO        | 6565}}
{{#vardefine:CROPS_AUTO          | 6541}}
{{#vardefine:CROPS_MANUAL        | 6542}}
{{#vardefine:ECLIPSE_MANUAL      | 6543}}
{{#vardefine:KERNEL_MANUAL        | 6714}}
{{#vardefine:METAYOCTO_MANUAL    | 6544}}
{{#vardefine:TOASTER_AUTO        | 6545}}
{{#vardefine:TOASTER_MANUAL      | 6546}}
{{#vardefine:ESDK_MANUAL          | 6547}}
{{#vardefine:ESDK_AUTO            | 6548}}
{{#vardefine:OE-CORE_AUTO        | 6549}}
{{#vardefine:BITBAKE_AUTO        | 6550}}
{{#vardefine:RUNTIME_AUTO        | 6551}}
{{#vardefine:BUILD-APPLIANCE_AUTO | 6552}}
|}
<!-- ********************* -->
 
= Reversion history =
{|class="wikitable"
! Version || Modifier || Comments
|-
| 1.0 || Jose Perez Carranza ||  First draft
|-
|}
 
=References=
Documents that support this test plan
 
= Introduction =
 
 
This is the the overall test plan for version 2.4  of the {{ns:4}}, for all the generic information this document will be pointing to [[QA Master Test Plan]]
 
= Objectives =
 
This test plan for Yocto Project 2.4 release supports the following objectives:
*To define the the tools to be used throughout the testing process.
*To communicate to the responsible parties the items to be tested, set expectations around schedule, and define environmental needs.
*To define how the tests will be conducted.
*Release Criteria
*Risks and contingencies
 
[[QA_Master_Test_Plan#Objectives_and_Tasks]]
 
= Test Strategy =
 
Is followed the general strategy defined on [[QA_Master_Test_Plan#Testing_Strategy]] and below is showed specific information applicable only for 2.4 Release
 
== Bugs Tracking ==
 
*Bugs that need to be implemented by QA Team
**[[2.4_QA_OWNED_BUGS | 2.4 QA Assigned Bugs]]
 
*Bugs that need to be verified by QA Team
**[[2.4_QA_Bugs_To_Verify]]
<!--
*Features to verify
**[[2.2 qa_owned features to verify]]
 
*Features to implement
**[[2.2 qa owned features]]
-->
 
*Old Bugs that need to be Verified
**[[Old resolved bugs and features]]
 
= Test Items =
The Yocto Project QA team verifies each release against recent versions of the most popular Linux distributions that provide stable releases.
* centos 7
* Debian 8
* Fedora 24
* Fedora 25
* OpenSuse132
* OpenSuse422
* ubuntu 16.04
* ubuntu 16.10
 
= Test Cases =
 
*[[Yocto 2.4 Test Cases]]
 
 
= Test Matrix =
 
The components to be tested  for version 2.4  of the {{ns:4}} against the different milestones  are included in this section:
 
{| class="wikitable"
! style="text-align: center;" | Code of Colors
|-bgcolor="93DB70"
| style="text-align: center;" | Active
Tracked on Testpoia
|-bgcolor="67E6EC"
| style="text-align: center;" | Active
Non Tracked on Testopia
|-bgcolor="ffbf00"
| style="text-align: center;" | In Design
Will be tracked on Tesopia
|- bgcolor="gray"
| style="text-align: center;" | Inactive
|}
 
 
== ADT ==
 
{| class="wikitable"
! style="text-align: center; font-weight: bold;" | Component
! style="text-align: center; font-weight: bold;" | Environment
! style="text-align: center; font-weight: bold;" | Auto
! style="text-align: center; font-weight: bold;" | Manual
! style="text-align: center; font-weight: bold;" | Owner
|-
| rowspan="2" style="text-align: center;" | ADT
| Fedora 25 i686
| Y
| Y
| WR
|-
| Ubuntu 16.04 x86_64
| Y
| Y
| WR
|}
 
== Eclipse Plugin ==
 
{| class="wikitable"
! style="text-align: center;" | Component
! style="text-align: center;" | Environment
! style="text-align: center;" | Auto
! style="text-align: center;" | Manual
! style="text-align: center;" | Owner
|-
| rowspan="2" style="text-align: center;" | Eclipse Plugin
| Mars + Any Distro
|
| Y
| GDC
|-
| Neon + Any Distro
|
| Y
| GDC
|}
 
== Kernel ==
 
{| class="wikitable"
! style="text-align: center;" | Component
! style="text-align: center;" | Environment
! style="text-align: center;" | Auto
! style="text-align: center;" | Manual
! style="text-align: center;" | Owner
|-
| Kernel
| Any Distro
|
| Y
| GDC
|}
 
== Meta-Yocto ==
 
{| class="wikitable"
! style="text-align: center;" | Component
! style="text-align: center;" | Environment
! style="text-align: center;" | Auto
! style="text-align: center;" | Manual
! style="text-align: center;" | Owner
|-
| Meta-Yocto
| Any Distro
|
| Y
| WR
|}
 
== Toaster ==
 
{| class="wikitable"
! style="text-align: center;" | Component
! style="text-align: center;" | Environment
! style="text-align: center;" | Auto
! style="text-align: center;" | Manual
! style="text-align: center;" | Owner
|-
| Toaster
| Any Distro
| Y
| Y
| GDC
|}
 
== eSDK ==
 
{| class="wikitable"
! style="text-align: center;" | Component
! style="text-align: center;" | Environment
! style="text-align: center;" | Auto
! style="text-align: center;" | Manual
! style="text-align: center;" | Owner
|-
| rowspan="2" style="text-align: center;" | eSDK
| Any Distro + runtime
| Y
|
| GDC
|-
| Any Distro + selftest
| Y
|
| GDC
|}
 
== Selftest ==
 
{| class="wikitable"
! style="text-align: center;" | Component
! style="text-align: center;" | Environment
! style="text-align: center;" | Auto
! style="text-align: center;" | Manual
! style="text-align: center;" | Owner
|-
| rowspan="8" style="text-align: center;" | OE-CORE
| Ubuntu 16.10
| Y
|
| GDC
|-
| Ubuntu 16.04
| Y
|
| GDC
|-
| OpenSuse 42.2
| Y
|
| GDC
|-
| Opensuse 13.2
| Y
|
| GDC
|-
| Debian 8
| Y
|
| GDC
|-
| Centos 7
| Y
|
| GDC
|-
| Fedora 24
| Y
|
| GDC
|-
| Fedora 25
| Y
|
| GDC
|}
 
== Bitbake ==
 
{| class="wikitable"
! style="text-align: center;" | Component
! style="text-align: center;" | Environment
! style="text-align: center;" | Auto
! style="text-align: center;" | Manual
! style="text-align: center;" | Owner
|-
| Bitbake
| Any Distro
|
| Y
| GDC
|}
 
== Build - Apliance ==
 
{| class="wikitable"
! style="text-align: center;" | Component
! style="text-align: center;" | Environment
! style="text-align: center;" | Auto
! style="text-align: center;" | Manual
! style="text-align: center;" | Owner
|-
| Build - Apliance
| VM - Any Distro
|
| Y
| GDC
|}


<br />Ran on a NUC and compared with[[Ptest a| b]]
== Package Manager ==


[<put-here-link-of-the-log> < FULL_LOG]
{| class="wikitable"
! style="text-align: center;" | Component
! style="text-align: center;" | Environment
! style="text-align: center;" | Auto
! style="text-align: center;" | Manual
! style="text-align: center;" | Owner
|-
| Package Manager
| Any Distro
|
| Y
| GDC
|}


{| style="border-spacing:0;"
== Performance ==
| style="background-color:#ff8080;border:0.05pt solid #000000;padding:0.0382in;"| Failures
 
{| class="wikitable"
! style="text-align: center;" | Component
! style="text-align: center;" | Environment
! style="text-align: center;" | Auto
! style="text-align: center;" | Manual
! style="text-align: center;" | Owner
|-
|-
| style="background-color:#99ff66;border-top:none;border-bottom:0.05ptsolid #000000;border-left:0.05pt solid #000000;border-right:0.05ptsolid #000000;padding:0.0201in;"| % increased
| rowspan="2" style="text-align: center;" | Performance
| Fedora 23
| Y
|
| GDC
|-
|-
| style="background-color:#c5000b;border-top:none;border-bottom:0.05ptsolid #000000;border-left:0.05pt solid #000000;border-right:0.05ptsolid #000000;padding:0.0201in;color:#ffffff"| % decreased
| Ubuntu 15
| Y
|
| GDC
|}
 
== BSP/QEMU ==
 
{| class="wikitable"
! Component
! Arch
! HW
! Environment
! Image
! Auto
! Manual
! pTest
! Compliance
|-
| rowspan="29" style="text-align: center;" | BSP
| rowspan="11" style="text-align: center;" | x86
| rowspan="2" style="text-align: center;" | MinwMax 32bit
| rowspan="2" style="text-align: center;" | genericx86
| core-image-sato-sdk
| Y
| Y
|
|
|-
| core-image-lsb-sdk
| Y
|
|
|
|-
| rowspan="3" style="text-align: center;" | MinwMax 64bit
| rowspan="3" style="text-align: center;" | genericx86-64
| core-image-sato-sdk
| Y
|
|
|
|-
| core-image-lsb-sdk
| Y
|
|
|
|-
| core-image-tiny-initramfs
| Y
|
|
|
|-
|-
| style="background-color:gray;border-top:none;border-bottom:0.05ptsolid #000000;border-left:0.05pt solid #000000;border-right:0.05ptsolid #000000;padding:0.0201in;color:#000000"| Module not executed
| rowspan="2" style="text-align: center;" | NUC 64
| style="text-align: center;" | genericx86-64
| core-image-sato-sdk
| Y
| Y
| Y
|
|-
|
| core-image-lsb-sdk
| Y
|
|
| Y
|-
| rowspan="4" style="text-align: center;" | QEMU
| rowspan="2" style="text-align: center;" | qemux86
| core-image-sato-sdk
| Y
| Y
|
|
|-
| core-image-lsb-sdk
| Y
|
|
|
|-
| rowspan="2" style="text-align: center;" | qemux86-64
| core-image-sato-sdk
| Y
| Y
|
|
|-
| core-image-lsb-sdk
| Y
|
|
|
|-
| rowspan="8" style="text-align: center;" | non x86
| style="text-align: center;" | EdgeRouter
| EdgeRouter
| core-image-sato-sdk
| Y
| Y
|
|
|-
| style="text-align: center;" | MPC8315e-rdb
| MPC8315e-rdb
| core-image-sato-sdk
| Y
| Y
|
|
|-
| style="text-align: center;" | Beaglebone Black
| Beaglebone
| core-image-sato-sdk
| Y
| Y
|
|
|-
| rowspan="5" style="text-align: center;" | QEMU
| qemuarm
| core-image-sato-sdk
| Y
| Y
|
|
|-
| qemuarm-64
| core-image-sato-sdk
| Y
| Y
|
|
|-
| qemuppc
| core-image-sato-sdk
| Y
| Y
|
|
|-
| qemumips
| core-image-sato-sdk
| Y
| Y
|
|
|-
| qemumips64
| core-image-sato-sdk
| Y
| Y
|
|
|-
| rowspan="10" style="text-align: center;" | Meta-Intel
| rowspan="2" style="text-align: center;" | MinnowMax 32bit
| rowspan="2" style="text-align: center;" | core2-32
| core-image-sato-sdk
| Y
| Y
|
|
|-
| core-image-lsb-sdk
| Y
|
|
|
|-
| rowspan="2" style="text-align: center;" | MinnowMax 64bit
| rowspan="2" style="text-align: center;" | corei7-64
| core-image-sato-sdk
| Y
|
|
|
|-
| core-image-lsb-sdk
| Y
|
|
|
|-
| rowspan="2" style="text-align: center;" | NUC
| rowspan="2" style="text-align: center;" | corei7-64
| core-image-sato-sdk
| Y
| Y
|
|
|-
| core-image-lsb-sdk
| Y
|
|
|
|-
| rowspan="2" style="text-align: center;" | CherryHill
| rowspan="2" style="text-align: center;" | corei7-64
| core-image-sato-sdk
| Y
|
|
|
|-
| core-image-lsb-sdk
| Y
|
|
|
|-
| style="text-align: center;" | Joule
| style="text-align: center;" | corei7-64
| core-image-sato-sdk
| Y
|
|
|
|-
| style="text-align: center;" | meta-intel-quark 32bit
| style="text-align: center;" | intel-quark
| Core-image-sato-sdk
| Y
|
|
|
|}
|}
All component to be tested on 2.4 are in [[#Test Items]] a detailed list of features per component can be found in Test Plan of the components [[QA_Master_Test_Plan#Component]]
= Testopia Templates =


{| class="wikitable"
{| class="wikitable"
|+Results
! style="font-weight: bold;" | COMPONENT
! style="font-weight: bold;" | TEMPLATE
|-
| BitBake
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7403 TRTEMP_2.4_MANUAL_BitBake]
|-
| rowspan="15" | BSP/QEMU
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7404 TRTEMP_2.4_MANUAL_BSP/QEMU_core-image-sato-sdk_Genericx86_MMAX32]
|-
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7405 TRTEMP_2.4_MANUAL_BSP/QEMU_core-image-sato-sdk_core2-32_MMAX32]
|-
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7406 TRTEMP_2.4_MANUAL_BSP/QEMU_core-image-sato-sdk_Genericx86-64_NUC]
|-
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7407 TRTEMP_2.4_MANUAL_BSP/QEMU_core-image-sato-sdk_corei7-64_NUC]
|-
|-
| align=center style="background-color:#ccffff;border-top:0.05pt solid #000000;border-bottom:0.5pt solid#000000;border-left:0.05pt solid #000000;border-right:0.5ptsolid #000000;padding:0.0201in;"| '''Package Name'''
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7408 TRTEMP_2.4_MANUAL_BSP/QEMU_core-image-sato-sdk_corei7-64_MMAX64]
| align=center style="background-color:#ccffff;border-top:0.05pt solid #000000;border-bottom:0.5pt solid#000000;border-left:0.05pt solid #000000;border-right:0.5ptsolid #000000;padding:0.0201in;"| '''Tests ran'''
| align=center style="background-color:#ccffff;border-top:0.05pt solid #000000;border-bottom:0.5pt solid#000000;border-left:0.05pt solid #000000;border-right:0.5ptsolid #000000;padding:0.0201in;"| '''Passed'''
| align=center style="background-color:#ccffff;border-top:0.05pt solid #000000;border-bottom:0.5pt solid#000000;border-left:0.05pt solid #000000;border-right:0.5ptsolid #000000;padding:0.0201in;"| '''Failed'''
| align=center style="background-color:#ccffff;border-top:0.05pt solid #000000;border-bottom:0.5pt solid#000000;border-left:0.05pt solid #000000;border-right:0.5ptsolid #000000;padding:0.0201in;"| '''Passrate'''
| align=center style="background-color:#ccffff;border-top:0.05pt solid #000000;border-bottom:0.5pt solid#000000;border-left:0.05pt solid #000000;border-right:0.5ptsolid #000000;padding:0.0201in;"| '''Last Passrate'''
|-
|-
| align=left style="background-color:trasnparentborder-color:#000000"| acl
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7409 TRTEMP_2.4_AUTO_BSP/QEMU_core-image-sato-sdk_ANYBSP]
| align=center style="background-color:transparent;border-color:#000000"|380
| align=center style="background-color:transparent;border-color:#000000"|380
| align=center style="background-color:transparent;border-color:#000000"|0
| align=center style="background-color:transparent;border-color:#000000;color:#000000"|100.0
| align=center style="background-color:transparent;border-color:#000000;color:#000000"|100.0
|-
|-
| align=left style="background-color:trasnparentborder-color:#000000"| attr
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7410 TRTEMP_2.4_AUTO_BSP/QEMU_core-image-sato-sdk_wic_ANYBSP]
| align=center style="background-color:transparent;border-color:#000000"|143
| align=center style="background-color:transparent;border-color:#000000"|143
| align=center style="background-color:transparent;border-color:#000000"|0
| align=center style="background-color:transparent;border-color:#000000;color:#000000"|100.0
| align=center style="background-color:transparent;border-color:#000000;color:#000000"|100.0
|-
|-
| align=left style="background-color:trasnparentborder-color:#000000"| bash
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7411 TRTEMP_2.4_AUTO_BSP/QEMU_core-image-lsb-sdk_ANYBSP]
| align=center style="background-color:transparent;border-color:#000000"|79
| align=center style="background-color:transparent;border-color:#000000"|74
| align=center style="background-color:#ff8080;border-color:#000000"|5
| align=center style="background-color:#c5000b;border-color:#000000;color:#ffffff"|93.67
| align=center style="background-color:#c5000b;border-color:#000000;color:#ffffff"|94.94
| align=center style="background-color:transparent;border-color:transparent" |[[File:down_rate.png|15px]]
|-
|-
| align=left style="background-color:trasnparentborder-color:#000000"| bluez5
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7412 TRTEMP_2.4_MANUAL_BSP/QEMU_core-image-sato-sdk_ANYQEMU]
| align=center style="background-color:transparent;border-color:#000000"|7
| align=center style="background-color:transparent;border-color:#000000"|7
| align=center style="background-color:transparent;border-color:#000000"|0
| align=center style="background-color:transparent;border-color:#000000;color:#000000"|100.0
| align=center style="background-color:transparent;border-color:#000000;color:#000000"|100.0
|-
|-
| align=left style="background-color:trasnparentborder-color:#000000"| busybox
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7413 TRTEMP_2.4_AUTO_BSP/QEMU_core-image-sato-sdk_ANYQEMU]
| align=center style="background-color:transparent;border-color:#000000"|518
| align=center style="background-color:transparent;border-color:#000000"|508
| align=center style="background-color:#ff8080;border-color:#000000"|10
| align=center style="background-color:#c5000b;border-color:#000000;color:#ffffff"|98.07
| align=center style="background-color:#c5000b;border-color:#000000;color:#ffffff"|98.26
| align=center style="background-color:transparent;border-color:transparent" |[[File:down_rate.png|15px]]
|-
|-
| align=left style="background-color:trasnparentborder-color:#000000"| bzip2
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7414 TRTEMP_2.4_AUTO_BSP/QEMU_core-image-sato-sdk_intel-quark]
| align=center style="background-color:transparent;border-color:#000000"|6
| align=center style="background-color:transparent;border-color:#000000"|6
| align=center style="background-color:transparent;border-color:#000000"|0
| align=center style="background-color:transparent;border-color:#000000;color:#000000"|100.0
| align=center style="background-color:transparent;border-color:#000000;color:#000000"|100.0
|-
|-
| align=left style="background-color:trasnparentborder-color:#000000"| dbus-test
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7415 TRTEMP_2.4_MANUAL_BSP/QEMU_core-image-sato-sdk_Beaglebone-Black]
| align=center style="background-color:transparent;border-color:#000000"|15
| align=center style="background-color:transparent;border-color:#000000"|4
| align=center style="background-color:#ff8080;border-color:#000000"|11
| align=center style="background-color:transparent;border-color:#000000;color:#000000"|26.67
| align=center style="background-color:transparent;border-color:#000000;color:#000000"|26.67
|-
|-
| align=left style="background-color:trasnparentborder-color:#000000"| diffstat
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7416 TRTEMP_2.4_MANUAL_BSP/QEMU_core-image-sato-sdk_EdgeRouter]
| align=center style="background-color:transparent;border-color:#000000"|440
| align=center style="background-color:transparent;border-color:#000000"|440
| align=center style="background-color:transparent;border-color:#000000"|0
| align=center style="background-color:transparent;border-color:#000000;color:#000000"|100.0
| align=center style="background-color:transparent;border-color:#000000;color:#000000"|100.0
|-
|-
| align=left style="background-color:trasnparentborder-color:#000000"| diffutils
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7417 TRTEMP_2.4_MANUAL_BSP/QEMU_core-image-sato-sdk_MPC8315e-rdb]
| align=center style="background-color:transparent;border-color:#000000"|19
| align=center style="background-color:transparent;border-color:#000000"|19
| align=center style="background-color:transparent;border-color:#000000"|0
| align=center style="background-color:transparent;border-color:#000000;color:#000000"|100.0
| align=center style="background-color:transparent;border-color:#000000;color:#000000"|100.0
|-
|-
| align=left style="background-color:trasnparentborder-color:#000000"| e2fsprogs
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7418 TRTEMP_2.4_AUTO_BSP/QEMU_core-image-lsb-sdk_ANYQEMU]
| align=center style="background-color:transparent;border-color:#000000"|1
| align=center style="background-color:transparent;border-color:#000000"|0
| align=center style="background-color:#ff8080;border-color:#000000"|1
| align=center style="background-color:#c5000b;border-color:#000000;color:#ffffff"|0.0
| align=center style="background-color:#c5000b;border-color:#000000;color:#ffffff"|100.0
| align=center style="background-color:transparent;border-color:transparent" |[[File:down_rate.png|15px]]
|-
|-
| align=left style="background-color:trasnparentborder-color:#000000"| flex
| Meta-Yocto
| align=center style="background-color:transparent;border-color:#000000"|114
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7419 TRTEMP_2.4_MANUAL_Meta-yocto]
| align=center style="background-color:transparent;border-color:#000000"|114
| align=center style="background-color:transparent;border-color:#000000"|0
| align=center style="background-color:transparent;border-color:#000000;color:#000000"|100.0
| align=center style="background-color:transparent;border-color:#000000;color:#000000"|100.0
|-
|-
| align=left style="background-color:trasnparentborder-color:#000000"| gawk
| OE-CORE
| align=center style="background-color:transparent;border-color:#000000"|298
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7420 TRTEMP_2.4_AUTO_OE-Core]
| align=center style="background-color:transparent;border-color:#000000"|297
| align=center style="background-color:#ff8080;border-color:#000000"|1
| align=center style="background-color:transparent;border-color:#000000;color:#000000"|99.66
| align=center style="background-color:transparent;border-color:#000000;color:#000000"|99.66
|-
|-
| align=left style="background-color:trasnparentborder-color:#000000"| gdbm
| Build-Appliance
| align=center style="background-color:transparent;border-color:#000000"|25
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7421 TRTEMP_2.4_MANUAL_Build-Appliance]
| align=center style="background-color:transparent;border-color:#000000"|25
| align=center style="background-color:transparent;border-color:#000000"|0
| align=center style="background-color:transparent;border-color:#000000;color:#000000"|100.0
| align=center style="background-color:transparent;border-color:#000000;color:#000000"|100.0
|-
|-
| align=left style="background-color:trasnparentborder-color:#000000"| gdk-pixbuf
| Runtime
| align=center style="background-color:transparent;border-color:#000000"|20
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7422 TRTEMP_2.4_AUTO_Runtime]
| align=center style="background-color:transparent;border-color:#000000"|20
| align=center style="background-color:transparent;border-color:#000000"|0
| align=center style="background-color:transparent;border-color:#000000;color:#000000"|100.0
| align=center style="background-color:transparent;border-color:#000000;color:#000000"|100.0
|-
|-
| align=left style="background-color:trasnparentborder-color:#000000"| glib-2.0
| rowspan="2" | Toaster
| align=center style="background-color:transparent;border-color:#000000"|253
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7423 TRTEMP_2.4_AUTO_Toaster_UI]
| align=center style="background-color:transparent;border-color:#000000"|253
| align=center style="background-color:transparent;border-color:#000000"|0
| align=center style="background-color:#99ff66;border-color:#000000;color:#000000"|100.0
| align=center style="background-color:#99ff66;border-color:#000000;color:#000000"|99.6
| align=center style="background-color:transparent;border-color:transparent" |[[File:up_rate.png|15px]]
|-
|-
| align=left style="background-color:trasnparentborder-color:#000000"| gzip
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7424 TRTEMP_2.4_MANUAL_Toaster_UI-managed-mode]
| align=center style="background-color:transparent;border-color:#000000"|18
| align=center style="background-color:transparent;border-color:#000000"|18
| align=center style="background-color:transparent;border-color:#000000"|0
| align=center style="background-color:transparent;border-color:#000000;color:#000000"|100.0
| align=center style="background-color:transparent;border-color:#000000;color:#000000"|100.0
|-
|-
| align=left style="background-color:trasnparentborder-color:#000000"| kbd
| rowspan="2" | ADT
| align=center style="background-color:transparent;border-color:#000000"|7
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7425 TRTEMP_2.4_MANUAL_ADT]
| align=center style="background-color:transparent;border-color:#000000"|7
| align=center style="background-color:transparent;border-color:#000000"|0
| align=center style="background-color:transparent;border-color:#000000;color:#000000"|100.0
| align=center style="background-color:transparent;border-color:#000000;color:#000000"|100.0
|-
|-
| align=left style="background-color:trasnparentborder-color:#000000"| libpcre
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7426 TRTEMP_2.4_AUTO_ADT]
| align=center style="background-color:transparent;border-color:#000000"|3
| align=center style="background-color:transparent;border-color:#000000"|3
| align=center style="background-color:transparent;border-color:#000000"|0
| align=center style="background-color:transparent;border-color:#000000;color:#000000"|100.0
| align=center style="background-color:transparent;border-color:#000000;color:#000000"|100.0
|-
|-
| align=left style="background-color:trasnparentborder-color:#000000"| libxml2
| rowspan="2" | CROPS
| align=center style="background-color:transparent;border-color:#000000"|39
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7430 TRTEMP_2.4_AUTO_CROPS_Poky]
| align=center style="background-color:transparent;border-color:#000000"|36
| align=center style="background-color:#ff8080;border-color:#000000"|3
| align=center style="background-color:transparent;border-color:#000000;color:#000000"|92.31
| align=center style="background-color:transparent;border-color:#000000;color:#000000"|92.31
|-
|-
| align=left style="background-color:trasnparentborder-color:#000000"| lttng-tools
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7431 TRTEMP_2.4_MANUAL_CROPS_eSDK]
| align=center style="background-color:transparent;border-color:#000000"|2057
| align=center style="background-color:transparent;border-color:#000000"|1988
| align=center style="background-color:#ff8080;border-color:#000000"|69
| align=center style="background-color:transparent;border-color:#000000;color:#000000"|96.65
| align=center style="background-color:transparent;border-color:#000000;color:#000000"|96.65
|-
|-
| align=left style="background-color:trasnparentborder-color:#000000"| nettle
| Eclipse
| align=center style="background-color:transparent;border-color:#000000"|88
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7432 TRTEMP_2.4_MANUAL_Eclipse]
| align=center style="background-color:transparent;border-color:#000000"|87
| align=center style="background-color:#ff8080;border-color:#000000"|1
| align=center style="background-color:transparent;border-color:#000000;color:#000000"|98.86
| align=center style="background-color:transparent;border-color:#000000;color:#000000"|98.86
|-
|-
| align=left style="background-color:trasnparentborder-color:#000000"| openssh
| rowspan="2" | eSDK
| align=center style="background-color:transparent;border-color:#000000"|58
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7433 TRTEMP_2.4_AUTO_eSDK_sdkext]
| align=center style="background-color:transparent;border-color:#000000"|58
| align=center style="background-color:transparent;border-color:#000000"|0
| align=center style="background-color:transparent;border-color:#000000;color:#000000"|100.0
| align=center style="background-color:transparent;border-color:#000000;color:#000000"|100.0
|-
|-
| align=left style="background-color:trasnparentborder-color:#000000"| openssl
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7434 TRTEMP_2.4_AUTO_eSDK_oeselftest]
| align=center style="background-color:transparent;border-color:#000000"|52
|-
| align=center style="background-color:transparent;border-color:#000000"|52
| Kernel
| align=center style="background-color:transparent;border-color:#000000"|0
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7435 TRTEMP_2.4_MANUAL_Kernel]
| align=center style="background-color:transparent;border-color:#000000;color:#000000"|100.0
|}
| align=center style="background-color:transparent;border-color:#000000;color:#000000"|100.0
 
= Features pending to be tested =
 
Here is a list of areas that are not yet covered and need some work, implement test cases and add it to a component that belongs or create a new test plan for new component.
 
'''ADD FEATURES !!!!!!!!!!'''
 
 
The complete List of QA Implementations for YP 2.4 can be found at [https://wiki.yoctoproject.org/wiki/2.4_QA_OWNED_BUGS Features Implementation by QA Team]
 
= Features Not to be tested =
*
 
=Approach=
Tests will be conducted per the documented test cases stored in Testopia. The test manager will create test runs for each tester. The tester will execute the tests in Testopia and mark each case as Pass / Fail / Skip /Blocked.
The tester should leave notes on actual results and any other relevant details when possible.
 
Once complete, the test manager should review the test run reports in Testopia and report back to the team by mail and in the wiki as a test report.
 
= Pass / Fail Criteria =
 
This criteria is defined in [[Yocto_Project_v2.4_Status#Yocto_Project_v2.4_Release_Criteria]]
 
= Exit Criteria =
 
As defined in [[QA_Master_Test_Plan#Entry and Exit Criteria]]
 
= Test Deliverables =
 
 
* Live Status [[2.4_QA_Status]]
* Final execution Report [[#Execution History]]
* Mail sent to the mailing list
 
= Test Environment =
 
There are different environments and tools for every type of testing refer to [[QA_Master_Test_Plan#Test_Areas]] for more detailed information.
 
= Schedule =
== M1 (May 8, 2017 to Jun. 23, 2017  ==
=== M1 Feature/Tasks Development / Cutoff is Jun. 12, 2017 ===
 
{{#bugzilla:
  |columns=id,from,to,estimated,summary,severity,priority,whiteboard,status
  |total=estimated
  |milestone=2.4 M1
  |severity=enhancement
  |sort=to,priority
|to=corneliux.stoicescu@intel.com,alexandru.c.georgescu@intel.com,ionutx.chisanovici@intel.com,georgex.l.musat@intel.com,stanciux.mihail@intel.com,andrax.g.necula@intel.com,costin.c.constantin@intel.com,daniel.alexandrux.istrate@intel.com,alexandru.costinx.roman@intel.com,valentinx.hangan@intel.com,libertad.gonzalez.de.la.cruz@intel.com,francisco.j.pedraza.gonzalez@intel.com,jose.perez.carranza@intel.com,humberto.ibarra.lopez@intel.com,davidx.lopez.barriba@intel.com,juan.fernandox.ramos.frayle@intel.com,jair.de.jesus.gonzalez.plascencia@intel.com,sonia.leon@intel.com
}}
 
=== M1 Bugs ===
 
{{#bugzilla:
  |columns=id,from,to,estimated,summary,severity,priority,whiteboard,status
  |total=estimated
  |milestone=2.4 M1
  |severity=!enhancement
  |sort=to,priority
|to=corneliux.stoicescu@intel.com,alexandru.c.georgescu@intel.com,ionutx.chisanovici@intel.com,georgex.l.musat@intel.com,stanciux.mihail@intel.com,andrax.g.necula@intel.com,costin.c.constantin@intel.com,daniel.alexandrux.istrate@intel.com,alexandru.costinx.roman@intel.com,valentinx.hangan@intel.com,libertad.gonzalez.de.la.cruz@intel.com,francisco.j.pedraza.gonzalez@intel.com,jose.perez.carranza@intel.com,humberto.ibarra.lopez@intel.com,david.israelx.rodriguez.castellanos@intel.com,davidx.lopez.barriba@intel.com,juan.fernandox.ramos.frayle@intel.com,sonia.delgado@intel.com,jair.de.jesus.gonzalez.plascencia@intel.com
}}
 
 
The planed schedule for 2.4 can be found at [[Yocto_2.4_Schedule]]
 
== Live Schedule ==
 
{| class="wikitable"
! MILESTONE
! STARTED
! FINISHED
! STATUS
|- bgcolor="00FF00"
| M1 rc1
| 06/15/2017
| 06/22/2017
| DONE
|- bgcolor="00FF00"
| M1 rc1 meta-intel
| 06/23/2017
| 06/27/2017
| DONE
|- bgcolor="00FF00"  
| M2 rc2
| 07/26/2017
| 08/02/2017
| DONE
|- bgcolor="00FF00"
| M2 rc3
| 08/03/2017
| 08/10/2017
| DONE
|- bgcolor="00FF00"
| M2 rc2 meta-intel
| 08/11/2017
| 08/16/2017
| DONE
|- bgcolor="00FF00"  
| M3
| 08/23/2017
| 08/30/2017
| DONE
|- bgcolor="00FF00"
| M3 rc1 meta-intel
| 09/01/2017
| 09/05/2017
| DONE
|- bgcolor="00FF00"
| M4 RC1
| 09/26/2017
| 10/06/2017
| DONE
|- bgcolor="00FF00"
| M4 RC2
| 10/09/2017
| 10/13/2017
| DONE
|- bgcolor="00FF00"
| M4 RC3
| 10/17/2017
| 10/23/2017
| DONE
|- bgcolor="00FF00"
| M4 rc1 meta-intel
| 10/26/2017
| 10/31/2017
| DONE
|- bgcolor="00FF00"
| M4 rc2 meta-intel
| 11/03/2017
| 11/06/2017
| DONE
|- bgcolor="grey"
|
|
|
|
|- bgcolor="grey"
|
|
|
|
|}
 
= Execution History =
 
 
{| class="wikitable"
! MILESTONE
! REPORT
|-
|-
| align=left style="background-color:trasnparentborder-color:#000000"| pango
| M1
| align=center style="background-color:transparent;border-color:#000000"|12
|https://wiki.yoctoproject.org/wiki/WW25_-_2017-06-22_-_Full_Test_Cycle_2.4_M1_rc1
| align=center style="background-color:transparent;border-color:#000000"|12
| align=center style="background-color:transparent;border-color:#000000"|0
| align=center style="background-color:transparent;border-color:#000000;color:#000000"|100.0
| align=center style="background-color:transparent;border-color:#000000;color:#000000"|100.0
|-
|-
| align=left style="background-color:trasnparentborder-color:#000000"| perl
| M1 meta-intel
| align=center style="background-color:transparent;border-color:#000000"|2286
|https://wiki.yoctoproject.org/wiki/WW26_-_2017-06-27-_Full_Test_Cycle_-_2.4_M1_rc1_-_meta-intel
| align=center style="background-color:transparent;border-color:#000000"|2285
| align=center style="background-color:#ff8080;border-color:#000000"|1
| align=center style="background-color:transparent;border-color:#000000;color:#000000"|99.96
| align=center style="background-color:transparent;border-color:#000000;color:#000000"|99.96
|-
|-
| align=left style="background-color:trasnparentborder-color:#000000"| perl5
| M2
| align=center style="background-color:transparent;border-color:#000000"|2286
|https://wiki.yoctoproject.org/wiki/WW31_-_2017-07-31_-_Full_Test_Cycle_2.4_M2_rc2
| align=center style="background-color:transparent;border-color:#000000"|2285
| align=center style="background-color:#ff8080;border-color:#000000"|1
| align=center style="background-color:transparent;border-color:#000000;color:#000000"|99.96
| align=center style="background-color:transparent;border-color:#000000;color:#000000"|99.96
|-
|-
| align=left style="background-color:trasnparentborder-color:#000000"| python
| M2 - rc3
| align=center style="background-color:gray;border-color:#000000"|0
|https://wiki.yoctoproject.org/wiki/WW32_-_2017-08-09_-_Full_Test_Cycle_2.4_M2_rc3
| align=center style="background-color:gray;border-color:#000000"|0
| align=center style="background-color:gray;border-color:#000000"|0
| align=center style="background-color:gray;border-color:#000000;color:#000000"|0
| align=center style="background-color:gray;border-color:#000000;color:#000000"|0
| align=center style="background-color:transparent;border-color:transparent" |[[File:no_rate.png|15px]]
|-
|-
| align=left style="background-color:trasnparentborder-color:#000000"| quilt
| M2 - rc2 meta-intel
| align=center style="background-color:transparent;border-color:#000000"|57
|https://wiki.yoctoproject.org/wiki/WW33_-_2017-08-15-_Full_Test_Cycle_-_2.4_M2_rc2_-_meta-intel
| align=center style="background-color:transparent;border-color:#000000"|57
|-
| align=center style="background-color:transparent;border-color:#000000"|0
| M3
| align=center style="background-color:transparent;border-color:#000000;color:#000000"|100.0
|https://wiki.yoctoproject.org/wiki/WW35_-_2017-08-30_-_Full_Test_Cycle_2.4_M3
| align=center style="background-color:transparent;border-color:#000000;color:#000000"|100.0
|-
| M3 - rc1 meta-intel
|https://wiki.yoctoproject.org/wiki/WW36_-_2017-09-05-_Full_Test_Cycle_-_2.4_M3_rc1_-_meta-intel
|-
|-
| align=left style="background-color:trasnparentborder-color:#000000"| slang
| M4 - rc1
| align=center style="background-color:transparent;border-color:#000000"|97
|https://wiki.yoctoproject.org/wiki/WW40_-_2017-10-06_-_Full_Test_Cycle_2.4_RC1
| align=center style="background-color:transparent;border-color:#000000"|97
| align=center style="background-color:transparent;border-color:#000000"|0
| align=center style="background-color:transparent;border-color:#000000;color:#000000"|100.0
| align=center style="background-color:transparent;border-color:#000000;color:#000000"|100.0
|-
|-
| align=left style="background-color:trasnparentborder-color:#000000"| strace
| M4 - rc2
| align=center style="background-color:transparent;border-color:#000000"|356
|https://wiki.yoctoproject.org/wiki/WW41_-_2017-10-13_-_Full_Test_Cycle_2.4_RC2
| align=center style="background-color:transparent;border-color:#000000"|355
| align=center style="background-color:#ff8080;border-color:#000000"|1
| align=center style="background-color:transparent;border-color:#000000;color:#000000"|99.72
| align=center style="background-color:transparent;border-color:#000000;color:#000000"|99.72
|-
|-
| align=left style="background-color:trasnparentborder-color:#000000"| tcl
| M4 - rc3
| align=center style="background-color:transparent;border-color:#000000"|148
|https://wiki.yoctoproject.org/wiki/WW43_-_2017-10-23_-_Full_Test_Cycle_2.4_RC3
| align=center style="background-color:transparent;border-color:#000000"|147
| align=center style="background-color:#ff8080;border-color:#000000"|1
| align=center style="background-color:transparent;border-color:#000000;color:#000000"|99.32
| align=center style="background-color:transparent;border-color:#000000;color:#000000"|99.32
|-
|-
| align=left style="background-color:trasnparentborder-color:#000000"| util-linux
| M4 - rc1 meta-intel
| align=center style="background-color:transparent;border-color:#000000"|123
|https://wiki.yoctoproject.org/wiki/WW44_-_2017-10-31-_Full_Test_Cycle_-_2.4_M4_rc1_-_meta-intel
| align=center style="background-color:transparent;border-color:#000000"|102
| align=center style="background-color:#ff8080;border-color:#000000"|21
| align=center style="background-color:transparent;border-color:#000000;color:#000000"|82.93
| align=center style="background-color:transparent;border-color:#000000;color:#000000"|82.93
|-
|-
| align=left style="background-color:trasnparentborder-color:#000000"| valgrind
| M4 - rc2 meta-intel
| align=center style="background-color:transparent;border-color:#000000"|559
|https://wiki.yoctoproject.org/wiki/WW45_-_2017-11-06-_Full_Test_Cycle_-_2.4_M4_rc2_-_meta-intel
| align=center style="background-color:transparent;border-color:#000000"|299
| align=center style="background-color:#ff8080;border-color:#000000"|260
| align=center style="background-color:transparent;border-color:#000000;color:#000000"|53.49
| align=center style="background-color:transparent;border-color:#000000;color:#000000"|53.49
|-
|-
| align=left style="background-color:trasnparentborder-color:#000000"| zlib
|  
| align=center style="background-color:transparent;border-color:#000000"|2
|
| align=center style="background-color:transparent;border-color:#000000"|2
| align=center style="background-color:transparent;border-color:#000000"|0
| align=center style="background-color:transparent;border-color:#000000;color:#000000"|100.0
| align=center style="background-color:transparent;border-color:#000000;color:#000000"|100.0
|-
|-
|
|
|}

Revision as of 15:06, 9 November 2017

{{#vardefine:ADT_MAUAL | 6539}} {{#vardefine:ADT_AUTO | 6540}} {{#vardefine:ANYQEMU_SATO_MANUAL | 6563}} {{#vardefine:ANYQEMU_SATO_AUTO | 6564}} {{#vardefine:ANYBSP_SATO_AUTO | 6558}} {{#vardefine:ANYBSP_LSB_AUTO | 6561}} {{#vardefine:ANYBSP_WIC_AUTO | 6560}} {{#vardefine:MMAX_CORE2_MANUAL | 6554}} {{#vardefine:MMAX_I7_MANUAL | 6557}} {{#vardefine:NUC_I7_MANUAL | 6556}} {{#vardefine:NUC_GEX86-64_MANUAL | 6555}} {{#vardefine:MMAX_GEX86-32_MANUAL | 6553}} {{#vardefine:GAL_SATO_AUTO | 6565}} {{#vardefine:CROPS_AUTO | 6541}} {{#vardefine:CROPS_MANUAL | 6542}} {{#vardefine:ECLIPSE_MANUAL | 6543}} {{#vardefine:KERNEL_MANUAL | 6714}} {{#vardefine:METAYOCTO_MANUAL | 6544}} {{#vardefine:TOASTER_AUTO | 6545}} {{#vardefine:TOASTER_MANUAL | 6546}} {{#vardefine:ESDK_MANUAL | 6547}} {{#vardefine:ESDK_AUTO | 6548}} {{#vardefine:OE-CORE_AUTO | 6549}} {{#vardefine:BITBAKE_AUTO | 6550}} {{#vardefine:RUNTIME_AUTO | 6551}} {{#vardefine:BUILD-APPLIANCE_AUTO | 6552}}

Reversion history

Version Modifier Comments
1.0 Jose Perez Carranza First draft

References

Documents that support this test plan

Introduction

This is the the overall test plan for version 2.4 of the Yocto Project, for all the generic information this document will be pointing to QA Master Test Plan

Objectives

This test plan for Yocto Project 2.4 release supports the following objectives:

  • To define the the tools to be used throughout the testing process.
  • To communicate to the responsible parties the items to be tested, set expectations around schedule, and define environmental needs.
  • To define how the tests will be conducted.
  • Release Criteria
  • Risks and contingencies

QA_Master_Test_Plan#Objectives_and_Tasks

Test Strategy

Is followed the general strategy defined on QA_Master_Test_Plan#Testing_Strategy and below is showed specific information applicable only for 2.4 Release

Bugs Tracking

Test Items

The Yocto Project QA team verifies each release against recent versions of the most popular Linux distributions that provide stable releases.

* centos 7 
* Debian 8
* Fedora 24 
* Fedora 25
* OpenSuse132 
* OpenSuse422 
* ubuntu 16.04 
* ubuntu 16.10

Test Cases


Test Matrix

The components to be tested for version 2.4 of the Yocto Project against the different milestones are included in this section:

Code of Colors
Active

Tracked on Testpoia

Active

Non Tracked on Testopia

In Design

Will be tracked on Tesopia

Inactive


ADT

Component Environment Auto Manual Owner
ADT Fedora 25 i686 Y Y WR
Ubuntu 16.04 x86_64 Y Y WR

Eclipse Plugin

Component Environment Auto Manual Owner
Eclipse Plugin Mars + Any Distro Y GDC
Neon + Any Distro Y GDC

Kernel

Component Environment Auto Manual Owner
Kernel Any Distro Y GDC

Meta-Yocto

Component Environment Auto Manual Owner
Meta-Yocto Any Distro Y WR

Toaster

Component Environment Auto Manual Owner
Toaster Any Distro Y Y GDC

eSDK

Component Environment Auto Manual Owner
eSDK Any Distro + runtime Y GDC
Any Distro + selftest Y GDC

Selftest

Component Environment Auto Manual Owner
OE-CORE Ubuntu 16.10 Y GDC
Ubuntu 16.04 Y GDC
OpenSuse 42.2 Y GDC
Opensuse 13.2 Y GDC
Debian 8 Y GDC
Centos 7 Y GDC
Fedora 24 Y GDC
Fedora 25 Y GDC

Bitbake

Component Environment Auto Manual Owner
Bitbake Any Distro Y GDC

Build - Apliance

Component Environment Auto Manual Owner
Build - Apliance VM - Any Distro Y GDC

Package Manager

Component Environment Auto Manual Owner
Package Manager Any Distro Y GDC

Performance

Component Environment Auto Manual Owner
Performance Fedora 23 Y GDC
Ubuntu 15 Y GDC

BSP/QEMU

Component Arch HW Environment Image Auto Manual pTest Compliance
BSP x86 MinwMax 32bit genericx86 core-image-sato-sdk Y Y
core-image-lsb-sdk Y
MinwMax 64bit genericx86-64 core-image-sato-sdk Y
core-image-lsb-sdk Y
core-image-tiny-initramfs Y
NUC 64 genericx86-64 core-image-sato-sdk Y Y Y
core-image-lsb-sdk Y Y
QEMU qemux86 core-image-sato-sdk Y Y
core-image-lsb-sdk Y
qemux86-64 core-image-sato-sdk Y Y
core-image-lsb-sdk Y
non x86 EdgeRouter EdgeRouter core-image-sato-sdk Y Y
MPC8315e-rdb MPC8315e-rdb core-image-sato-sdk Y Y
Beaglebone Black Beaglebone core-image-sato-sdk Y Y
QEMU qemuarm core-image-sato-sdk Y Y
qemuarm-64 core-image-sato-sdk Y Y
qemuppc core-image-sato-sdk Y Y
qemumips core-image-sato-sdk Y Y
qemumips64 core-image-sato-sdk Y Y
Meta-Intel MinnowMax 32bit core2-32 core-image-sato-sdk Y Y
core-image-lsb-sdk Y
MinnowMax 64bit corei7-64 core-image-sato-sdk Y
core-image-lsb-sdk Y
NUC corei7-64 core-image-sato-sdk Y Y
core-image-lsb-sdk Y
CherryHill corei7-64 core-image-sato-sdk Y
core-image-lsb-sdk Y
Joule corei7-64 core-image-sato-sdk Y
meta-intel-quark 32bit intel-quark Core-image-sato-sdk Y

All component to be tested on 2.4 are in #Test Items a detailed list of features per component can be found in Test Plan of the components QA_Master_Test_Plan#Component

Testopia Templates

COMPONENT TEMPLATE
BitBake TRTEMP_2.4_MANUAL_BitBake
BSP/QEMU TRTEMP_2.4_MANUAL_BSP/QEMU_core-image-sato-sdk_Genericx86_MMAX32
TRTEMP_2.4_MANUAL_BSP/QEMU_core-image-sato-sdk_core2-32_MMAX32
TRTEMP_2.4_MANUAL_BSP/QEMU_core-image-sato-sdk_Genericx86-64_NUC
TRTEMP_2.4_MANUAL_BSP/QEMU_core-image-sato-sdk_corei7-64_NUC
TRTEMP_2.4_MANUAL_BSP/QEMU_core-image-sato-sdk_corei7-64_MMAX64
TRTEMP_2.4_AUTO_BSP/QEMU_core-image-sato-sdk_ANYBSP
TRTEMP_2.4_AUTO_BSP/QEMU_core-image-sato-sdk_wic_ANYBSP
TRTEMP_2.4_AUTO_BSP/QEMU_core-image-lsb-sdk_ANYBSP
TRTEMP_2.4_MANUAL_BSP/QEMU_core-image-sato-sdk_ANYQEMU
TRTEMP_2.4_AUTO_BSP/QEMU_core-image-sato-sdk_ANYQEMU
TRTEMP_2.4_AUTO_BSP/QEMU_core-image-sato-sdk_intel-quark
TRTEMP_2.4_MANUAL_BSP/QEMU_core-image-sato-sdk_Beaglebone-Black
TRTEMP_2.4_MANUAL_BSP/QEMU_core-image-sato-sdk_EdgeRouter
TRTEMP_2.4_MANUAL_BSP/QEMU_core-image-sato-sdk_MPC8315e-rdb
TRTEMP_2.4_AUTO_BSP/QEMU_core-image-lsb-sdk_ANYQEMU
Meta-Yocto TRTEMP_2.4_MANUAL_Meta-yocto
OE-CORE TRTEMP_2.4_AUTO_OE-Core
Build-Appliance TRTEMP_2.4_MANUAL_Build-Appliance
Runtime TRTEMP_2.4_AUTO_Runtime
Toaster TRTEMP_2.4_AUTO_Toaster_UI
TRTEMP_2.4_MANUAL_Toaster_UI-managed-mode
ADT TRTEMP_2.4_MANUAL_ADT
TRTEMP_2.4_AUTO_ADT
CROPS TRTEMP_2.4_AUTO_CROPS_Poky
TRTEMP_2.4_MANUAL_CROPS_eSDK
Eclipse TRTEMP_2.4_MANUAL_Eclipse
eSDK TRTEMP_2.4_AUTO_eSDK_sdkext
TRTEMP_2.4_AUTO_eSDK_oeselftest
Kernel TRTEMP_2.4_MANUAL_Kernel

Features pending to be tested

Here is a list of areas that are not yet covered and need some work, implement test cases and add it to a component that belongs or create a new test plan for new component.

ADD FEATURES !!!!!!!!!!


The complete List of QA Implementations for YP 2.4 can be found at Features Implementation by QA Team

Features Not to be tested

Approach

Tests will be conducted per the documented test cases stored in Testopia. The test manager will create test runs for each tester. The tester will execute the tests in Testopia and mark each case as Pass / Fail / Skip /Blocked. The tester should leave notes on actual results and any other relevant details when possible.

Once complete, the test manager should review the test run reports in Testopia and report back to the team by mail and in the wiki as a test report.

Pass / Fail Criteria

This criteria is defined in Yocto_Project_v2.4_Status#Yocto_Project_v2.4_Release_Criteria

Exit Criteria

As defined in QA_Master_Test_Plan#Entry and Exit Criteria

Test Deliverables

Test Environment

There are different environments and tools for every type of testing refer to QA_Master_Test_Plan#Test_Areas for more detailed information.

Schedule

M1 (May 8, 2017 to Jun. 23, 2017

M1 Feature/Tasks Development / Cutoff is Jun. 12, 2017

IDRequesterAssigneeESummary (4 tasks) SeverityPWhiteboardStatus
11244Jair GonzalezJair Gonzalez2QA: Validate Reference kit has BT audio support on gateway profileenhancementMediumRESOLVED
11246Jair GonzalezJair Gonzalez5QA: Validate Reference kit supports image installerenhancementMediumRESOLVED
10817Paul EggletonJose Perez C4oe-selftest: add a test for devtool modify on the kernelenhancementMedium+RESOLVED
11550Jose Perez CJuan Ramospoky-tiny: Add tests for poky-tiny imagesenhancementMediumVERIFIED
   11     

M1 Bugs

IDRequesterAssigneeESummary (4 tasks) SeverityPWhiteboardStatus
11436Mariano LopezHumberto Ibarra3selftest: runtime-test install test is testing two different featuresnormalMedium+RESOLVED
10848Jose Perez CJose Perez C2Add support for testing wic images on qemux86*normalMedium+RESOLVED
11300Jose Perez CJose Perez C2selftest: do_populate_sdk_ext task failing on eSDK test on GDC ABsnormalMedium+VERIFIED
10947LibertadLibertad3Reintegrate fully functional and corrected toaster automation testcases from toaster_automation_tests.pynormalMediumRESOLVED
   10     


The planed schedule for 2.4 can be found at Yocto_2.4_Schedule

Live Schedule

MILESTONE STARTED FINISHED STATUS
M1 rc1 06/15/2017 06/22/2017 DONE
M1 rc1 meta-intel 06/23/2017 06/27/2017 DONE
M2 rc2 07/26/2017 08/02/2017 DONE
M2 rc3 08/03/2017 08/10/2017 DONE
M2 rc2 meta-intel 08/11/2017 08/16/2017 DONE
M3 08/23/2017 08/30/2017 DONE
M3 rc1 meta-intel 09/01/2017 09/05/2017 DONE
M4 RC1 09/26/2017 10/06/2017 DONE
M4 RC2 10/09/2017 10/13/2017 DONE
M4 RC3 10/17/2017 10/23/2017 DONE
M4 rc1 meta-intel 10/26/2017 10/31/2017 DONE
M4 rc2 meta-intel 11/03/2017 11/06/2017 DONE

Execution History

MILESTONE REPORT
M1 https://wiki.yoctoproject.org/wiki/WW25_-_2017-06-22_-_Full_Test_Cycle_2.4_M1_rc1
M1 meta-intel https://wiki.yoctoproject.org/wiki/WW26_-_2017-06-27-_Full_Test_Cycle_-_2.4_M1_rc1_-_meta-intel
M2 https://wiki.yoctoproject.org/wiki/WW31_-_2017-07-31_-_Full_Test_Cycle_2.4_M2_rc2
M2 - rc3 https://wiki.yoctoproject.org/wiki/WW32_-_2017-08-09_-_Full_Test_Cycle_2.4_M2_rc3
M2 - rc2 meta-intel https://wiki.yoctoproject.org/wiki/WW33_-_2017-08-15-_Full_Test_Cycle_-_2.4_M2_rc2_-_meta-intel
M3 https://wiki.yoctoproject.org/wiki/WW35_-_2017-08-30_-_Full_Test_Cycle_2.4_M3
M3 - rc1 meta-intel https://wiki.yoctoproject.org/wiki/WW36_-_2017-09-05-_Full_Test_Cycle_-_2.4_M3_rc1_-_meta-intel
M4 - rc1 https://wiki.yoctoproject.org/wiki/WW40_-_2017-10-06_-_Full_Test_Cycle_2.4_RC1
M4 - rc2 https://wiki.yoctoproject.org/wiki/WW41_-_2017-10-13_-_Full_Test_Cycle_2.4_RC2
M4 - rc3 https://wiki.yoctoproject.org/wiki/WW43_-_2017-10-23_-_Full_Test_Cycle_2.4_RC3
M4 - rc1 meta-intel https://wiki.yoctoproject.org/wiki/WW44_-_2017-10-31-_Full_Test_Cycle_-_2.4_M4_rc1_-_meta-intel
M4 - rc2 meta-intel https://wiki.yoctoproject.org/wiki/WW45_-_2017-11-06-_Full_Test_Cycle_-_2.4_M4_rc2_-_meta-intel