View source for Yocto Project 2.6 Release Test Plan

<!-- Variables for Template -->
<!-- ********************** -->
{|
{{#vardefine:ADT_MAUAL            | 6539}}
{{#vardefine:ADT_AUTO             | 6540}}
{{#vardefine:ANYQEMU_SATO_MANUAL  | 6563}}
{{#vardefine:ANYQEMU_SATO_AUTO    | 6564}}
{{#vardefine:ANYBSP_SATO_AUTO     | 6558}}
{{#vardefine:ANYBSP_LSB_AUTO      | 6561}}
{{#vardefine:ANYBSP_WIC_AUTO      | 6560}}
{{#vardefine:MMAX_CORE2_MANUAL    | 6554}}
{{#vardefine:MMAX_I7_MANUAL       | 6557}}
{{#vardefine:NUC_I7_MANUAL        | 6556}}
{{#vardefine:NUC_GEX86-64_MANUAL  | 6555}}
{{#vardefine:MMAX_GEX86-32_MANUAL | 6553}}
{{#vardefine:GAL_SATO_AUTO        | 6565}}
{{#vardefine:CROPS_AUTO           | 6541}}
{{#vardefine:CROPS_MANUAL         | 6542}}
{{#vardefine:ECLIPSE_MANUAL       | 6543}}
{{#vardefine:KERNEL_MANUAL        | 6714}}
{{#vardefine:METAYOCTO_MANUAL     | 6544}}
{{#vardefine:TOASTER_AUTO         | 6545}}
{{#vardefine:TOASTER_MANUAL       | 6546}}
{{#vardefine:ESDK_MANUAL          | 6547}}
{{#vardefine:ESDK_AUTO            | 6548}}
{{#vardefine:OE-CORE_AUTO         | 6549}}
{{#vardefine:BITBAKE_AUTO         | 6550}}
{{#vardefine:RUNTIME_AUTO         | 6551}}
{{#vardefine:BUILD-APPLIANCE_AUTO | 6552}}
|}
<!-- ********************* -->

= Reversion history =
{|class="wikitable"
! Version || Modifier || Comments
|-
| 1.0 || Yeoh Ee Peng ||  First draft
|-
|}

=References=
Documents that support this test plan 

= Introduction =


This is the the overall test plan for version 2.6  of the {{ns:4}}, for all the generic information this document will be pointing to [[QA Master Test Plan]]

= Objectives =

This test plan for Yocto Project 2.6 release supports the following objectives:
*To define the tools to be used throughout the testing process.
*To communicate to the responsible parties the items to be tested, set expectations around schedule, and define environmental needs.
*To define how the tests will be conducted.
*Release Criteria
*Risks and contingencies

[[QA_Master_Test_Plan#Objectives_and_Tasks]]

= Risk Analysis & Contingency Plan =

This is the document for Risk Analysis https://docs.google.com/spreadsheets/d/1K-MZYo5sfabwI2UozGp6Uuh2sw5y309bXkyEZRPEEgk/edit#gid=0

= Test Strategy =

Is followed the general strategy defined on [[QA_Master_Test_Plan#Testing_Strategy]] and below is showed specific information applicable only for 2.6 Release

== Bugs Tracking ==

*Bugs that need to be implemented by QA Team
**[[2.6_QA_OWNED_BUGS | 2.6 QA Assigned Bugs]]

*Bugs that need to be verified by QA Team 
**[[2.6_QA_Bugs_To_Verify]]
<!--
*Features to verify 
**[[2.2 qa_owned features to verify]]

*Features to implement 
**[[2.2 qa owned features]]
-->

*Old Bugs that need to be Verified
**[[Old resolved bugs and features]]

= Test Items =
The Yocto Project QA team verifies each release against recent versions of the most popular Linux distributions that provide stable releases. 
 * centos 7 
 * Debian 9
 * Fedora 26
 * Fedora 27
 * OpenSuse42.3
 * ubuntu 16.04 
 * ubuntu 17.10

= Test Cases =

*[[Yocto 2.6 Test Cases]]


= Test Matrix =

The components to be tested  for version 2.6  of the {{ns:4}} against the different milestones  are included in this section:

{| class="wikitable"
! style="text-align: center;" | Code of Colors
|-bgcolor="93DB70"
| style="text-align: center;" | Active
Tracked on Testpoia
|-bgcolor="67E6EC" 
| style="text-align: center;" | Active
Non Tracked on Testopia
|-bgcolor="ffbf00"
| style="text-align: center;" | In Design
Will be tracked on Tesopia
|- bgcolor="gray"
| style="text-align: center;" | Inactive
|}


== ADT ==

{| class="wikitable"
! style="text-align: center; font-weight: bold;" | Component
! style="text-align: center; font-weight: bold;" | Environment
! style="text-align: center; font-weight: bold;" | Auto
! style="text-align: center; font-weight: bold;" | Manual
! style="text-align: center; font-weight: bold;" | Owner
|-
| rowspan="2" style="text-align: center;" | ADT
| Fedora 25 i686
| Y
| Y
| PG
|-
| Ubuntu 16.04 x86_64
| Y
| Y
| PG
|}

== Eclipse Plugin ==

{| class="wikitable"
! style="text-align: center;" | Component
! style="text-align: center;" | Environment
! style="text-align: center;" | Auto
! style="text-align: center;" | Manual
! style="text-align: center;" | Owner
|-
| Neon + Any Distro
| 
|
| Y
| PG
|-
| Oxygen + Any Distro
|
| 
| Y
| PG
|}

== Kernel ==

{| class="wikitable"
! style="text-align: center;" | Component
! style="text-align: center;" | Environment
! style="text-align: center;" | Auto
! style="text-align: center;" | Manual
! style="text-align: center;" | Owner
|-
| Kernel
| Any Distro
| 
| Y
| PG
|}

== Meta-Yocto ==

{| class="wikitable"
! style="text-align: center;" | Component
! style="text-align: center;" | Environment
! style="text-align: center;" | Auto
! style="text-align: center;" | Manual
! style="text-align: center;" | Owner
|-
| Meta-Yocto
| Any Distro
| 
| Y
| PG
|}

== Toaster ==

{| class="wikitable"
! style="text-align: center;" | Component
! style="text-align: center;" | Environment
! style="text-align: center;" | Auto
! style="text-align: center;" | Manual
! style="text-align: center;" | Owner
|-
| Toaster
| Any Distro
| 
| Y
| PG
|}

== eSDK ==

{| class="wikitable"
! style="text-align: center;" | Component
! style="text-align: center;" | Environment
! style="text-align: center;" | Auto
! style="text-align: center;" | Manual
! style="text-align: center;" | Owner
|-
| rowspan="2" style="text-align: center;" | eSDK
| Any Distro + runtime
| Y
| 
| PG
|-
| Any Distro + selftest
| Y
| 
| PG
|}

== Selftest ==

{| class="wikitable"
! style="text-align: center;" | Component
! style="text-align: center;" | Environment
! style="text-align: center;" | Auto
! style="text-align: center;" | Manual
! style="text-align: center;" | Owner
|-
| rowspan="8" style="text-align: center;" | OE-CORE
| Ubuntu 16.04
| Y
| 
| PG
|-
| Ubuntu 17.10
| Y
| 
| PG
|-
| OpenSuse 42.3
| Y
| 
| PG
|-
| Debian 9
| Y
| 
| PG
|-
| Centos 7
| Y
| 
| PG
|-
| Fedora 26
| Y
| 
| PG
|-
| Fedora 27
| Y
| 
| PG
|}

== Bitbake ==

{| class="wikitable"
! style="text-align: center;" | Component
! style="text-align: center;" | Environment
! style="text-align: center;" | Auto
! style="text-align: center;" | Manual
! style="text-align: center;" | Owner
|-
| Bitbake
| Any Distro
| 
| Y
| PG
|}

== Build - Apliance ==

{| class="wikitable"
! style="text-align: center;" | Component
! style="text-align: center;" | Environment
! style="text-align: center;" | Auto
! style="text-align: center;" | Manual
! style="text-align: center;" | Owner
|-
| Build - Apliance
| VM - Any Distro
| 
| Y
| PG
|}

== Package Manager ==

{| class="wikitable"
! style="text-align: center;" | Component
! style="text-align: center;" | Environment
! style="text-align: center;" | Auto
! style="text-align: center;" | Manual
! style="text-align: center;" | Owner
|-
| Package Manager
| Any Distro
| Y
| 
| PG
|}

== Performance == 

{| class="wikitable"
! style="text-align: center;" | Component
! style="text-align: center;" | Environment
! style="text-align: center;" | Auto
! style="text-align: center;" | Manual
! style="text-align: center;" | Owner
|-
| rowspan="2" style="text-align: center;" | Performance
| Centos 7
| Y
| 
| PG
|-
| Ubuntu 16.04
| Y
| 
| PG
|}

== BSP/QEMU ==

{| class="wikitable"
! Component
! Arch
! HW
! Environment
! Image
! Auto
! Manual
! pTest
! Compliance
|-
| rowspan="30" style="text-align: center;" | BSP
| rowspan="12" style="text-align: center;" | x86
| rowspan="2" style="text-align: center;" | MinwTurbot 32bit
| rowspan="2" style="text-align: center;" | genericx86
| core-image-sato-sdk
| Y
| 
| 
| 
|-
| core-image-lsb-sdk
| Y
| 
| 
| 
|-
| rowspan="2" style="text-align: center;" | MinwTurbot 64bit
| rowspan="2" style="text-align: center;" | genericx86-64
| core-image-sato-sdk
| Y
| Y
| 
| 
|-
| core-image-lsb-sdk
| Y
| 
| 
| 
|-
| rowspan="2" style="text-align: center;" | NUC 6
| rowspan="2" style="text-align: center;" | genericx86-64
| core-image-sato-sdk
| Y
| 
| Y
| 
|-
| core-image-lsb-sdk
| Y
| 
| 
| Y
|-
| rowspan="2" style="text-align: center;" | NUC 7
| rowspan="2" style="text-align: center;" | genericx86-64
| core-image-sato-sdk
| Y
| Y
| 
| 
|-
| core-image-lsb-sdk
| Y
| 
| 
| 
|-
| rowspan="4" style="text-align: center;" | QEMU
| rowspan="2" style="text-align: center;" | qemux86
| core-image-sato-sdk
| Y
| Y
| 
| 
|-
| core-image-lsb-sdk
| Y
| 
| 
| 
|-
| rowspan="2" style="text-align: center;" | qemux86-64
| core-image-sato-sdk
| Y
| Y
| 
| 
|-
| core-image-lsb-sdk
| Y
| 
| 
| 
|-
| rowspan="8" style="text-align: center;" | non x86
| style="text-align: center;" | EdgeRouter
| EdgeRouter
| core-image-sato-sdk
| Y
| Y
| 
| 
|-
| style="text-align: center;" | MPC8315e-rdb
| MPC8315e-rdb
| core-image-sato-sdk
| Y
| Y
| 
| 
|-
| style="text-align: center;" | Beaglebone Black
| Beaglebone
| core-image-sato-sdk
| Y
| Y
| 
| 
|-
| rowspan="5" style="text-align: center;" | QEMU
| qemuarm
| core-image-sato-sdk
| Y
| Y
| 
| 
|-
| qemuarm-64
| core-image-sato-sdk
| Y
| Y
| 
| 
|-
| qemuppc
| core-image-sato-sdk
| Y
| Y
| 
| 
|-
| qemumips
| core-image-sato-sdk
| Y
| Y
| 
| 
|-
| qemumips64
| core-image-sato-sdk
| Y
| Y
| 
| 
|-
| rowspan="10" style="text-align: center;" | Meta-Intel
| rowspan="2" style="text-align: center;" | MinwTurbot 32bit
| rowspan="2" style="text-align: center;" | core2-32
| core-image-sato-sdk
| Y
| 
| 
| 
|-
| core-image-lsb-sdk
| Y
| 
| 
| 
|-
| rowspan="2" style="text-align: center;" | MinwTurbot 64bit
| rowspan="2" style="text-align: center;" | corei7-64
| core-image-sato-sdk
| Y
| Y
| 
| 
|-
| core-image-lsb-sdk
| Y
| 
| 
| 
|-
| rowspan="2" style="text-align: center;" | NUC 6
| rowspan="2" style="text-align: center;" | corei7-64
| core-image-sato-sdk
| Y
| 
| 
| 
|-
| core-image-lsb-sdk
| Y
| 
| 
| 
|-
| rowspan="3" style="text-align: center;" | NUC 7
| rowspan="3" style="text-align: center;" | corei7-64
| core-image-sato-sdk
| Y
| Y
| 
| 
|-
| core-image-lsb-sdk
| Y
| 
| 
| 
|-
| poky-tiny
| 
| Y
| 
| 
|}

All component to be tested on 2.6 are in [[#Test Items]] a detailed list of features per component can be found in Test Plan of the components [[QA_Master_Test_Plan#Component]]

= Testopia Templates =

{| class="wikitable"
! style="font-weight: bold;" | COMPONENT
! style="font-weight: bold;" | TEMPLATE
|-
| BitBake
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7403 TRTEMP_2.6_MANUAL_BitBake]
|-
| rowspan="15" | BSP/QEMU
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7404 TRTEMP_2.6_MANUAL_BSP/QEMU_core-image-sato-sdk_Genericx86_MMAX32]
|-
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7405 TRTEMP_2.6_MANUAL_BSP/QEMU_core-image-sato-sdk_core2-32_MMAX32]
|-
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7406 TRTEMP_2.6_MANUAL_BSP/QEMU_core-image-sato-sdk_Genericx86-64_NUC]
|-
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7407 TRTEMP_2.6_MANUAL_BSP/QEMU_core-image-sato-sdk_corei7-64_NUC]
|-
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7408 TRTEMP_2.6_MANUAL_BSP/QEMU_core-image-sato-sdk_corei7-64_MMAX64]
|-
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7409 TRTEMP_2.6_AUTO_BSP/QEMU_core-image-sato-sdk_ANYBSP]
|-
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7410 TRTEMP_2.6_AUTO_BSP/QEMU_core-image-sato-sdk_wic_ANYBSP]
|-
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7411 TRTEMP_2.6_AUTO_BSP/QEMU_core-image-lsb-sdk_ANYBSP]
|-
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7412 TRTEMP_2.6_MANUAL_BSP/QEMU_core-image-sato-sdk_ANYQEMU]
|-
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7413 TRTEMP_2.6_AUTO_BSP/QEMU_core-image-sato-sdk_ANYQEMU]
|-
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7414 TRTEMP_2.6_AUTO_BSP/QEMU_core-image-sato-sdk_intel-quark]
|-
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7415 TRTEMP_2.6_MANUAL_BSP/QEMU_core-image-sato-sdk_Beaglebone-Black]
|-
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7416 TRTEMP_2.6_MANUAL_BSP/QEMU_core-image-sato-sdk_EdgeRouter]
|-
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7417 TRTEMP_2.6_MANUAL_BSP/QEMU_core-image-sato-sdk_MPC8315e-rdb]
|-
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7418 TRTEMP_2.6_AUTO_BSP/QEMU_core-image-lsb-sdk_ANYQEMU]
|-
| Meta-Yocto
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7419 TRTEMP_2.6_MANUAL_Meta-yocto]
|-
| OE-CORE
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7420 TRTEMP_2.6_AUTO_OE-Core]
|-
| Build-Appliance
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7421 TRTEMP_2.6_MANUAL_Build-Appliance]
|-
| Runtime
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7422 TRTEMP_2.6_AUTO_Runtime]
|-
| rowspan="2" | Toaster
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7423 TRTEMP_2.6_AUTO_Toaster_UI]
|-
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7424 TRTEMP_2.6_MANUAL_Toaster_UI-managed-mode]
|-
| rowspan="2" | ADT
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7425 TRTEMP_2.6_MANUAL_ADT]
|-
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7426 TRTEMP_2.6_AUTO_ADT]
|-
| rowspan="2" | CROPS
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7430 TRTEMP_2.6_AUTO_CROPS_Poky]
|-
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7431 TRTEMP_2.6_MANUAL_CROPS_eSDK]
|-
| Eclipse
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7432 TRTEMP_2.6_MANUAL_Eclipse]
|-
| rowspan="2" | eSDK
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7433 TRTEMP_2.6_AUTO_eSDK_sdkext]
|-
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7434 TRTEMP_2.6_AUTO_eSDK_oeselftest]
|-
| Kernel
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7435 TRTEMP_2.6_MANUAL_Kernel]
|-
| Package Management
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=7489 TRTEMP_2.6_MANUAL_Package-Management-Updates]
|}

= Features pending to be tested =

Here is a list of areas that are not yet covered and need some work, implement test cases and add it to a component that belongs or create a new test plan for new component.

'''ADD FEATURES !!!!!!!!!!'''


The complete List of QA Implementations for YP 2.6 can be found at [https://wiki.yoctoproject.org/wiki/2.6_QA_OWNED_BUGS Features Implementation by QA Team]

= Features Not to be tested =
*

=Approach=
Tests will be conducted per the documented test cases stored in Testopia. The test manager will create test runs for each tester. The tester will execute the tests in Testopia and mark each case as Pass / Fail / Skip /Blocked.
The tester should leave notes on actual results and any other relevant details when possible. 

Once complete, the test manager should review the test run reports in Testopia and report back to the team by mail and in the wiki as a test report.

= Pass / Fail Criteria =

This criteria is defined in [[Yocto_Project_v2.6_Status#Yocto_Project_v2.6_Release_Criteria]]

= Exit Criteria =

As defined in [[QA_Master_Test_Plan#Entry and Exit Criteria]]

= Test Deliverables =


* Live Status [[2.6_QA_Status]]
* Final execution Report [[#Execution History]]
* Mail sent to the mailing list

= Test Environment =

There are different environments and tools for every type of testing refer to [[QA_Master_Test_Plan#Test_Areas]] for more detailed information.

= Schedule =
== M1 (Apr 30, 2018 to Jun 22, 2018)  ==
=== M1 Feature/Tasks Development / Cutoff is Jun. 11, 2017 ===

{{#bugzilla:
  |columns=id,from,to,estimated,summary,severity,priority,whiteboard,status
  |total=estimated
  |milestone=2.6 M1, 2.6 M2
  |severity=enhancement
  |sort=to,priority
}}

=== M1 Bugs ===

{{#bugzilla:
  |columns=id,from,to,estimated,summary,severity,priority,whiteboard,status
  |total=estimated
  |milestone=2.6 M1, 2.6 M2
  |severity=!enhancement
  |sort=to,priority
}}


The planed schedule for 2.6 can be found at [[Yocto_2.6_Schedule]]

== Live Schedule & Execution History ==

{| class="wikitable"
! MILESTONE
! STARTED
! FINISHED
! STATUS
! REPORT
|- bgcolor="93DB70"
| M1 rc1
| 6/19/2018
| 7/2/2018
| DONE
| https://wiki.yoctoproject.org/wiki/WW27_-_2018-07-02_-_Full_Test_Cycle_2.6_M1_rc1
|- bgcolor="grey"
|}