Yocto Project 2.2 Release Test Plan: Difference between revisions

From Yocto Project
Jump to navigationJump to search
No edit summary
No edit summary
 
(124 intermediate revisions by 4 users not shown)
Line 13: Line 13:


[[QA_Master_Test_Plan#Objectives_and_Tasks]]
[[QA_Master_Test_Plan#Objectives_and_Tasks]]
= Test Strategy =
Is followed the general strategy defined on [[QA_Master_Test_Plan#Testing_Strategy]] and below is showed specific information applicable only for 2.2 Release
== Owned features ==
As QA team we are also in charge for some features to implemetn/verify
*To Verify  [[2.2_qa_owned_features_to_verify]]
*To Implement [[2.2_qa_owned_features]]
* Assigned to QA Team [[2.2_qa_assigned_bugs]]
== Bugs to Verify ==
*Bugs that need verification form QA team  [[2.2_qa_owned_bugs]]
*Old Resolved bugs to verify  [[Old_resolved_bugs_and_features]]


= Test Items =
= Test Items =
{| class="wikitable"
! style="text-align: center;" | Code of Colors
|-bgcolor="93DB70"
| style="text-align: center;" | Active
Tracked on Testpoia
|-bgcolor="67E6EC"
| style="text-align: center;" | Active
Non Tracked on Testopia
|-bgcolor="ffbf00"
| style="text-align: center;" | In Design
Will be tracked on Tesopia
|- bgcolor="gray"
| style="text-align: center;" | Inactive
|}


{| class="wikitable"
{| class="wikitable"
! COMPONENT
! COMPONENT
! TEMPLATES
! style="text-align: center;" | TEMPLATE
|-
| bgcolor="93DB70" rowspan="2" style="text-align: center;"  | ADT
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=5995 TRTEMP_2.2_MANUAL_ADT]
|-
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=5996 TRTEMP_2.2_AUTO_ADT]
|-
| bgcolor="93DB70" rowspan="11" style="text-align: center;" | BSP
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=6007 TRTEMP_2.2_MANUAL_BSP/QEMU_core-image-sato-sdk_ANYQEMU]
|-
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=6009 TRTEMP_2.2_AUTO_BSP/QEMU_core-image-sato-sdk_ANYQEMU]
|-
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=6010 TRTEMP_2.2_AUTO_BSP/QEMU_ore-image-lsb-sdk_ANYBSP]
|-
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=6011 TRTEMP_2.2_AUTO_BSP/QEMU_core-image-sato-wic_ANYBSP]
|-
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=6012 TRTEMP_2.2_MANUAL_BSP/QEMU_core2-32_MinnowMax]
|-
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=6013 TRTEMP_2.2_MANUAL_BSP/QEMU_corei7-64_MinnowMax]
|-
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=6014 TRTEMP_2.2_MANUAL_BSP/QEMU_corei7-64_NUC]
|-
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=6015 TRTEMP_2.2_MANUAL_BSP/QEMU_GenericX86-64_NUC]
|-
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=6016 TRTEMP_2.2_AUTO_BSP/QEMU_core-image-sato-sdk_ANYBSP]
|-
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=6017 TRTEMP_2.2_MANUAL_BSP/QEMU_GenericX86_MMAX32]
|-
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=6018 TRTEMP_2.2_AUTO_BSP/QEMU_core-image-sato-sdk_intel-quark]
|-
| bgcolor="93DB70" rowspan="2" style="text-align: center;" | CROPS
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=5997 TRTEMP_2.2_AUTO_CROPS]
|-
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=5998 TRTEMP_2.2_MANUAL_CROPS]
|-
|-
| ADT
| bgcolor="93DB70" style="text-align: center;" | Eclipse-Plugin
|  
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=5999 TRTEMP_2.2._MANUAL_Eclipse-Plugin]
|-
| bgcolor="93DB70" style="text-align: center;" | Meta-Yocto
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=6000 TRTEMP_2.2_MANUAL_Meta-yocto]
|-
| bgcolor="93DB70" rowspan="2" style="text-align: center;" | Toaster
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=6021 TRTEMP_2.2_AUTO_TOASTER_UI]
|-
|-
| BSP
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=6020 TRTEMP_2.2_MANUAL_TOASTER_UI-managed-mode]
|
|-
|-
| CROPS
| bgcolor="93DB70" rowspan="2" style="text-align: center;" | eSDK
|  
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=6033 TRTEMP_2.2_AUTO_eSDK]
|-
|-
| Eclipse-Plugin
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=6034 TRTEMP_2.2_MANUAL_eSDK]
|
|-
|-
| Meta-Yocto
| bgcolor="93DB70" rowspan="2" style="text-align: center;" | OE-CORE / BitBake
|  
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=6001 TRTEMP_2.2_AUTO_OE-Core]
|-
|-
| Toaster
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=6002 TRTEMP_2.2_MANUAL_BitBake]
|  
|-
| bgcolor="gray" style="text-align: center;" | Automated Build Test / Distro
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=6022 TRTEMP_2.2_AUTO_Automated-Build-Testing]
|-
|-
| eSDK
| bgcolor="93DB70" style="text-align: center;" | Runtime (Compliance)
|  
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=6004 TRTEMP_2.2_AUTO_Runtime]
|-
|-
| OE-CORE / Bitbake
| bgcolor="93DB70" style="text-align: center;" | Build  Apliance
|  
| [https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=6006 TRTEMP_2.2_MANUAL_Build-Appliance]
|-
|-
| Distro Testing
| bgcolor="67E6EC" style="text-align: center;" | pTest
|  
|  
|-
|-
| Runtime (Compliance)
| bgcolor="67E6EC" style="text-align: center;" | Performance
|  
| [https://wiki.yoctoproject.org/wiki/Performance_Test Performance Test]
|-
|-
| Build Apliance
| bgcolor="ffbf00" style="text-align: center;" | SWUpd (Software Updater)
|  
|  
|-
|-
| pTest
| bgcolor="ffbf00" style="text-align: center;" | Distributed Builds
|  
|  
|}
|}
= Test Cases =
*[[Yocto 2.2 Test Cases]]


= Features to be tested =
= Features to be tested =


Features to be tested are defined in component Test Plan [[QA_Master_Test_Plan#Component]]
All component to be tested on 2.2 are in [[#Test Items]] a detailed list of features per component can be found in Test Plan of the components [[QA_Master_Test_Plan#Component]]
 
 
= Features pending to be tested =
 
Here is a list of areas that are not yet covered and need some work, implement test cases and add it to a component that belongs or create a new test plan for new component.
 
 
* build-appliance - can we run builds in the build-appliance image?
* Verify if  -c menuconfig work?
* Verify if -c devshell work?
* Verify if BBPOSTCONF and BBPRECONF work in bitbake?
* Verify if oe-pkgdata-util print help when no parameters specified?
* Verify if parsing time is improved with a hot codeparser cache (build metrics did catch a regression here but we had to manually spot it).
* Verify Bitbake sane error if a layer doesn't exist? (no backtrace?) send-error-report - no unittesting, only caught python3 issues when it failed to send errors
* Verify if buildhistory_analsysis.py is tested
* Verify if compression image types is tested
* Verify combo-layer
* Verify yocto-bsp/yocto-layer 
* Verify oe-selftest coverage
* Verify bitbake -u depexp -g XXX


= Pass / Fail Criteria =
= Pass / Fail Criteria =
Line 67: Line 166:
= Exit Criteria =
= Exit Criteria =


TBD
As defined in [[QA_Master_Test_Plan#Entry and Exit Criteria]]


= Test Deliverables =
= Test Deliverables =


* Live Status [[2.2_QA_Status]]
* Final execution Report [[#Execution History]]
* Mail sent to the mailing list


= Test Environment =
= Test Environment =
There are different environments and tools for every type of testing refer to [[QA_Master_Test_Plan#Test_Areas]] for more detailed information.


= Schedule =
= Schedule =


= Test Responsible =
The planed schedule for 2.2 can be found at [[Yocto_2.2_Schedule]]
 
== Live Schedule ==
 
{| class="wikitable"
! MILESTONE
! STARTED
! FINISHED
! STATUS
|- bgcolor="00FF00"
| M1.rc2
| 2016-06-21
| 2016-06-29
| DONE
|- bgcolor="00FF00"
| M2.rc1
| 2016-07-21
| 2016-06-28
| DONE
|- bgcolor="00FF00"
| M2 Meta Intel.
| 2016-08-04
| 2016-08-10
| DONE
|- bgcolor="00FF00"
| 2.2 Master: f038f0699.
| 2016-08-19
| 2016-08-26
| DONE
|- bgcolor="00FF00"
| M3 rc1
| 2016-09-09
| 2016-09-20
| DONE
|- bgcolor="00FF00"
| M4 rc4
| 2016-10-18
| 2016-10-25
| DONE
|- bgcolor="00FF00"
| M4 rc6
| 2016-10-26
| 2016-10-28
| DONE
|- bgcolor="00FF00"
| 2.2rc1 Meta Intel 6.0
| 2016-11-03
| 2016-11-10
| DONE
|- bgcolor="00FF00"
| 2.2rc2 Meta Intel 6.0
| 2016-11-10
| 2016-11-16
| DONE
|- bgcolor="00FF00"
| 2.2.1 rc1 Point Release
| 2017-02-13
| 2017-02-20
| DONE
|- bgcolor="00FF00"
| 2.2.2 rc1 Point Release
| 2017-05-29
| 2017-06-02
| DONE
|- bgcolor="00FF00"
| 2.2.2 rc2 Point Release
| 2017-09-08
| 2017-09-13
| DONE
|- bgcolor="00FF00"
| 2.2.3 rc2 Point Release
| 2018-01-22
| 2018-02-23
| DONE
|- bgcolor="00FF00"
| 2.2.4 rc1 Point Release
| 2018-05-25
| 2018-07-13
| DONE
|}
 
= Execution History =
 
 
{| class="wikitable"
! MILESTONE
! REPORT
|-
| M1.rc2
| [[WW26 - 2016-06-21 - Full Test Cycle 2.2 M1.rc2]]
|-
| M2.rc1
| [[WW30 - 2016-07-21 - Full Test Cycle 2.2 M2.rc1]]
|-
| M2 Meta-Intel
| [[WW33 - 2016-08-04 - Full Test Cycle - 2.2 M2 - meta-intel]]
|-
| 2.2 Master: f038f0699
| [[ WW35 - 2016-08-19 - Test Cycle - 2.2_Master ]]
|-
| M3
| [[ WW38 - 2016-09-09 - Full Test Cycle 2.2_M3.rc1 ]]
|-
| 2.2 rc4
| [[ WW43 - 2016-10-18 - Full Test Cycle 2.2 rc4 ]]
|-
| 2.2 rc6
| [[ WW44 - 2016-10-26 - Full Test Cycle 2.2 rc6 ]]
|-
| 2.2rc1 Meta Intel 6.0
| [[ WW46 - 2016-11-03 - Full Test Cycle - 2.2_rc1 - meta-intel-6.0]]
|-
| 2.2rc2 Meta Intel 6.0
| [[ WW48 - 2016-11-03 - Partial Test_Cycle - 2.2_rc2 - meta-intel-6.0]]
|-
| 2.2.1rc1 Point Release
| [[ WW07 - 2017-02-13 - Full Point Release Test Cycle 2.2.1 rc1]]
|-
| 2.2.2 rc1 Point Release
| [[ WW22_-_2017-06-05_-_Full_Point_Release_Test_Cycle_2.2.2_rc1]]
|-
| 2.2.2 rc2 Point Release
| [[ WW37_-_2017-09-13_-_Full_Point_Release_Test_Cycle_2.2.2_rc2]]
|-
| 2.2.3 rc2 Point Release
| [[ WW06_-_2018-02-08_-_Full_Point_Release_Test_Cycle_2.2.3_rc2 ]]
|-
| 2.2.4 rc1 Point Release
| [[ WW28_-_2018-07-13_-_Full_Test_Cycle_2.2.4_rc1 ]]
|}

Latest revision as of 05:22, 31 July 2018

Reversion history

Version Modifier Comments
1.0 Jose Perez Carranza First draft

Introduction

This article is the overall test plan for version 2.2 of the Yocto Project, for all the generic information this document will be pointing QA Master Test Plan

Objectives

QA_Master_Test_Plan#Objectives_and_Tasks

Test Strategy

Is followed the general strategy defined on QA_Master_Test_Plan#Testing_Strategy and below is showed specific information applicable only for 2.2 Release

Owned features

As QA team we are also in charge for some features to implemetn/verify

Bugs to Verify

Test Items

Code of Colors
Active

Tracked on Testpoia

Active

Non Tracked on Testopia

In Design

Will be tracked on Tesopia

Inactive


COMPONENT TEMPLATE
ADT TRTEMP_2.2_MANUAL_ADT
TRTEMP_2.2_AUTO_ADT
BSP TRTEMP_2.2_MANUAL_BSP/QEMU_core-image-sato-sdk_ANYQEMU
TRTEMP_2.2_AUTO_BSP/QEMU_core-image-sato-sdk_ANYQEMU
TRTEMP_2.2_AUTO_BSP/QEMU_ore-image-lsb-sdk_ANYBSP
TRTEMP_2.2_AUTO_BSP/QEMU_core-image-sato-wic_ANYBSP
TRTEMP_2.2_MANUAL_BSP/QEMU_core2-32_MinnowMax
TRTEMP_2.2_MANUAL_BSP/QEMU_corei7-64_MinnowMax
TRTEMP_2.2_MANUAL_BSP/QEMU_corei7-64_NUC
TRTEMP_2.2_MANUAL_BSP/QEMU_GenericX86-64_NUC
TRTEMP_2.2_AUTO_BSP/QEMU_core-image-sato-sdk_ANYBSP
TRTEMP_2.2_MANUAL_BSP/QEMU_GenericX86_MMAX32
TRTEMP_2.2_AUTO_BSP/QEMU_core-image-sato-sdk_intel-quark
CROPS TRTEMP_2.2_AUTO_CROPS
TRTEMP_2.2_MANUAL_CROPS
Eclipse-Plugin TRTEMP_2.2._MANUAL_Eclipse-Plugin
Meta-Yocto TRTEMP_2.2_MANUAL_Meta-yocto
Toaster TRTEMP_2.2_AUTO_TOASTER_UI
TRTEMP_2.2_MANUAL_TOASTER_UI-managed-mode
eSDK TRTEMP_2.2_AUTO_eSDK
TRTEMP_2.2_MANUAL_eSDK
OE-CORE / BitBake TRTEMP_2.2_AUTO_OE-Core
TRTEMP_2.2_MANUAL_BitBake
Automated Build Test / Distro TRTEMP_2.2_AUTO_Automated-Build-Testing
Runtime (Compliance) TRTEMP_2.2_AUTO_Runtime
Build Apliance TRTEMP_2.2_MANUAL_Build-Appliance
pTest
Performance Performance Test
SWUpd (Software Updater)
Distributed Builds

Test Cases

Features to be tested

All component to be tested on 2.2 are in #Test Items a detailed list of features per component can be found in Test Plan of the components QA_Master_Test_Plan#Component


Features pending to be tested

Here is a list of areas that are not yet covered and need some work, implement test cases and add it to a component that belongs or create a new test plan for new component.


  • build-appliance - can we run builds in the build-appliance image?
  • Verify if -c menuconfig work?
  • Verify if -c devshell work?
  • Verify if BBPOSTCONF and BBPRECONF work in bitbake?
  • Verify if oe-pkgdata-util print help when no parameters specified?
  • Verify if parsing time is improved with a hot codeparser cache (build metrics did catch a regression here but we had to manually spot it).
  • Verify Bitbake sane error if a layer doesn't exist? (no backtrace?) send-error-report - no unittesting, only caught python3 issues when it failed to send errors
  • Verify if buildhistory_analsysis.py is tested
  • Verify if compression image types is tested
  • Verify combo-layer
  • Verify yocto-bsp/yocto-layer
  • Verify oe-selftest coverage
  • Verify bitbake -u depexp -g XXX

Pass / Fail Criteria

This criteria is defined in Yocto_Project_v2.2_Status#Yocto_Project_v2.2_Release_Criteria

Exit Criteria

As defined in QA_Master_Test_Plan#Entry and Exit Criteria

Test Deliverables

Test Environment

There are different environments and tools for every type of testing refer to QA_Master_Test_Plan#Test_Areas for more detailed information.

Schedule

The planed schedule for 2.2 can be found at Yocto_2.2_Schedule

Live Schedule

MILESTONE STARTED FINISHED STATUS
M1.rc2 2016-06-21 2016-06-29 DONE
M2.rc1 2016-07-21 2016-06-28 DONE
M2 Meta Intel. 2016-08-04 2016-08-10 DONE
2.2 Master: f038f0699. 2016-08-19 2016-08-26 DONE
M3 rc1 2016-09-09 2016-09-20 DONE
M4 rc4 2016-10-18 2016-10-25 DONE
M4 rc6 2016-10-26 2016-10-28 DONE
2.2rc1 Meta Intel 6.0 2016-11-03 2016-11-10 DONE
2.2rc2 Meta Intel 6.0 2016-11-10 2016-11-16 DONE
2.2.1 rc1 Point Release 2017-02-13 2017-02-20 DONE
2.2.2 rc1 Point Release 2017-05-29 2017-06-02 DONE
2.2.2 rc2 Point Release 2017-09-08 2017-09-13 DONE
2.2.3 rc2 Point Release 2018-01-22 2018-02-23 DONE
2.2.4 rc1 Point Release 2018-05-25 2018-07-13 DONE

Execution History

MILESTONE REPORT
M1.rc2 WW26 - 2016-06-21 - Full Test Cycle 2.2 M1.rc2
M2.rc1 WW30 - 2016-07-21 - Full Test Cycle 2.2 M2.rc1
M2 Meta-Intel WW33 - 2016-08-04 - Full Test Cycle - 2.2 M2 - meta-intel
2.2 Master: f038f0699 WW35 - 2016-08-19 - Test Cycle - 2.2_Master
M3 WW38 - 2016-09-09 - Full Test Cycle 2.2_M3.rc1
2.2 rc4 WW43 - 2016-10-18 - Full Test Cycle 2.2 rc4
2.2 rc6 WW44 - 2016-10-26 - Full Test Cycle 2.2 rc6
2.2rc1 Meta Intel 6.0 WW46 - 2016-11-03 - Full Test Cycle - 2.2_rc1 - meta-intel-6.0
2.2rc2 Meta Intel 6.0 WW48 - 2016-11-03 - Partial Test_Cycle - 2.2_rc2 - meta-intel-6.0
2.2.1rc1 Point Release WW07 - 2017-02-13 - Full Point Release Test Cycle 2.2.1 rc1
2.2.2 rc1 Point Release WW22_-_2017-06-05_-_Full_Point_Release_Test_Cycle_2.2.2_rc1
2.2.2 rc2 Point Release WW37_-_2017-09-13_-_Full_Point_Release_Test_Cycle_2.2.2_rc2
2.2.3 rc2 Point Release WW06_-_2018-02-08_-_Full_Point_Release_Test_Cycle_2.2.3_rc2
2.2.4 rc1 Point Release WW28_-_2018-07-13_-_Full_Test_Cycle_2.2.4_rc1
Retrieved from "https://wiki.yoctoproject.org/wiki/index.php?title=Yocto_Project_2.2_Release_Test_Plan&oldid=40864"