Latest revision as of 05:22, 31 July 2018

Reversion history

Version	Modifier	Comments
1.0	Jose Perez Carranza	First draft

Introduction

This article is the overall test plan for version 2.2 of the Yocto Project, for all the generic information this document will be pointing QA Master Test Plan

Objectives

QA_Master_Test_Plan#Objectives_and_Tasks

Test Strategy

Is followed the general strategy defined on QA_Master_Test_Plan#Testing_Strategy and below is showed specific information applicable only for 2.2 Release

Owned features

As QA team we are also in charge for some features to implemetn/verify

To Verify 2.2_qa_owned_features_to_verify

To Implement 2.2_qa_owned_features

Assigned to QA Team 2.2_qa_assigned_bugs

Bugs to Verify

Bugs that need verification form QA team 2.2_qa_owned_bugs

Old Resolved bugs to verify Old_resolved_bugs_and_features

Test Items

Code of Colors
Active Tracked on Testpoia
Active Non Tracked on Testopia
In Design Will be tracked on Tesopia
Inactive

COMPONENT	TEMPLATE
ADT	TRTEMP_2.2_MANUAL_ADT
ADT	TRTEMP_2.2_AUTO_ADT
BSP	TRTEMP_2.2_MANUAL_BSP/QEMU_core-image-sato-sdk_ANYQEMU
	TRTEMP_2.2_AUTO_BSP/QEMU_core-image-sato-sdk_ANYQEMU
	TRTEMP_2.2_AUTO_BSP/QEMU_ore-image-lsb-sdk_ANYBSP
	TRTEMP_2.2_AUTO_BSP/QEMU_core-image-sato-wic_ANYBSP
	TRTEMP_2.2_MANUAL_BSP/QEMU_core2-32_MinnowMax
	TRTEMP_2.2_MANUAL_BSP/QEMU_corei7-64_MinnowMax
	TRTEMP_2.2_MANUAL_BSP/QEMU_corei7-64_NUC
	TRTEMP_2.2_MANUAL_BSP/QEMU_GenericX86-64_NUC
	TRTEMP_2.2_AUTO_BSP/QEMU_core-image-sato-sdk_ANYBSP
	TRTEMP_2.2_MANUAL_BSP/QEMU_GenericX86_MMAX32
	TRTEMP_2.2_AUTO_BSP/QEMU_core-image-sato-sdk_intel-quark
CROPS	TRTEMP_2.2_AUTO_CROPS
CROPS	TRTEMP_2.2_MANUAL_CROPS
Eclipse-Plugin	TRTEMP_2.2._MANUAL_Eclipse-Plugin
Meta-Yocto	TRTEMP_2.2_MANUAL_Meta-yocto
Toaster	TRTEMP_2.2_AUTO_TOASTER_UI
Toaster	TRTEMP_2.2_MANUAL_TOASTER_UI-managed-mode
eSDK	TRTEMP_2.2_AUTO_eSDK
eSDK	TRTEMP_2.2_MANUAL_eSDK
OE-CORE / BitBake	TRTEMP_2.2_AUTO_OE-Core
OE-CORE / BitBake	TRTEMP_2.2_MANUAL_BitBake
Automated Build Test / Distro	TRTEMP_2.2_AUTO_Automated-Build-Testing
Runtime (Compliance)	TRTEMP_2.2_AUTO_Runtime
Build Apliance	TRTEMP_2.2_MANUAL_Build-Appliance
pTest
Performance	Performance Test
SWUpd (Software Updater)
Distributed Builds

Test Cases

Yocto 2.2 Test Cases

Features to be tested

All component to be tested on 2.2 are in #Test Items a detailed list of features per component can be found in Test Plan of the components QA_Master_Test_Plan#Component

Features pending to be tested

Here is a list of areas that are not yet covered and need some work, implement test cases and add it to a component that belongs or create a new test plan for new component.

build-appliance - can we run builds in the build-appliance image?
Verify if -c menuconfig work?
Verify if -c devshell work?
Verify if BBPOSTCONF and BBPRECONF work in bitbake?
Verify if oe-pkgdata-util print help when no parameters specified?
Verify if parsing time is improved with a hot codeparser cache (build metrics did catch a regression here but we had to manually spot it).
Verify Bitbake sane error if a layer doesn't exist? (no backtrace?) send-error-report - no unittesting, only caught python3 issues when it failed to send errors
Verify if buildhistory_analsysis.py is tested
Verify if compression image types is tested
Verify combo-layer
Verify yocto-bsp/yocto-layer
Verify oe-selftest coverage
Verify bitbake -u depexp -g XXX

Pass / Fail Criteria

This criteria is defined in Yocto_Project_v2.2_Status#Yocto_Project_v2.2_Release_Criteria

Exit Criteria

As defined in QA_Master_Test_Plan#Entry and Exit Criteria

Test Deliverables

Live Status 2.2_QA_Status
Final execution Report #Execution History
Mail sent to the mailing list

Test Environment

There are different environments and tools for every type of testing refer to QA_Master_Test_Plan#Test_Areas for more detailed information.

Schedule

The planed schedule for 2.2 can be found at Yocto_2.2_Schedule

Live Schedule

MILESTONE	STARTED	FINISHED	STATUS
M1.rc2	2016-06-21	2016-06-29	DONE
M2.rc1	2016-07-21	2016-06-28	DONE
M2 Meta Intel.	2016-08-04	2016-08-10	DONE
2.2 Master: f038f0699.	2016-08-19	2016-08-26	DONE
M3 rc1	2016-09-09	2016-09-20	DONE
M4 rc4	2016-10-18	2016-10-25	DONE
M4 rc6	2016-10-26	2016-10-28	DONE
2.2rc1 Meta Intel 6.0	2016-11-03	2016-11-10	DONE
2.2rc2 Meta Intel 6.0	2016-11-10	2016-11-16	DONE
2.2.1 rc1 Point Release	2017-02-13	2017-02-20	DONE
2.2.2 rc1 Point Release	2017-05-29	2017-06-02	DONE
2.2.2 rc2 Point Release	2017-09-08	2017-09-13	DONE
2.2.3 rc2 Point Release	2018-01-22	2018-02-23	DONE
2.2.4 rc1 Point Release	2018-05-25	2018-07-13	DONE

Execution History

MILESTONE	REPORT
M1.rc2	WW26 - 2016-06-21 - Full Test Cycle 2.2 M1.rc2
M2.rc1	WW30 - 2016-07-21 - Full Test Cycle 2.2 M2.rc1
M2 Meta-Intel	WW33 - 2016-08-04 - Full Test Cycle - 2.2 M2 - meta-intel
2.2 Master: f038f0699	WW35 - 2016-08-19 - Test Cycle - 2.2_Master
M3	WW38 - 2016-09-09 - Full Test Cycle 2.2_M3.rc1
2.2 rc4	WW43 - 2016-10-18 - Full Test Cycle 2.2 rc4
2.2 rc6	WW44 - 2016-10-26 - Full Test Cycle 2.2 rc6
2.2rc1 Meta Intel 6.0	WW46 - 2016-11-03 - Full Test Cycle - 2.2_rc1 - meta-intel-6.0
2.2rc2 Meta Intel 6.0	WW48 - 2016-11-03 - Partial Test_Cycle - 2.2_rc2 - meta-intel-6.0
2.2.1rc1 Point Release	WW07 - 2017-02-13 - Full Point Release Test Cycle 2.2.1 rc1
2.2.2 rc1 Point Release	WW22_-_2017-06-05_-_Full_Point_Release_Test_Cycle_2.2.2_rc1
2.2.2 rc2 Point Release	WW37_-_2017-09-13_-_Full_Point_Release_Test_Cycle_2.2.2_rc2
2.2.3 rc2 Point Release	WW06_-_2018-02-08_-_Full_Point_Release_Test_Cycle_2.2.3_rc2
2.2.4 rc1 Point Release	WW28_-_2018-07-13_-_Full_Test_Cycle_2.2.4_rc1

@@ Line 125: / Line 125: @@
 | [https://wiki.yoctoproject.org/wiki/Performance_Test Performance Test]
 |-
-| bgcolor="ffbf00" style="text-align: center;" | WUpd (Software Updater)
+| bgcolor="ffbf00" style="text-align: center;" | SWUpd (Software Updater)
 |
 |-
@@ Line 194: / Line 194: @@
 | 2016-06-21
 | 2016-06-29
-| [https://wiki.yoctoproject.org/wiki/WW26_-_2016-06-21_-_Full_Test_Cycle_2.2_M1.rc2]
+| DONE
 |- bgcolor="00FF00"
 | M2.rc1
 | 2016-07-21
 | 2016-06-28
-| [https://wiki.yoctoproject.org/wiki/WW30_-_2016-07-21_-_Full_Test_Cycle_2.2_M2.rc1]
+| DONE
 |- bgcolor="00FF00"
 | M2 Meta Intel.
 | 2016-08-04
 | 2016-08-10
-| [https://wiki.yoctoproject.org/wiki/WW33_-_2016-08-04_-_Full_Test_Cycle_-_2.2_M2_-_meta-intel]
+| DONE
 |- bgcolor="00FF00"
 | 2.2 Master: f038f0699.
 | 2016-08-19
 | 2016-08-26
-| [https://wiki.yoctoproject.org/wiki/2.2_QA_Status#Summary_Report]
+| DONE
-|- bgcolor="Gray"
+|- bgcolor="00FF00"
-| M3
+| M3 rc1
-|
+| 2016-09-09
-|
+| 2016-09-20
-|
+| DONE
-|- bgcolor="Gray"
+|- bgcolor="00FF00"
-| M4
+| M4 rc4
-|
+| 2016-10-18
-|
+| 2016-10-25
-|
+| DONE
+|- bgcolor="00FF00"
+| M4 rc6
+| 2016-10-26
+| 2016-10-28
+| DONE
+|- bgcolor="00FF00"
+| 2.2rc1 Meta Intel 6.0
+| 2016-11-03
+| 2016-11-10
+| DONE
+|- bgcolor="00FF00"
+| 2.2rc2 Meta Intel 6.0
+| 2016-11-10
+| 2016-11-16
+| DONE
+|- bgcolor="00FF00"
+| 2.2.1 rc1 Point Release
+| 2017-02-13
+| 2017-02-20
+| DONE
+|- bgcolor="00FF00"
+| 2.2.2 rc1 Point Release
+| 2017-05-29
+| 2017-06-02
+| DONE
+|- bgcolor="00FF00"
+| 2.2.2 rc2 Point Release
+| 2017-09-08
+| 2017-09-13
+| DONE
+|- bgcolor="00FF00"
+| 2.2.3 rc2 Point Release
+| 2018-01-22
+| 2018-02-23
+| DONE
+|- bgcolor="00FF00"
+| 2.2.4 rc1 Point Release
+| 2018-05-25
+| 2018-07-13
+| DONE
 |}
 = Execution History =
 {| class="wikitable"
@@ Line 237: / Line 278: @@
 | [[WW33 - 2016-08-04 - Full Test Cycle - 2.2 M2 - meta-intel]]
 |-
-| M4
+| 2.2 Master: f038f0699
-|
+| [[ WW35 - 2016-08-19 - Test Cycle - 2.2_Master ]]
+|-
+| M3
+| [[ WW38 - 2016-09-09 - Full Test Cycle 2.2_M3.rc1 ]]
+|-
+| 2.2 rc4
+| [[ WW43 - 2016-10-18 - Full Test Cycle 2.2 rc4 ]]
+|-
+| 2.2 rc6
+| [[ WW44 - 2016-10-26 - Full Test Cycle 2.2 rc6 ]]
+|-
+| 2.2rc1 Meta Intel 6.0
+| [[ WW46 - 2016-11-03 - Full Test Cycle - 2.2_rc1 - meta-intel-6.0]]
+|-
+| 2.2rc2 Meta Intel 6.0
+| [[ WW48 - 2016-11-03 - Partial Test_Cycle - 2.2_rc2 - meta-intel-6.0]]
+|-
+| 2.2.1rc1 Point Release
+| [[ WW07 - 2017-02-13 - Full Point Release Test Cycle 2.2.1 rc1]]
+|-
+| 2.2.2 rc1 Point Release
+| [[ WW22_-_2017-06-05_-_Full_Point_Release_Test_Cycle_2.2.2_rc1]]
+|-
+| 2.2.2 rc2 Point Release
+| [[ WW37_-_2017-09-13_-_Full_Point_Release_Test_Cycle_2.2.2_rc2]]
+|-
+| 2.2.3 rc2 Point Release
+| [[ WW06_-_2018-02-08_-_Full_Point_Release_Test_Cycle_2.2.3_rc2 ]]
+|-
+| 2.2.4 rc1 Point Release
+| [[ WW28_-_2018-07-13_-_Full_Test_Cycle_2.2.4_rc1 ]]
 |}

Yocto Project 2.2 Release Test Plan: Difference between revisions