Yocto Project 2.3 Release Test Plan

From Yocto Project
Jump to navigationJump to search
{{#vardefine:ADT_MAUAL | 6539}} {{#vardefine:ADT_AUTO | 6540}} {{#vardefine:ANYQEMU_SATO_MANUAL | 6563}} {{#vardefine:ANYQEMU_SATO_AUTO | 6564}} {{#vardefine:ANYBSP_SATO_AUTO | 6558}} {{#vardefine:ANYBSP_LSB_AUTO | 6561}} {{#vardefine:ANYBSP_WIC_AUTO | 6560}} {{#vardefine:MMAX_CORE2_MANUAL | 6554}} {{#vardefine:MMAX_I7_MANUAL | 6557}} {{#vardefine:NUC_I7_MANUAL | 6556}} {{#vardefine:NUC_GEX86-64_MANUAL | 6555}} {{#vardefine:MMAX_GEX86-32_MANUAL | 6553}} {{#vardefine:GAL_SATO_AUTO | 6565}} {{#vardefine:CROPS_AUTO | 6541}} {{#vardefine:CROPS_MANUAL | 6542}} {{#vardefine:ECLIPSE_MANUAL | 6543}} {{#vardefine:KERNEL_MANUAL | 6714}} {{#vardefine:METAYOCTO_MANUAL | 6544}} {{#vardefine:TOASTER_AUTO | 6545}} {{#vardefine:TOASTER_MANUAL | 6546}} {{#vardefine:ESDK_MANUAL | 6547}} {{#vardefine:ESDK_AUTO | 6548}} {{#vardefine:OE-CORE_AUTO | 6549}} {{#vardefine:BITBAKE_AUTO | 6550}} {{#vardefine:RUNTIME_AUTO | 6551}} {{#vardefine:BUILD-APPLIANCE_AUTO | 6552}}

Reversion history

Version Modifier Comments
1.0 Jose Perez Carranza First draft

References

Documents that support this test plan

Introduction

This is the the overall test plan for version 2.3 of the Yocto Project, for all the generic information this document will be pointing to QA Master Test Plan

Objectives

This test plan for Yocto Project 2.3 release supports the following objectives:

  • To define the the tools to be used throughout the testing process.
  • To communicate to the responsible parties the items to be tested, set expectations around schedule, and define environmental needs.
  • To define how the tests will be conducted.
  • Release Criteria
  • Risks and contingencies

QA_Master_Test_Plan#Objectives_and_Tasks

Test Strategy

Is followed the general strategy defined on QA_Master_Test_Plan#Testing_Strategy and below is showed specific information applicable only for 2.3 Release

Bugs Tracking

Test Items

The Yocto Project QA team verifies each release against recent versions of the most popular Linux distributions that provide stable releases.

* centos 7 
* Debian 8
* Fedora 23 
* Fedora 24
* OpenSuse132 
* OpenSuse421 
* ubuntu 14.04 
* ubuntu 16.04

Test Cases


Components to be tested

The components to be tested for version 2.3 of the Yocto Project against the different milestones are included in this section:

Code of Colors
Active

Tracked on Testpoia

Active

Non Tracked on Testopia

In Design

Will be tracked on Tesopia

Inactive


COMPONENT TEMPLATE
ADT TRTEMP_2.3_MANUAL_ADT
TRTEMP_2.3_AUTO_ADT
BSP TRTEMP_2.3_AUTO_BSP/QEMU_core-image-sato-sdk_ANYQEMU
TRTEMP_2.3_AUTO_BSP/QEMU_core-image-sato-sdk_ANYBSP
TRTEMP_2.3_AUTO_BSP/QEMU_core-image-lsb-sdk_ANYBSP
TRTEMP_2.3_AUTO_BSP/QEMU_core-image-sato-sdk-wic_ANYBSP
TRTEMP_2.3_AUTO_BSP/QEMU_core-image-sato-sdk_intel-quark
TRTEMP_2.3_MANUAL_BSP/QEMU_core-image-sato-sdk_ANYQEMU
TRTEMP_2.3_MANUAL_BSP/QEMU_core2-32_MinnowMax
TRTEMP_2.3_MANUAL_BSP/QEMU_corei7-64_MinnowMax
TRTEMP_2.3_MANUAL_BSP/QEMU_corei7-64_NUC
}} TRTEMP_2.3_MANUAL_BSP/QEMU_GenericX86-64_NUC
TRTEMP_2.3_MANUAL_BSP/QEMU_GenericX86_MMAX32
CROPS TRTEMP_2.3_AUTO_CROPS
TRTEMP_2.3_MANUAL_CROPS
Eclipse-Plugin TRTEMP_2.3._MANUAL_Eclipse-Plugin
Kernel TRTEMP_2.3_MANUAL_Kernel
Meta-Yocto TRTEMP_2.3_MANUAL_Meta-yocto
Toaster TRTEMP_2.3_AUTO_TOASTER_UI
TRTEMP_2.3_MANUAL_TOASTER_UI-managed-mode
eSDK TRTEMP_2.3_AUTO_eSDK
TRTEMP_2.3_MANUAL_eSDK
OE-CORE / BitBake TRTEMP_2.3_AUTO_OE-Core
TRTEMP_2.3_MANUAL_BitBake
Runtime (Compliance) TRTEMP_2.3_AUTO_Runtime
Build Apliance TRTEMP_2.3_MANUAL_Build-Appliance
pTest
Performance Performance Test
WUpd (Software Updater)
Distributed Builds
MRAA
Meta-SwUpd

All component to be tested on 2.3 are in #Test Items a detailed list of features per component can be found in Test Plan of the components QA_Master_Test_Plan#Component

Features pending to be tested

Here is a list of areas that are not yet covered and need some work, implement test cases and add it to a component that belongs or create a new test plan for new component.

Features
Bug ID Description
Bug 10723 Build-Appliance - Run builds in the build-appliance image
Bug 10724 Verify if -c menuconfig work
Bug 10724 Verify if -c devshell work
Bug 10725 Verify if BBPOSTCONF and BBPRECONF work in bitbakek
Bug 10726 Verify if oe-pkgdata-util print help when no parameters specified
Bug 10727 Verify if buildhistory_analsysis.py is tested
Bug 10728 Verify combo-layer
Bug 6926 Verify yocto-bsp/yocto-layer
Bug 10724 Verify bitbake -u depexp -g XXX

The complete List of QA Implementations for YP 2.3 can be found at Features Implementation by QA Team

Features Not to be tested

Approach

Tests will be conducted per the documented test cases stored in Testopia. The test manager will create test runs for each tester. The tester will execute the tests in Testopia and mark each case as Pass / Fail / Skip /Blocked. The tester should leave notes on actual results and any other relevant details when possible.

Once complete, the test manager should review the test run reports in Testopia and report back to the team by mail and in the wiki as a test report.

Pass / Fail Criteria

This criteria is defined in Yocto_Project_v2.3_Status#Yocto_Project_v2.3_Release_Criteria

Exit Criteria

As defined in QA_Master_Test_Plan#Entry and Exit Criteria

Test Deliverables

Test Environment

There are different environments and tools for every type of testing refer to QA_Master_Test_Plan#Test_Areas for more detailed information.

Schedule

M1 (Oct. 31, 2016 to Dec. 23, 2016 -- 9 weeks: planning done in week 1, development week 2-6, stabilization week 7-8, release week 9)

M1 Feature/Tasks Development / Cutoff is Dec. 12, 2016

IDRequesterAssigneeESummary (6 tasks) SeverityPWhiteboardStatus
9088Aníbal LimónFrancisco Pedraza3Extensible SDK add tests for devtoolenhancementMedium+RESOLVED
10753Francisco PedrazaFrancisco Pedraza5QA Kernel-dev test cases basic functionalityenhancementMedium+VERIFIED
10394Jair GonzalezJair Gonzalez2Creation of component to emulate keyboard with USB-KM232enhancementHighRESOLVED
7376Ross BurtonJose Perez C3oe-selftest doesn't run lib/oe/tests/*enhancementMedium+VERIFIED
8433Ross BurtonJose Perez C4Add test for package postinstsenhancementMedium+RESOLVED
8584Ross BurtonJuan Ramos3Enable building API documentationenhancementMedium+21 Nov 2016: RESOLVEDVERIFIED
   20     

M1 Bugs

IDRequesterAssigneeESummary (4 tasks) SeverityPWhiteboardStatus
5319Mark HatleFrancisco Pedraza3Need a test to verify firstboot postinst script ordernormalMedium+RESOLVED
10432Leonardo Sandoval GonzalezFrancisco Pedraza2eSDK test test_install_libraries_headers fail when distro is nodistronormalMedium+VERIFIED
10481Jair GonzalezJair Gonzalez1[Test Case 1059] Parselogs is failing on genericx86 (32 and 64 bits) WIC giving error: GPT: Use GNU Parted to correct GPT errors.normalMediumRESOLVED
10594Jose Perez CJair Gonzalez2Ensure all options of WIC script are testednormalMedium+VERIFIED
   8     

M1 Stabilize schedule

Release Candidate and Milestone Release schedule

  • RC1: 12/12/16
  • M1 release: 12/23/16

The planed schedule for 2.3 can be found at Yocto_2.3_Schedule

Live Schedule

MILESTONE STARTED FINISHED STATUS
M1 rc1 12/19/2016 12/23/2016 DONE
M2 rc1 01/24/2017 01/31/2017 DONE
M2 rc3 02/03/2017 02/08/2017 DONE
M3 rc1 03/15/2017 03/23/2017 DONE
M3 rc2 03/28/2017 04/05/2017 DONE
M3 rc1 Meta-Intel 04/05/2017 04/12/2017 DONE
M4 rc2 04/21/2017 04/29/2017 DONE
M4 rc6 05/02/2017 05/08/2017 DONE
2.3 Meta-Intel rc1 05/11/2017 05/16/2017 DONE
2.3 Meta-Intel rc4 05/22/2017 05/25/2017 DONE
2.3.1 rc2 07/12/2017 07/18/2017 DONE
2.3.2 rc1 09/12/2017 09/20/2017 DONE
2.3.3 rc1 12/15/2017 12/20/2017 DONE
2.3.4 rc1 6/13/2018 7/10/2018 DONE

Execution History

MILESTONE REPORT
M1 rc1 WW52 - 2016-12-19 - Full Test Cycle 2.3 M1 rc1
M2 rc1 WW05 - 2017-01-24 - Full Test Cycle 2.3_M2 rc1
M2 rc3 WW06 - 2017-02-03 - Full Test Cycle 2.3_M2 rc3
M3 rc1 WW11 - 2017-03-15 - Full Test Cycle 2.3 M3 rc1
M3 rc2 WW14 - 2017-03-28 - Full Test Cycle 2.3 M3 rc2
M3 rc1 Meta-Intel WW15 - 2017-04-05 - Full Test Cycle - 2.3 M3 - meta-intel
M4 rc2 (2.3 release - pyro) WW17 - 2017-04-21 - Full Test Cycle 2.3 rc2
M4 rc4 (2.3 release - pyro) WW19 - 2017-05-02 - Full Test Cycle 2.3 rc4
Meta-Intel rc1 (2.3 release - pyro) WW20 - 2017-05-11 - Full Test Cycle - 2.3 rc1 - meta-intel
Meta-Intel rc4 (2.3 release - pyro) WW21 - 2017-05-22 - Full Test Cycle - 2.3 rc4 - meta-intel
2.3.1 rc2 WW29 - 2017-07-18- Full Test Cycle - 2.3.1 rc2
2.3.2 rc1 WW38_-_2017-09-20-_Full_Test_Cycle_-_2.3.2_rc1
2.3.3 rc1 WW51_-_2017-12-20-_Full_Test_Cycle_-_2.3.3_rc1
2.3.4 rc1 WW25_-_2018-06-22-_Full_Test_Cycle_-_2.3.4_rc1