Yocto Project 2.7 Release Test Plan

From Yocto Project
Jump to navigationJump to search
{{#vardefine:ADT_MAUAL | 6539}} {{#vardefine:ADT_AUTO | 6540}} {{#vardefine:ANYQEMU_SATO_MANUAL | 6563}} {{#vardefine:ANYQEMU_SATO_AUTO | 6564}} {{#vardefine:ANYBSP_SATO_AUTO | 6558}} {{#vardefine:ANYBSP_LSB_AUTO | 6561}} {{#vardefine:ANYBSP_WIC_AUTO | 6560}} {{#vardefine:MMAX_CORE2_MANUAL | 6554}} {{#vardefine:MMAX_I7_MANUAL | 6557}} {{#vardefine:NUC_I7_MANUAL | 6556}} {{#vardefine:NUC_GEX86-64_MANUAL | 6555}} {{#vardefine:MMAX_GEX86-32_MANUAL | 6553}} {{#vardefine:GAL_SATO_AUTO | 6565}} {{#vardefine:CROPS_AUTO | 6541}} {{#vardefine:CROPS_MANUAL | 6542}} {{#vardefine:ECLIPSE_MANUAL | 6543}} {{#vardefine:KERNEL_MANUAL | 6714}} {{#vardefine:METAYOCTO_MANUAL | 6544}} {{#vardefine:TOASTER_AUTO | 6545}} {{#vardefine:TOASTER_MANUAL | 6546}} {{#vardefine:ESDK_MANUAL | 6547}} {{#vardefine:ESDK_AUTO | 6548}} {{#vardefine:OE-CORE_AUTO | 6549}} {{#vardefine:BITBAKE_AUTO | 6550}} {{#vardefine:RUNTIME_AUTO | 6551}} {{#vardefine:BUILD-APPLIANCE_AUTO | 6552}}

Reversion history

Version Modifier Comments
1.0 Yeoh Ee Peng First draft

References

Documents that support this test plan

Introduction

This is the the overall test plan for version 2.7 of the Yocto Project, for all the generic information this document will be pointing to QA Master Test Plan

Objectives

This test plan for Yocto Project 2.7 release supports the following objectives:

  • To define the tools to be used throughout the testing process.
  • To communicate to the responsible parties the items to be tested, set expectations around schedule, and define environmental needs.
  • To define how the tests will be conducted.
  • Release Criteria
  • Risks and contingencies

QA_Master_Test_Plan#Objectives_and_Tasks

Risk Analysis & Contingency Plan

This is the document for Risk Analysis https://docs.google.com/spreadsheets/d/1K-MZYo5sfabwI2UozGp6Uuh2sw5y309bXkyEZRPEEgk/edit#gid=0

Test Strategy

Is followed the general strategy defined on QA_Master_Test_Plan#Testing_Strategy and below is showed specific information applicable only for 2.7 Release

Bugs Tracking

Test Items

The Yocto Project QA team verifies each release against recent versions of the most popular Linux distributions that provide stable releases.

* centos 7 
* Debian 9
* Fedora 26
* Fedora 27
* OpenSuse42.3
* ubuntu 16.04 
* ubuntu 17.10

Test Cases


Test Matrix

The components to be tested for version 2.7 of the Yocto Project against the different milestones are included in this section:

Code of Colors
Active

Tracked on Testpoia

Active

Non Tracked on Testopia

In Design

Will be tracked on Tesopia

Inactive


ADT

Component Environment Auto Manual Owner
ADT Fedora 25 i686 Y Y PG
Ubuntu 16.04 x86_64 Y Y PG

Eclipse Plugin

Component Environment Auto Manual Owner
Neon + Any Distro Y PG
Oxygen + Any Distro Y PG

Kernel

Component Environment Auto Manual Owner
Kernel Any Distro Y PG

Meta-Yocto

Component Environment Auto Manual Owner
Meta-Yocto Any Distro Y PG

Toaster

Component Environment Auto Manual Owner
Toaster Any Distro Y PG

eSDK

Component Environment Auto Manual Owner
eSDK Any Distro + runtime Y PG
Any Distro + selftest Y PG

Selftest

Component Environment Auto Manual Owner
OE-CORE Ubuntu 16.04 Y PG
Ubuntu 17.10 Y PG
OpenSuse 42.3 Y PG
Debian 9 Y PG
Centos 7 Y PG
Fedora 26 Y PG
Fedora 27 Y PG

Bitbake

Component Environment Auto Manual Owner
Bitbake Any Distro Y PG

Build - Apliance

Component Environment Auto Manual Owner
Build - Apliance VM - Any Distro Y PG

Package Manager

Component Environment Auto Manual Owner
Package Manager Any Distro Y PG

Performance

Component Environment Auto Manual Owner
Performance Centos 7 Y PG
Ubuntu 16.04 Y PG

BSP/QEMU

Component Arch HW Environment Image Auto Manual pTest Compliance
BSP x86 MinwTurbot 32bit genericx86 core-image-sato-sdk Y
core-image-lsb-sdk Y
MinwTurbot 64bit genericx86-64 core-image-sato-sdk Y Y
core-image-lsb-sdk Y
NUC 6 genericx86-64 core-image-sato-sdk Y Y
core-image-lsb-sdk Y Y
NUC 7 genericx86-64 core-image-sato-sdk Y Y
core-image-lsb-sdk Y
QEMU qemux86 core-image-sato-sdk Y Y
core-image-lsb-sdk Y
qemux86-64 core-image-sato-sdk Y Y
core-image-lsb-sdk Y
non x86 EdgeRouter EdgeRouter core-image-sato-sdk Y Y
MPC8315e-rdb MPC8315e-rdb core-image-sato-sdk Y Y
Beaglebone Black Beaglebone core-image-sato-sdk Y Y
QEMU qemuarm core-image-sato-sdk Y Y
qemuarm-64 core-image-sato-sdk Y Y
qemuppc core-image-sato-sdk Y Y
qemumips core-image-sato-sdk Y Y
qemumips64 core-image-sato-sdk Y Y
Meta-Intel MinwTurbot 32bit core2-32 core-image-sato-sdk Y
core-image-lsb-sdk Y
MinwTurbot 64bit corei7-64 core-image-sato-sdk Y Y
core-image-lsb-sdk Y
NUC 6 corei7-64 core-image-sato-sdk Y
core-image-lsb-sdk Y
NUC 7 corei7-64 core-image-sato-sdk Y Y
core-image-lsb-sdk Y
poky-tiny Y

All component to be tested on 2.7 are in #Test Items a detailed list of features per component can be found in Test Plan of the components QA_Master_Test_Plan#Component

Testopia Templates

COMPONENT TEMPLATE
BitBake TRTEMP_2.6_MANUAL_BitBake
BSP/QEMU TRTEMP_2.6_MANUAL_BSP/QEMU_core-image-sato-sdk_Genericx86_MMAX32
TRTEMP_2.6_MANUAL_BSP/QEMU_core-image-sato-sdk_core2-32_MMAX32
TRTEMP_2.6_MANUAL_BSP/QEMU_core-image-sato-sdk_Genericx86-64_NUC
TRTEMP_2.6_MANUAL_BSP/QEMU_core-image-sato-sdk_corei7-64_NUC
TRTEMP_2.6_MANUAL_BSP/QEMU_core-image-sato-sdk_corei7-64_MMAX64
TRTEMP_2.6_AUTO_BSP/QEMU_core-image-sato-sdk_ANYBSP
TRTEMP_2.6_AUTO_BSP/QEMU_core-image-sato-sdk_wic_ANYBSP
TRTEMP_2.6_AUTO_BSP/QEMU_core-image-lsb-sdk_ANYBSP
TRTEMP_2.6_MANUAL_BSP/QEMU_core-image-sato-sdk_ANYQEMU
TRTEMP_2.6_AUTO_BSP/QEMU_core-image-sato-sdk_ANYQEMU
TRTEMP_2.6_AUTO_BSP/QEMU_core-image-sato-sdk_intel-quark
TRTEMP_2.6_MANUAL_BSP/QEMU_core-image-sato-sdk_Beaglebone-Black
TRTEMP_2.6_MANUAL_BSP/QEMU_core-image-sato-sdk_EdgeRouter
TRTEMP_2.6_MANUAL_BSP/QEMU_core-image-sato-sdk_MPC8315e-rdb
TRTEMP_2.6_AUTO_BSP/QEMU_core-image-lsb-sdk_ANYQEMU
Meta-Yocto TRTEMP_2.6_MANUAL_Meta-yocto
OE-CORE TRTEMP_2.6_AUTO_OE-Core
Build-Appliance TRTEMP_2.6_MANUAL_Build-Appliance
Runtime TRTEMP_2.6_AUTO_Runtime
Toaster TRTEMP_2.6_AUTO_Toaster_UI
TRTEMP_2.6_MANUAL_Toaster_UI-managed-mode
ADT TRTEMP_2.6_MANUAL_ADT
TRTEMP_2.6_AUTO_ADT
CROPS TRTEMP_2.6_AUTO_CROPS_Poky
TRTEMP_2.6_MANUAL_CROPS_eSDK
Eclipse TRTEMP_2.6_MANUAL_Eclipse
eSDK TRTEMP_2.6_AUTO_eSDK_sdkext
TRTEMP_2.6_AUTO_eSDK_oeselftest
Kernel TRTEMP_2.6_MANUAL_Kernel
Package Management TRTEMP_2.6_MANUAL_Package-Management-Updates

Features pending to be tested

Here is a list of areas that are not yet covered and need some work, implement test cases and add it to a component that belongs or create a new test plan for new component.

ADD FEATURES !!!!!!!!!!


The complete List of QA Implementations for YP 2.7 can be found at Features Implementation by QA Team

Features Not to be tested

Approach

Tests will be conducted per the documented test cases stored in Testopia. The test manager will create test runs for each tester. The tester will execute the tests in Testopia and mark each case as Pass / Fail / Skip /Blocked. The tester should leave notes on actual results and any other relevant details when possible.

Once complete, the test manager should review the test run reports in Testopia and report back to the team by mail and in the wiki as a test report.

Pass / Fail Criteria

This criteria is defined in Yocto_Project_v2.7_Status#Yocto_Project_v2.7_Release_Criteria

Exit Criteria

As defined in QA_Master_Test_Plan#Entry and Exit Criteria

Test Deliverables

Test Environment

There are different environments and tools for every type of testing refer to QA_Master_Test_Plan#Test_Areas for more detailed information.

Schedule

M1 (Apr 30, 2018 to Jun 22, 2018)

M1 Feature/Tasks Development / Cutoff is Jun. 11, 2017

IDRequesterAssigneeESummary (12 tasks) SeverityPWhiteboardStatus
13058Chen QiAndré Draszik2split util-linux automatically instead of hard codingenhancementMedium+RESOLVED
13094David ReynaDavid Reyna3support custom extensions to reportsenhancementMediumRESOLVED
13110David ReynaDavid Reyna3Add "release" records to product recordsenhancementMediumRESOLVED
11583Aníbal Limónyeoh ee peng4oeqa: core add support to publish resultsenhancementMedium+RESOLVED
12774Lei YangHongzhioeqa: kernel config depenency for some kernel test casesenhancementMedium+RESOLVED
12931Joshua WattJoshua Watt5Change default PACKAGE_DEBUG_SPLIT_STYLE to "debug-with-srcpkg"enhancementMedium+RESOLVED
13030Joshua WattJoshua WattAdd sstate hash equivalence supportenhancementMedium+RESOLVED
13088Richard PurdieAníbal Limón0.3Add support for extracting individual ptest timingenhancementMedium+RESOLVED
12332Paul EggletonMichael Halstead2Regularly purge the error report databaseenhancementMedium+RESOLVED
12421Juro BystrickyMingli YuHost-compiled Python bytecode not reproducibleenhancementMedium+RESOLVED
13011Armin KusterRichard PurdieStart collecting ptest for arm and x86 on Vtargets.enhancementMedium+RESOLVED
13114 Ross BurtonSplit python3-ctypes tests packageenhancementMedium+RESOLVED
   19.3     

M1 Bugs

IDRequesterAssigneeESummary (62 tasks) SeverityPWhiteboardStatus
12985Christian AndersenAlejandro Hernandezmulticonfig in v2.6 no longer works with COMPATIBLE_MACHINE (e.g. in image recipes)normalMedium+Backport to 2.6.1RESOLVED
13028Denys DmytriyenkoAlexander KanavinENABLE_BINARY_LOCALE_GENERATION = "0" is now broken by postinst refactoringnormalMedium+RESOLVED
5697Chong LuChangqing Li1busybox: ptest cases failednormalMediumRESOLVED
8832Alexandru GeorgescuChangqing Li1pTest - bash tests fail for 2.1_M1.rc1normalMediumRESOLVED
9042Francisco PedrazaChangqing Li3pTest - strace fails on NUC 5425WYK for YP 2.1_M2.rc1normalMediumRESOLVED
12997Ross BurtonDavid Reyna1MITRE import not workingnormalMediumRESOLVED
12998Ross BurtonDavid Reyna2Triage page should also have Investigate buttonnormalMediumRESOLVED
12994Ross BurtonDavid Reyna2Make checkboxes easier to use in triagenormalMediumRESOLVED
12999Ross BurtonDavid Reyna4Triage prediction isn't workingnormalMediumRESOLVED
13093David ReynaDavid Reyna1Fix pylint reported issuesnormalMediumRESOLVED
13099Ross BurtonDavid Reyna1String/list confusion in user groupsnormalMediumRESOLVED
13121Karin JuleniusDavid Reyna2CVE Resource information not available in SRToolnormalMediumRESOLVED
13125Ross BurtonDavid Reyna1CPEs for Yocto Project not correctnormalMediumRESOLVED
13126Ross BurtonDavid Reyna1Don't try Py2 importsnormalMediumRESOLVED
13131Ross BurtonDavid Reyna3update script failsnormalMediumRESOLVED
13134Ross BurtonDavid Reyna1Doesn't fetch 2019 CVE datanormalMediumRESOLVED
13135Ross BurtonDavid Reyna2New CVEs are HistoricalnormalMediumRESOLVED
13138Ross BurtonDavid Reyna1srtool_update throws exceptionnormalMediumRESOLVED
13139Ross BurtonDavid Reyna2Triage failsnormalMediumRESOLVED
13091David ReynaDavid Reyna1SRTool broken for Django-2.2normalMedium+RESOLVED
10713brian averyyeoh ee peng3Allow testimage to work with slirp in addition to working with tun/tapnormalMedium+RESOLVED
2655veeraHongxu JiaIn /usr/bin/ldd shared library path mentioned multiple times ( RTLDLIST="/lib/ld-linux.so.2 /lib/ld-linux.so.2 )normalLowRESOLVED
12480Juro BystrickyHongxu Jia3libc6-dbg: non deterministic build (multilib)normalMediumRESOLVED
12524Juro BystrickyHongxu Jia2(reproducibility) autoconf-info contains build datenormalMediumreproducibilityRESOLVED
12525Juro BystrickyHongxu Jia2(reproducibility) python3-iniparse contains build host referencesnormalMediumreproducibilityRESOLVED
12543Juro BystrickyHongxu Jia2(reproducibility) python-xcbgen contains host references and timestampsnormalMedium+reproducibilityRESOLVED
12709Juro BystrickyHongxu Jia2(reproducibility) strace-ptest contains build host referencenormalMedium+reproducibilityRESOLVED
13128Ross BurtonJoshua Wattnativesdk-icecc doesn't work for multilib SDKsnormalMediumRESOLVED
13020Ross BurtonJoshua Wattmeta-mingw needs testingnormalMedium+RESOLVED
13108 Joshua Wattlog show many fetch error after commit: bitbake: persist_data: Retry database setupnormalMedium+RESOLVED
13124Alejandro HernandezJoshua Wattmulticonfig builds break when preparing the task hash (runqueue)normalMedium+RESOLVED
11969Carlos Alberto Lopez PerezKai Kang2issues with network-manager handling of resolv.conf when systemd-resolved is not enablednormalMediumRESOLVED
8586Jussi KukkonenKai Kang1multilib build includes 32bit connman without good reasonnormalMedium+RESOLVED
12288Robert YangKai Kang2lib32 packages are wrongly installednormalMedium+Backport 2.3.3 and 2.4.1?RESOLVED
12298Robert YangKai Kang2bitbake lib32-meta-ide-support Manifest not foundnormalMedium+Backport to 2.4.1 and 2.3.3 if needed.RESOLVED
13074Kai KangKai Kang2It fails to build systemd on qemux86-64 with default tune x86-64-x32normalMedium+RESOLVED
13081Kai KangKai Kang1lttng-tools fails to build for x32normalMedium+RESOLVED
12987Frederik Noe-SdunChristopher LarsonRemove variables containing shell substitutions from vardeps of shell function if it cannot be expandednormalMedium+RESOLVED
12380Jackie HuangRobert Yang2Variable definition: leading spaces stripping behavior is not consistentnormalMediumRESOLVED
12112Christian AndersenRobert Yang3Public key for U-Boot verified boot is not inserted in DTB when rebuilding from sstatemajorMedium+Backport 2.4.4, 2.5.1RESOLVED
12898Paul EggletonRobert Yang2Bitbake does not print server errors when there is no TTYnormalMedium+RESOLVED
12992Mohamad Noor AlimMark Hatle[2.6 M4 rc1] test_devtool_add_fetch_git failed on Fedora 27normalMedium+RESOLVED
11020brian averyMichael Halstead1We need a patchwork instance pointing to the Yocto mailing listnormalMedium+RESOLVED
5057IlovetogetspamedMingli Yu4Conflict when building SDK for image containing dropbear and openssh-sftp-servernormalMediumRESOLVED
12639Juro BystrickyMingli Yu(reproducibility) nspr packages contain timestampsnormalMedium+reproducibilityRESOLVED
13043Ross BurtonRichard Purdierunqemu.RunqemuTests.* failingnormalHighRESOLVED
13044Ross BurtonRichard PurdieBitbake failing to start in wic testnormalHighRESOLVED
13045Ross BurtonRichard Purdiegpg agent socket not foundnormalHighRESOLVED
13046Ross BurtonRichard Purdieccache failure in oe-selftestnormalHighRESOLVED
13068Enrico ScholzRichard Purdiefetch: module 'os' has no attribute 'errno'normalHighRESOLVED
12586Denys DmytriyenkoRichard Purdieyocto-check-layer fails signatures due to TUNES between ARM archsnormalMedium+RESOLVED
13003Michael HalsteadRichard PurdieBuild hangs if expected bitbake.lock lock file is missingnormalMedium+RESOLVED
13007Armin KusterRichard Purdieoe-selftest on each supported distronormalMedium+RESOLVED
13018Armin KusterRichard PurdieSeparate logic on autobuilder for a lighter test run and a longer more comprehensive onenormalMedium+RESOLVED
13127JinRichard Purdiechanges in bbappend do not trigger a rebuildnormalMedium+RESOLVED
13042Ross BurtonRoss Burtonimagefeatures.ImageFeatures.test_image_fstypes failed in cpionormalHighRESOLVED
12966 Ross BurtonMesa depends on xorgproto even when x11 isn't in DISTRO_FEATURESnormalMedium+Backport to 2.6.1RESOLVED
12441LibertadSudhir Sharmaptest comparison script is not showing correct resultscriticalMedium+RESOLVED
13115Richard WeinbergerScott Rifenbark1Clarify IMAGE_FSTYPES notenormalMedium03 January 2019: RESOLVEDRESOLVED
12399brian averyScott Rifenbark5add non conflicting parts of the bitbake manual to the mega manualnormalMedium+14 February 2019: RESOLVEDRESOLVED
13112Richard PurdieScott Rifenbark2FILESPATH and FILESOVERRIDES documentation lacks key detailnormalMedium+18 February 2019: RESOLVEDRESOLVED
13101Anuj MittalTim Orlingeclipse oxygen-master build failing consistentlynormalMedium+RESOLVED
   74     


The planed schedule for 2.7 can be found at Yocto_2.7_Schedule

Live Schedule & Execution History

MILESTONE STARTED FINISHED STATUS REPORT
M1 rc1 12/19/2018 1/2/2019 DONE https://wiki.yoctoproject.org/wiki/WW01_-_2019-01-02_-_Full_Test_Cycle_2.7_M1_RC1
M2 rc1 01/28/2019 02/08/2019 DONE https://wiki.yoctoproject.org/wiki/WW01_-_2019-01-28_-_Full_Test_Cycle_2.7_M2_RC1
M2 rc2 02/14/2019 02/25/2019 DONE https://wiki.yoctoproject.org/wiki/WW07_-_2019-02-14_-_Full_Test_Cycle_2.7_M2_RC2