Yocto Project 2.8 Release Test Plan: Difference between revisions

From Yocto Project
Jump to navigationJump to search
Line 608: Line 608:
   |columns=id,from,to,estimated,summary,severity,priority,whiteboard,status
   |columns=id,from,to,estimated,summary,severity,priority,whiteboard,status
   |total=estimated
   |total=estimated
   |milestone=2.7 M1, 2.7 M2
   |milestone=2.8 M1, 2.8 M2
   |severity=!enhancement
   |severity=!enhancement
   |sort=to,priority
   |sort=to,priority
Line 614: Line 614:




The planed schedule for 2.7 can be found at [[Yocto_2.7_Schedule]]
The planed schedule for 2.8 can be found at [[Yocto_2.8_Schedule]]


== Live Schedule & Execution History ==
== Live Schedule & Execution History ==

Revision as of 08:16, 18 June 2019

{{#vardefine:ADT_MAUAL | 6539}} {{#vardefine:ADT_AUTO | 6540}} {{#vardefine:ANYQEMU_SATO_MANUAL | 6563}} {{#vardefine:ANYQEMU_SATO_AUTO | 6564}} {{#vardefine:ANYBSP_SATO_AUTO | 6558}} {{#vardefine:ANYBSP_LSB_AUTO | 6561}} {{#vardefine:ANYBSP_WIC_AUTO | 6560}} {{#vardefine:MMAX_CORE2_MANUAL | 6554}} {{#vardefine:MMAX_I7_MANUAL | 6557}} {{#vardefine:NUC_I7_MANUAL | 6556}} {{#vardefine:NUC_GEX86-64_MANUAL | 6555}} {{#vardefine:MMAX_GEX86-32_MANUAL | 6553}} {{#vardefine:GAL_SATO_AUTO | 6565}} {{#vardefine:CROPS_AUTO | 6541}} {{#vardefine:CROPS_MANUAL | 6542}} {{#vardefine:ECLIPSE_MANUAL | 6543}} {{#vardefine:KERNEL_MANUAL | 6714}} {{#vardefine:METAYOCTO_MANUAL | 6544}} {{#vardefine:TOASTER_AUTO | 6545}} {{#vardefine:TOASTER_MANUAL | 6546}} {{#vardefine:ESDK_MANUAL | 6547}} {{#vardefine:ESDK_AUTO | 6548}} {{#vardefine:OE-CORE_AUTO | 6549}} {{#vardefine:BITBAKE_AUTO | 6550}} {{#vardefine:RUNTIME_AUTO | 6551}} {{#vardefine:BUILD-APPLIANCE_AUTO | 6552}}

Reversion history

Version Modifier Comments
1.0 Jain Sangeeta First draft

References

Documents that support this test plan

Introduction

This is the the overall test plan for version 2.8 of the Yocto Project, for all the generic information this document will be pointing to QA Master Test Plan

Objectives

This test plan for Yocto Project 2.8 release supports the following objectives:

  • To define the tools to be used throughout the testing process.
  • To communicate to the responsible parties the items to be tested, set expectations around schedule, and define environmental needs.
  • To define how the tests will be conducted.
  • Release Criteria
  • Risks and contingencies

QA_Master_Test_Plan#Objectives_and_Tasks

Risk Analysis & Contingency Plan

This is the document for Risk Analysis https://docs.google.com/spreadsheets/d/1K-MZYo5sfabwI2UozGp6Uuh2sw5y309bXkyEZRPEEgk/edit#gid=0

Test Strategy

Is followed the general strategy defined on QA_Master_Test_Plan#Testing_Strategy and below is showed specific information applicable only for 2.8 Release

Bugs Tracking

Test Items

The Yocto Project QA team verifies each release against recent versions of the most popular Linux distributions that provide stable releases.

* centos 7 
* Debian 9
* Fedora 26
* Fedora 27
* OpenSuse42.3
* ubuntu 16.04 
* ubuntu 17.10

Test Cases

ADT

Component Environment Auto Manual Owner
ADT Fedora 25 i686 Y AB
Ubuntu 16.04 x86_64 Y AB

Eclipse Plugin

Component Environment Auto Manual Owner
Neon + Any Distro Y PG
Oxygen + Any Distro Y PG

Kernel

Component Environment Auto Manual Owner
Kernel Any Distro Y PG

Meta-Yocto

Component Environment Auto Manual Owner
Meta-Yocto Any Distro Y PG

Toaster

Component Environment Auto Manual Owner
Toaster Any Distro Y PG

eSDK

Component Environment Auto Manual Owner
eSDK Any Distro + runtime Y PG
Any Distro + selftest Y PG

Selftest

Component Environment Auto Manual Owner
OE-CORE Ubuntu 16.04 Y PG
Ubuntu 17.10 Y PG
OpenSuse 42.3 Y PG
Debian 9 Y PG
Centos 7 Y PG
Fedora 26 Y PG
Fedora 27 Y PG

Bitbake

Component Environment Auto Manual Owner
Bitbake Any Distro Y PG

Build - Apliance

Component Environment Auto Manual Owner
Build - Apliance VM - Any Distro Y PG

Package Manager

Component Environment Auto Manual Owner
Package Manager Any Distro Y PG

Performance

Component Environment Auto Manual Owner
Performance Centos 7 Y PG
Ubuntu 16.04 Y PG

BSP/QEMU

Component Arch HW Environment Image Auto Manual pTest Compliance
BSP x86 MinwTurbot 32bit genericx86 core-image-sato-sdk Y
core-image-lsb-sdk Y
MinwTurbot 64bit genericx86-64 core-image-sato-sdk Y Y
core-image-lsb-sdk Y
NUC 6 genericx86-64 core-image-sato-sdk Y Y
core-image-lsb-sdk Y Y
NUC 7 genericx86-64 core-image-sato-sdk Y Y
core-image-lsb-sdk Y
QEMU qemux86 core-image-sato-sdk Y Y
core-image-lsb-sdk Y
qemux86-64 core-image-sato-sdk Y Y
core-image-lsb-sdk Y
non x86 EdgeRouter EdgeRouter core-image-sato-sdk Y Y
MPC8315e-rdb MPC8315e-rdb core-image-sato-sdk Y Y
Beaglebone Black Beaglebone core-image-sato-sdk Y Y
QEMU qemuarm core-image-sato-sdk Y Y
qemuarm-64 core-image-sato-sdk Y Y
qemuppc core-image-sato-sdk Y Y
qemumips core-image-sato-sdk Y Y
qemumips64 core-image-sato-sdk Y Y
Meta-Intel MinwTurbot 32bit core2-32 core-image-sato-sdk Y
core-image-lsb-sdk Y
MinwTurbot 64bit corei7-64 core-image-sato-sdk Y Y
core-image-lsb-sdk Y
NUC 6 corei7-64 core-image-sato-sdk Y
core-image-lsb-sdk Y
NUC 7 corei7-64 core-image-sato-sdk Y Y
core-image-lsb-sdk Y
poky-tiny Y

All component to be tested on 2.7 are in #Test Items a detailed list of features per component can be found in Test Plan of the components QA_Master_Test_Plan#Component

Features pending to be tested

Here is a list of areas that are not yet covered and need some work, implement test cases and add it to a component that belongs or create a new test plan for new component.

ADD FEATURES !!!!!!!!!!


The complete List of QA Implementations for YP 2.7 can be found at Features Implementation by QA Team

Features Not to be tested

Approach

Tests will be conducted per the documented test cases stored in Testopia. The test manager will create test runs for each tester. The tester will execute the tests in Testopia and mark each case as Pass / Fail / Skip /Blocked. The tester should leave notes on actual results and any other relevant details when possible.

Once complete, the test manager should review the test run reports in Testopia and report back to the team by mail and in the wiki as a test report.

Pass / Fail Criteria

This criteria is defined in Yocto_Project_v2.7_Status#Yocto_Project_v2.7_Release_Criteria

Exit Criteria

As defined in QA_Master_Test_Plan#Entry and Exit Criteria

Test Deliverables

Test Environment

There are different environments and tools for every type of testing refer to QA_Master_Test_Plan#Test_Areas for more detailed information.

Schedule

M1 (Apr 30, 2018 to Jun 22, 2018)

M1 Feature/Tasks Development / Cutoff is Jun. 11, 2017

IDRequesterAssigneeESummary (12 tasks) SeverityPWhiteboardStatus
13058Chen QiAndré Draszik2split util-linux automatically instead of hard codingenhancementMedium+RESOLVED
13094David ReynaDavid Reyna3support custom extensions to reportsenhancementMediumRESOLVED
13110David ReynaDavid Reyna3Add "release" records to product recordsenhancementMediumRESOLVED
11583Aníbal Limónyeoh ee peng4oeqa: core add support to publish resultsenhancementMedium+RESOLVED
12774Lei YangHongzhioeqa: kernel config depenency for some kernel test casesenhancementMedium+RESOLVED
12931Joshua WattJoshua Watt5Change default PACKAGE_DEBUG_SPLIT_STYLE to "debug-with-srcpkg"enhancementMedium+RESOLVED
13030Joshua WattJoshua WattAdd sstate hash equivalence supportenhancementMedium+RESOLVED
13088Richard PurdieAníbal Limón0.3Add support for extracting individual ptest timingenhancementMedium+RESOLVED
12332Paul EggletonMichael Halstead2Regularly purge the error report databaseenhancementMedium+RESOLVED
12421Juro BystrickyMingli YuHost-compiled Python bytecode not reproducibleenhancementMedium+RESOLVED
13011Armin KusterRichard PurdieStart collecting ptest for arm and x86 on Vtargets.enhancementMedium+RESOLVED
13114 Ross BurtonSplit python3-ctypes tests packageenhancementMedium+RESOLVED
   19.3     

M1 Bugs

IDRequesterAssigneeESummary (49 tasks) SeverityPWhiteboardStatus
13295Joshua WattArmin Kusteropenssh ptest results are inconsistent per imagenormalMedium+RESOLVED
13391Richard PurdieAlexander Kanavinacl ptest timeout due to perl updatenormalHighRESOLVED
13363Sangeeta JainAnuj MittalLogrotateTest (lib/oeqa/runtime/cases/logrotate.py) failed.minorMedium+Backport to 2.7RESOLVED
13396Sudhir SharmaAnuj Mittal[QA 2.8 M1 RC1][BSP HW] parselogs.ParseLogsTest.test_parselogs failure on coffeelakenormalMedium+RESOLVED
13441Ross BurtonAnuj MittalEnable CONFIG_RANDOM_TRUST_CPUnormalMedium+RESOLVED
13248Richard PurdieApoorvNew QA process issuesnormalHighRESOLVED
12923GaryPaul Eggletonrecipetool create error on empty setup.pynormalMediumRESOLVED
13083Ross BurtonPaul Eggletonrecipetool create doesn't handle Python 3normalMedium+RESOLVED
13168Romuald JeannePaul Eggletondevtool: cannot add new recipe if reproducible_build class is inheritednormalMedium+Backport 2.5 and 2.6RESOLVED
13404Alexander KanavinPaul Eggleton'devtool upgrade' tracebacks when a recipe includes another recipe with PV in the namenormalMedium+RESOLVED
13422Ross BurtonPaul Eggletonrecipetool.RecipetoolTests.test_recipetool_load_plugin failsnormalMedium+RESOLVED
13284Armin KusterChangqing LiPatchtest seems to be off-line and not workingnormalHighRESOLVED
13416 Changqing Ligcc 9.1 runtime do_install errormajorMedium+RESOLVED
13356Emil MelarDavid Reyna1oe-core.xml in thud branch refers to wrong bitbake versionnormalMedium+also backport to Thud (2.6) and Warrior (2.7)RESOLVED
13382Jaewon LeeJaewon Leedevtool build should also deploy artifactnormalMediumRESOLVED
13377Jaewon LeeJaewon Leegst-inspect on target is looking for a .so on hostnormalMedium+RESOLVED
13379Armin KusterJon Masonqemuarm64 runtime syslog test failure in testimagenormalHighRESOLVED
12962Ross BurtonJoshua Watt[meta-mingw] update-alternative warningsnormalMediumRESOLVED
13301Joshua WattMariano Lopez2util-linux ptest results are inconsistent per imagenormalMedium+RESOLVED
13309Armin KusterKevin Hao[master] qemuppc fails to shutdown intermittentlynormalHighwith backportRESOLVED
13418Ross BurtonRobert YangMultilib and update-alternatives conflictnormalMedium+RESOLVED
13133Florian KauerMark HatleGit submodule fetcher fails if last component of path is different than the URLmajorMedium+RESOLVED
13340Matthias SchoepferMatthias Schoepferpython3 build fails when target is mips softfloatnormalMediumRESOLVED
13431Ross BurtonMichael HalsteadAdd new Debian Stable (Buster) autobuilder workernormalMedium+RESOLVED
13290Joshua WattMingli Yuelfutils ptest results are inconsistent per imagenormalMedium+RESOLVED
13294Joshua WattMingli Yunettle ptest results are inconsistent per imagenormalMedium+RESOLVED
13339Nam NguyenMingli YuRuby 2.4.4 does not run with Apalis iMX6normalMedium+RESOLVED
13243Paul EggletonNew Comer Bugsdevtool test failure in python scanning codenormalMedium+NEWCOMERRESOLVED
13258Kay LiuOtavio Salvadori.MX6 solox platform CPU hang in the case of using the Linux OSnormalMediumRESOLVED
11776Patrick OhlyChen Qi2read-only rootfs + systemd: systemd-tmpfiles failsnormalMedium+RESOLVED
13061Richard PurdieChen Qioe-selftest imagefeatures.ImageFeatures.test_image_fstypes (ubifs)normalMedium+RESOLVED
13390Richard PurdieChen Qioe-selftest qemu process handling issuenormalMedium+RESOLVED
12991pravinRichard Purdie[2.6 M4 RC1][Build-Appliance] Bitbake build-appliance-image getting failed during building image due to webkitgtk packagenormalMedium+RESOLVED
13208Jaewon LeeRichard PurdiePython2.7 check in oe-buildenv-internal is erroringmajorMedium+RESOLVED
13280Denys DmytriyenkoRichard PurdieRace condition in wic to package IMAGE_BOOT_FILES before EXTRA_IMAGEDEPENDS are builtnormalMedium+RESOLVED
13299Joshua WattRichard Purdiequilt ptest results are inconsistent per imagenormalMedium+NEWCOMERRESOLVED
12984Joe SlaterRichard PurdieSpecifying invalid PRSERV_HOST causes bitbake to hangnormalMedium+RESOLVED
13057Paul EggletonRoss BurtonNo default network configuration for systemd-networkdnormalMedium+RESOLVED
13366Richard PurdieRoss Burtonbluez5 ptest hangs with gcc 9normalMedium+RESOLVED
13072Tim OrlingRoss Burtonvulkan: upstream repo has changed; latest version is 1.1.92.1normalMedium+RESOLVED
13423Richard PurdieRoss Burtonefivar fails to build on opensuse423 with muslnormalMedium+RESOLVED
13291Joshua WattSakib Sajalflex ptest results are inconsistent per imagenormalMedium+RESOLVED
12857Kai RuhnauSeebsRunning anything dotnet/CoreCLR based in a devshell doesn't worknormalMedium+RESOLVED
12873Pau Espin PedrolTim Orling5cpan_build.bbclass is broken (no dependency Module::Build)normalMedium+Backport to 2.6.x, 2.7.xRESOLVED
13200Tim OrlingTim Orlingperl-rdepends.txt missing many dependencies for perl-module-test-morenormalMedium+Backport to 2.7RESOLVED
13381Tim OrlingTim Orling1SCons should migrate to Python 3normalUndecidedRESOLVED
13293Joshua WattUnassignedlibxml-parser-perl ptest results are inconsistent per imagenormalMedium+NEWCOMERRESOLVED
13300Joshua WattUnassignedtcl ptest results are inconsistent per imagenormalMedium+NEWCOMERRESOLVED
12886FerryHe ZheRemoval of VM_DEFAULT_KERNEL in Sumo breaks booting initramfs kernelnormalMedium+Move to YP 2.6.2, YP 2.5.3RESOLVED
   11     


The planed schedule for 2.8 can be found at Yocto_2.8_Schedule

Live Schedule & Execution History

MILESTONE STARTED FINISHED STATUS REPORT
M1 rc1 12/19/2018 1/2/2019 DONE https://wiki.yoctoproject.org/wiki/WW01_-_2019-01-02_-_Full_Test_Cycle_2.7_M1_RC1
M2 rc1 01/28/2019 02/08/2019 DONE https://wiki.yoctoproject.org/wiki/WW01_-_2019-01-28_-_Full_Test_Cycle_2.7_M2_RC1
M2 rc2 02/14/2019 02/25/2019 DONE https://wiki.yoctoproject.org/wiki/WW07_-_2019-02-14_-_Full_Test_Cycle_2.7_M2_RC2
M3 rc1 03/21/2019 04/02/2019 DONE https://wiki.yoctoproject.org/wiki/WW07_-_2019-04-02_-_Full_Test_Cycle_2.7_M3_RC1