Yocto Project 2.8 Release Test Plan: Difference between revisions

From Yocto Project
Jump to navigationJump to search
Line 583: Line 583:




* Live Status [[2.7_QA_Status]]
* Final Test results
* Final execution Report [[#Execution History]]
* Final Test report
* Mail sent to the mailing list
* Mail sent to the mailing list



Revision as of 08:24, 18 June 2019

{{#vardefine:ADT_MAUAL | 6539}} {{#vardefine:ADT_AUTO | 6540}} {{#vardefine:ANYQEMU_SATO_MANUAL | 6563}} {{#vardefine:ANYQEMU_SATO_AUTO | 6564}} {{#vardefine:ANYBSP_SATO_AUTO | 6558}} {{#vardefine:ANYBSP_LSB_AUTO | 6561}} {{#vardefine:ANYBSP_WIC_AUTO | 6560}} {{#vardefine:MMAX_CORE2_MANUAL | 6554}} {{#vardefine:MMAX_I7_MANUAL | 6557}} {{#vardefine:NUC_I7_MANUAL | 6556}} {{#vardefine:NUC_GEX86-64_MANUAL | 6555}} {{#vardefine:MMAX_GEX86-32_MANUAL | 6553}} {{#vardefine:GAL_SATO_AUTO | 6565}} {{#vardefine:CROPS_AUTO | 6541}} {{#vardefine:CROPS_MANUAL | 6542}} {{#vardefine:ECLIPSE_MANUAL | 6543}} {{#vardefine:KERNEL_MANUAL | 6714}} {{#vardefine:METAYOCTO_MANUAL | 6544}} {{#vardefine:TOASTER_AUTO | 6545}} {{#vardefine:TOASTER_MANUAL | 6546}} {{#vardefine:ESDK_MANUAL | 6547}} {{#vardefine:ESDK_AUTO | 6548}} {{#vardefine:OE-CORE_AUTO | 6549}} {{#vardefine:BITBAKE_AUTO | 6550}} {{#vardefine:RUNTIME_AUTO | 6551}} {{#vardefine:BUILD-APPLIANCE_AUTO | 6552}}

Reversion history

Version Modifier Comments
1.0 Jain Sangeeta First draft

References

Documents that support this test plan

Introduction

This is the the overall test plan for version 2.8 of the Yocto Project, for all the generic information this document will be pointing to QA Master Test Plan

Objectives

This test plan for Yocto Project 2.8 release supports the following objectives:

  • To define the tools to be used throughout the testing process.
  • To communicate to the responsible parties the items to be tested, set expectations around schedule, and define environmental needs.
  • To define how the tests will be conducted.
  • Release Criteria
  • Risks and contingencies

QA_Master_Test_Plan#Objectives_and_Tasks

Risk Analysis & Contingency Plan

This is the document for Risk Analysis https://docs.google.com/spreadsheets/d/1K-MZYo5sfabwI2UozGp6Uuh2sw5y309bXkyEZRPEEgk/edit#gid=0

Test Strategy

Is followed the general strategy defined on QA_Master_Test_Plan#Testing_Strategy and below is showed specific information applicable only for 2.8 Release

Bugs Tracking

Test Items

The Yocto Project QA team verifies each release against recent versions of the most popular Linux distributions that provide stable releases.

* centos 7 
* Debian 9
* Fedora 26
* Fedora 27
* OpenSuse42.3
* ubuntu 16.04 
* ubuntu 17.10

Test Cases

ADT

Component Environment Auto Manual Owner
ADT Fedora 25 i686 Y AB
Ubuntu 16.04 x86_64 Y AB

Eclipse Plugin

Component Environment Auto Manual Owner
Neon + Any Distro Y PG
Oxygen + Any Distro Y PG

Kernel

Component Environment Auto Manual Owner
Kernel Any Distro Y PG

Meta-Yocto

Component Environment Auto Manual Owner
Meta-Yocto Any Distro Y PG

Toaster

Component Environment Auto Manual Owner
Toaster Any Distro Y PG

eSDK

Component Environment Auto Manual Owner
eSDK Any Distro + runtime Y PG
Any Distro + selftest Y PG

Selftest

Component Environment Auto Manual Owner
OE-CORE Ubuntu 16.04 Y PG
Ubuntu 17.10 Y PG
OpenSuse 42.3 Y PG
Debian 9 Y PG
Centos 7 Y PG
Fedora 26 Y PG
Fedora 27 Y PG

Bitbake

Component Environment Auto Manual Owner
Bitbake Any Distro Y PG

Build - Apliance

Component Environment Auto Manual Owner
Build - Apliance VM - Any Distro Y PG

Package Manager

Component Environment Auto Manual Owner
Package Manager Any Distro Y PG

Performance

Component Environment Auto Manual Owner
Performance Centos 7 Y PG
Ubuntu 16.04 Y PG

BSP/QEMU

Component Arch HW Environment Image Auto Manual pTest Compliance
BSP x86 MinwTurbot 32bit genericx86 core-image-sato-sdk Y
core-image-lsb-sdk Y
MinwTurbot 64bit genericx86-64 core-image-sato-sdk Y Y
core-image-lsb-sdk Y
NUC 6 genericx86-64 core-image-sato-sdk Y Y
core-image-lsb-sdk Y Y
NUC 7 genericx86-64 core-image-sato-sdk Y Y
core-image-lsb-sdk Y
QEMU qemux86 core-image-sato-sdk Y Y
core-image-lsb-sdk Y
qemux86-64 core-image-sato-sdk Y Y
core-image-lsb-sdk Y
non x86 EdgeRouter EdgeRouter core-image-sato-sdk Y Y
MPC8315e-rdb MPC8315e-rdb core-image-sato-sdk Y Y
Beaglebone Black Beaglebone core-image-sato-sdk Y Y
QEMU qemuarm core-image-sato-sdk Y Y
qemuarm-64 core-image-sato-sdk Y Y
qemuppc core-image-sato-sdk Y Y
qemumips core-image-sato-sdk Y Y
qemumips64 core-image-sato-sdk Y Y
Meta-Intel MinwTurbot 32bit core2-32 core-image-sato-sdk Y
core-image-lsb-sdk Y
MinwTurbot 64bit corei7-64 core-image-sato-sdk Y Y
core-image-lsb-sdk Y
NUC 6 corei7-64 core-image-sato-sdk Y
core-image-lsb-sdk Y
NUC 7 corei7-64 core-image-sato-sdk Y Y
core-image-lsb-sdk Y
poky-tiny Y

All component to be tested on 2.7 are in #Test Items a detailed list of features per component can be found in Test Plan of the components QA_Master_Test_Plan#Component

Features pending to be tested

Here is a list of areas that are not yet covered and need some work, implement test cases and add it to a component that belongs or create a new test plan for new component.

ADD FEATURES !!!!!!!!!!


The complete List of QA Implementations for YP 2.7 can be found at Features Implementation by QA Team

Features Not to be tested

Approach

Tests will be conducted per the documented test cases stored in Testopia. The test manager will create test runs for each tester. The tester will execute the tests in Testopia and mark each case as Pass / Fail / Skip /Blocked. The tester should leave notes on actual results and any other relevant details when possible.

Once complete, the test manager should review the test run reports in Testopia and report back to the team by mail and in the wiki as a test report.

Pass / Fail Criteria

This criteria is defined in Yocto_Project_v2.7_Status#Yocto_Project_v2.7_Release_Criteria

Exit Criteria

As defined in QA_Master_Test_Plan#Entry and Exit Criteria

Test Deliverables

  • Final Test results
  • Final Test report
  • Mail sent to the mailing list

Test Environment

There are different environments and tools for every type of testing refer to QA_Master_Test_Plan#Test_Areas for more detailed information.

Schedule

M1 (Apr 29, 2019 to Jun 21, 2019)

M1 Feature/Tasks Development / Cutoff is Jun. 9, 2019

IDRequesterAssigneeESummary (12 tasks) SeverityPWhiteboardStatus
13336Adrian FreihoferAdrian Freihoferrunqemu: support non bootable wic imagesenhancementMedium+RESOLVED
12955Changqing LiChangqing Li2update-rc.d: Provide option to permanently disable service, also after upgrade.enhancementMediumRESOLVED
10416Paul Eggletonchandana kalluri5Extensible SDK: provide easy means of reconfiguring the kernelenhancementMedium+RESOLVED
13059Jon MasonJon Mason5Aarch64 builder supportenhancementHighRESOLVED
13060Jon MasonJon Mason5Arm ptestsenhancementMedium+RESOLVED
13264Tim OrlingMaciej PijanowskiRefactor recipetool/devtool to support python3 (only)enhancementMedium+RESOLVED
13318Richard PurdieNew Comer BugsAdd SPDX license headers to all source files for pseudoenhancementMedium+NEWCOMERRESOLVED
13327Richard PurdieRichard PurdieExtract ptest logs from ptest result files automatically on the autobuilderenhancementHighRESOLVED
11168Henry BruceRichard Purdiemulticonfig: use 'mc:' qualifier instead of 'multiconfig:'enhancementMedium+RESOLVED
13328Richard PurdieRichard PurdieUpdate pybootchart to work with python3enhancementMedium+RESOLVED
13316Richard PurdieRichard PurdieAdd SPDX license headers to all python/script files within OE-CoreenhancementMedium+RESOLVED
13326Richard PurdieUnassignedIndex the build artefacts directoryenhancementMedium+RESOLVED
   17     

M1 Bugs

IDRequesterAssigneeESummary (49 tasks) SeverityPWhiteboardStatus
13295Joshua WattArmin Kusteropenssh ptest results are inconsistent per imagenormalMedium+RESOLVED
13391Richard PurdieAlexander Kanavinacl ptest timeout due to perl updatenormalHighRESOLVED
13363Sangeeta JainAnuj MittalLogrotateTest (lib/oeqa/runtime/cases/logrotate.py) failed.minorMedium+Backport to 2.7RESOLVED
13396Sudhir SharmaAnuj Mittal[QA 2.8 M1 RC1][BSP HW] parselogs.ParseLogsTest.test_parselogs failure on coffeelakenormalMedium+RESOLVED
13441Ross BurtonAnuj MittalEnable CONFIG_RANDOM_TRUST_CPUnormalMedium+RESOLVED
13248Richard PurdieApoorvNew QA process issuesnormalHighRESOLVED
12923GaryPaul Eggletonrecipetool create error on empty setup.pynormalMediumRESOLVED
13083Ross BurtonPaul Eggletonrecipetool create doesn't handle Python 3normalMedium+RESOLVED
13168Romuald JeannePaul Eggletondevtool: cannot add new recipe if reproducible_build class is inheritednormalMedium+Backport 2.5 and 2.6RESOLVED
13404Alexander KanavinPaul Eggleton'devtool upgrade' tracebacks when a recipe includes another recipe with PV in the namenormalMedium+RESOLVED
13422Ross BurtonPaul Eggletonrecipetool.RecipetoolTests.test_recipetool_load_plugin failsnormalMedium+RESOLVED
13284Armin KusterChangqing LiPatchtest seems to be off-line and not workingnormalHighRESOLVED
13416 Changqing Ligcc 9.1 runtime do_install errormajorMedium+RESOLVED
13356Emil MelarDavid Reyna1oe-core.xml in thud branch refers to wrong bitbake versionnormalMedium+also backport to Thud (2.6) and Warrior (2.7)RESOLVED
13382Jaewon LeeJaewon Leedevtool build should also deploy artifactnormalMediumRESOLVED
13377Jaewon LeeJaewon Leegst-inspect on target is looking for a .so on hostnormalMedium+RESOLVED
13379Armin KusterJon Masonqemuarm64 runtime syslog test failure in testimagenormalHighRESOLVED
12962Ross BurtonJoshua Watt[meta-mingw] update-alternative warningsnormalMediumRESOLVED
13301Joshua WattMariano Lopez2util-linux ptest results are inconsistent per imagenormalMedium+RESOLVED
13309Armin KusterKevin Hao[master] qemuppc fails to shutdown intermittentlynormalHighwith backportRESOLVED
13418Ross BurtonRobert YangMultilib and update-alternatives conflictnormalMedium+RESOLVED
13133Florian KauerMark HatleGit submodule fetcher fails if last component of path is different than the URLmajorMedium+RESOLVED
13340Matthias SchoepferMatthias Schoepferpython3 build fails when target is mips softfloatnormalMediumRESOLVED
13431Ross BurtonMichael HalsteadAdd new Debian Stable (Buster) autobuilder workernormalMedium+RESOLVED
13290Joshua WattMingli Yuelfutils ptest results are inconsistent per imagenormalMedium+RESOLVED
13294Joshua WattMingli Yunettle ptest results are inconsistent per imagenormalMedium+RESOLVED
13339Nam NguyenMingli YuRuby 2.4.4 does not run with Apalis iMX6normalMedium+RESOLVED
13243Paul EggletonNew Comer Bugsdevtool test failure in python scanning codenormalMedium+NEWCOMERRESOLVED
13258Kay LiuOtavio Salvadori.MX6 solox platform CPU hang in the case of using the Linux OSnormalMediumRESOLVED
11776Patrick OhlyChen Qi2read-only rootfs + systemd: systemd-tmpfiles failsnormalMedium+RESOLVED
13061Richard PurdieChen Qioe-selftest imagefeatures.ImageFeatures.test_image_fstypes (ubifs)normalMedium+RESOLVED
13390Richard PurdieChen Qioe-selftest qemu process handling issuenormalMedium+RESOLVED
12991pravinRichard Purdie[2.6 M4 RC1][Build-Appliance] Bitbake build-appliance-image getting failed during building image due to webkitgtk packagenormalMedium+RESOLVED
13208Jaewon LeeRichard PurdiePython2.7 check in oe-buildenv-internal is erroringmajorMedium+RESOLVED
13280Denys DmytriyenkoRichard PurdieRace condition in wic to package IMAGE_BOOT_FILES before EXTRA_IMAGEDEPENDS are builtnormalMedium+RESOLVED
13299Joshua WattRichard Purdiequilt ptest results are inconsistent per imagenormalMedium+NEWCOMERRESOLVED
12984Joe SlaterRichard PurdieSpecifying invalid PRSERV_HOST causes bitbake to hangnormalMedium+RESOLVED
13057Paul EggletonRoss BurtonNo default network configuration for systemd-networkdnormalMedium+RESOLVED
13366Richard PurdieRoss Burtonbluez5 ptest hangs with gcc 9normalMedium+RESOLVED
13072Tim OrlingRoss Burtonvulkan: upstream repo has changed; latest version is 1.1.92.1normalMedium+RESOLVED
13423Richard PurdieRoss Burtonefivar fails to build on opensuse423 with muslnormalMedium+RESOLVED
13291Joshua WattSakib Sajalflex ptest results are inconsistent per imagenormalMedium+RESOLVED
12857Kai RuhnauSeebsRunning anything dotnet/CoreCLR based in a devshell doesn't worknormalMedium+RESOLVED
12873Pau Espin PedrolTim Orling5cpan_build.bbclass is broken (no dependency Module::Build)normalMedium+Backport to 2.6.x, 2.7.xRESOLVED
13200Tim OrlingTim Orlingperl-rdepends.txt missing many dependencies for perl-module-test-morenormalMedium+Backport to 2.7RESOLVED
13381Tim OrlingTim Orling1SCons should migrate to Python 3normalUndecidedRESOLVED
13293Joshua WattUnassignedlibxml-parser-perl ptest results are inconsistent per imagenormalMedium+NEWCOMERRESOLVED
13300Joshua WattUnassignedtcl ptest results are inconsistent per imagenormalMedium+NEWCOMERRESOLVED
12886FerryHe ZheRemoval of VM_DEFAULT_KERNEL in Sumo breaks booting initramfs kernelnormalMedium+Move to YP 2.6.2, YP 2.5.3RESOLVED
   11     


The planed schedule for 2.8 can be found at Yocto_2.8_Schedule

Live Schedule & Execution History

MILESTONE STARTED FINISHED STATUS REPORT
M1 rc1 12/19/2018 1/2/2019 DONE https://wiki.yoctoproject.org/wiki/WW01_-_2019-01-02_-_Full_Test_Cycle_2.7_M1_RC1
M2 rc1 01/28/2019 02/08/2019 DONE https://wiki.yoctoproject.org/wiki/WW01_-_2019-01-28_-_Full_Test_Cycle_2.7_M2_RC1
M2 rc2 02/14/2019 02/25/2019 DONE https://wiki.yoctoproject.org/wiki/WW07_-_2019-02-14_-_Full_Test_Cycle_2.7_M2_RC2
M3 rc1 03/21/2019 04/02/2019 DONE https://wiki.yoctoproject.org/wiki/WW07_-_2019-04-02_-_Full_Test_Cycle_2.7_M3_RC1