|
|
| Line 574: |
Line 574: |
| = Pass / Fail Criteria = | | = Pass / Fail Criteria = |
|
| |
|
| This criteria is defined in [[Yocto_Project_v2.7_Status#Yocto_Project_v2.7_Release_Criteria]] | | This criteria is defined in [[Yocto_Project_v2.8_Status#Yocto_Project_v2.8_Release_Criteria]] |
|
| |
|
| = Exit Criteria = | | = Exit Criteria = |
Revision as of 08:25, 18 June 2019
{{#vardefine:ADT_MAUAL | 6539}}
{{#vardefine:ADT_AUTO | 6540}}
{{#vardefine:ANYQEMU_SATO_MANUAL | 6563}}
{{#vardefine:ANYQEMU_SATO_AUTO | 6564}}
{{#vardefine:ANYBSP_SATO_AUTO | 6558}}
{{#vardefine:ANYBSP_LSB_AUTO | 6561}}
{{#vardefine:ANYBSP_WIC_AUTO | 6560}}
{{#vardefine:MMAX_CORE2_MANUAL | 6554}}
{{#vardefine:MMAX_I7_MANUAL | 6557}}
{{#vardefine:NUC_I7_MANUAL | 6556}}
{{#vardefine:NUC_GEX86-64_MANUAL | 6555}}
{{#vardefine:MMAX_GEX86-32_MANUAL | 6553}}
{{#vardefine:GAL_SATO_AUTO | 6565}}
{{#vardefine:CROPS_AUTO | 6541}}
{{#vardefine:CROPS_MANUAL | 6542}}
{{#vardefine:ECLIPSE_MANUAL | 6543}}
{{#vardefine:KERNEL_MANUAL | 6714}}
{{#vardefine:METAYOCTO_MANUAL | 6544}}
{{#vardefine:TOASTER_AUTO | 6545}}
{{#vardefine:TOASTER_MANUAL | 6546}}
{{#vardefine:ESDK_MANUAL | 6547}}
{{#vardefine:ESDK_AUTO | 6548}}
{{#vardefine:OE-CORE_AUTO | 6549}}
{{#vardefine:BITBAKE_AUTO | 6550}}
{{#vardefine:RUNTIME_AUTO | 6551}}
{{#vardefine:BUILD-APPLIANCE_AUTO | 6552}}
Reversion history
| Version |
Modifier |
Comments
|
| 1.0 |
Jain Sangeeta |
First draft
|
References
Documents that support this test plan
Introduction
This is the the overall test plan for version 2.8 of the Yocto Project, for all the generic information this document will be pointing to QA Master Test Plan
Objectives
This test plan for Yocto Project 2.8 release supports the following objectives:
- To define the tools to be used throughout the testing process.
- To communicate to the responsible parties the items to be tested, set expectations around schedule, and define environmental needs.
- To define how the tests will be conducted.
- Release Criteria
- Risks and contingencies
QA_Master_Test_Plan#Objectives_and_Tasks
Risk Analysis & Contingency Plan
This is the document for Risk Analysis https://docs.google.com/spreadsheets/d/1K-MZYo5sfabwI2UozGp6Uuh2sw5y309bXkyEZRPEEgk/edit#gid=0
Test Strategy
Is followed the general strategy defined on QA_Master_Test_Plan#Testing_Strategy and below is showed specific information applicable only for 2.8 Release
Bugs Tracking
- Bugs that need to be implemented by QA Team
- Bugs that need to be verified by QA Team
- Old Bugs that need to be Verified
Test Items
The Yocto Project QA team verifies each release against recent versions of the most popular Linux distributions that provide stable releases.
* centos 7
* Debian 9
* Fedora 26
* Fedora 27
* OpenSuse42.3
* ubuntu 16.04
* ubuntu 17.10
Test Cases
ADT
| Component
|
Environment
|
Auto
|
Manual
|
Owner
|
| ADT
|
Fedora 25 i686
|
Y
|
|
AB
|
| Ubuntu 16.04 x86_64
|
Y
|
|
AB
|
Eclipse Plugin
| Component
|
Environment
|
Auto
|
Manual
|
Owner
|
| Neon + Any Distro
|
|
|
Y
|
PG
|
| Oxygen + Any Distro
|
|
|
Y
|
PG
|
Kernel
| Component
|
Environment
|
Auto
|
Manual
|
Owner
|
| Kernel
|
Any Distro
|
|
Y
|
PG
|
Meta-Yocto
| Component
|
Environment
|
Auto
|
Manual
|
Owner
|
| Meta-Yocto
|
Any Distro
|
|
Y
|
PG
|
Toaster
| Component
|
Environment
|
Auto
|
Manual
|
Owner
|
| Toaster
|
Any Distro
|
|
Y
|
PG
|
eSDK
| Component
|
Environment
|
Auto
|
Manual
|
Owner
|
| eSDK
|
Any Distro + runtime
|
Y
|
|
PG
|
| Any Distro + selftest
|
Y
|
|
PG
|
Selftest
| Component
|
Environment
|
Auto
|
Manual
|
Owner
|
| OE-CORE
|
Ubuntu 16.04
|
Y
|
|
PG
|
| Ubuntu 17.10
|
Y
|
|
PG
|
| OpenSuse 42.3
|
Y
|
|
PG
|
| Debian 9
|
Y
|
|
PG
|
| Centos 7
|
Y
|
|
PG
|
| Fedora 26
|
Y
|
|
PG
|
| Fedora 27
|
Y
|
|
PG
|
Bitbake
| Component
|
Environment
|
Auto
|
Manual
|
Owner
|
| Bitbake
|
Any Distro
|
|
Y
|
PG
|
Build - Apliance
| Component
|
Environment
|
Auto
|
Manual
|
Owner
|
| Build - Apliance
|
VM - Any Distro
|
|
Y
|
PG
|
Package Manager
| Component
|
Environment
|
Auto
|
Manual
|
Owner
|
| Package Manager
|
Any Distro
|
Y
|
|
PG
|
Performance
| Component
|
Environment
|
Auto
|
Manual
|
Owner
|
| Performance
|
Centos 7
|
Y
|
|
PG
|
| Ubuntu 16.04
|
Y
|
|
PG
|
BSP/QEMU
| Component
|
Arch
|
HW
|
Environment
|
Image
|
Auto
|
Manual
|
pTest
|
Compliance
|
| BSP
|
x86
|
MinwTurbot 32bit
|
genericx86
|
core-image-sato-sdk
|
Y
|
|
|
|
| core-image-lsb-sdk
|
Y
|
|
|
|
| MinwTurbot 64bit
|
genericx86-64
|
core-image-sato-sdk
|
Y
|
Y
|
|
|
| core-image-lsb-sdk
|
Y
|
|
|
|
| NUC 6
|
genericx86-64
|
core-image-sato-sdk
|
Y
|
|
Y
|
|
| core-image-lsb-sdk
|
Y
|
|
|
Y
|
| NUC 7
|
genericx86-64
|
core-image-sato-sdk
|
Y
|
Y
|
|
|
| core-image-lsb-sdk
|
Y
|
|
|
|
| QEMU
|
qemux86
|
core-image-sato-sdk
|
Y
|
Y
|
|
|
| core-image-lsb-sdk
|
Y
|
|
|
|
| qemux86-64
|
core-image-sato-sdk
|
Y
|
Y
|
|
|
| core-image-lsb-sdk
|
Y
|
|
|
|
| non x86
|
EdgeRouter
|
EdgeRouter
|
core-image-sato-sdk
|
Y
|
Y
|
|
|
| MPC8315e-rdb
|
MPC8315e-rdb
|
core-image-sato-sdk
|
Y
|
Y
|
|
|
| Beaglebone Black
|
Beaglebone
|
core-image-sato-sdk
|
Y
|
Y
|
|
|
| QEMU
|
qemuarm
|
core-image-sato-sdk
|
Y
|
Y
|
|
|
| qemuarm-64
|
core-image-sato-sdk
|
Y
|
Y
|
|
|
| qemuppc
|
core-image-sato-sdk
|
Y
|
Y
|
|
|
| qemumips
|
core-image-sato-sdk
|
Y
|
Y
|
|
|
| qemumips64
|
core-image-sato-sdk
|
Y
|
Y
|
|
|
| Meta-Intel
|
MinwTurbot 32bit
|
core2-32
|
core-image-sato-sdk
|
Y
|
|
|
|
| core-image-lsb-sdk
|
Y
|
|
|
|
| MinwTurbot 64bit
|
corei7-64
|
core-image-sato-sdk
|
Y
|
Y
|
|
|
| core-image-lsb-sdk
|
Y
|
|
|
|
| NUC 6
|
corei7-64
|
core-image-sato-sdk
|
Y
|
|
|
|
| core-image-lsb-sdk
|
Y
|
|
|
|
| NUC 7
|
corei7-64
|
core-image-sato-sdk
|
Y
|
Y
|
|
|
| core-image-lsb-sdk
|
Y
|
|
|
|
| poky-tiny
|
|
Y
|
|
|
All component to be tested on 2.7 are in #Test Items a detailed list of features per component can be found in Test Plan of the components QA_Master_Test_Plan#Component
Features pending to be tested
Here is a list of areas that are not yet covered and need some work, implement test cases and add it to a component that belongs or create a new test plan for new component.
ADD FEATURES !!!!!!!!!!
The complete List of QA Implementations for YP 2.7 can be found at Features Implementation by QA Team
Features Not to be tested
Approach
Tests will be conducted per the documented test cases stored in Testopia. The test manager will create test runs for each tester. The tester will execute the tests in Testopia and mark each case as Pass / Fail / Skip /Blocked.
The tester should leave notes on actual results and any other relevant details when possible.
Once complete, the test manager should review the test run reports in Testopia and report back to the team by mail and in the wiki as a test report.
Pass / Fail Criteria
This criteria is defined in Yocto_Project_v2.8_Status#Yocto_Project_v2.8_Release_Criteria
Exit Criteria
As defined in QA_Master_Test_Plan#Entry and Exit Criteria
Test Deliverables
- Final Test results
- Final Test report
- Mail sent to the mailing list
Test Environment
There are different environments and tools for every type of testing refer to QA_Master_Test_Plan#Test_Areas for more detailed information.
Schedule
M1 (Apr 29, 2019 to Jun 21, 2019)
M1 Feature/Tasks Development / Cutoff is Jun. 9, 2019
| ID | Requester | Assignee | E | Summary (12 tasks) ⇒ | Severity | P | Whiteboard | Status |
|---|
| 13336 | Adrian Freihofer | Adrian Freihofer | | runqemu: support non bootable wic images | enhancement | Medium+ | | RESOLVED |
| 12955 | Changqing Li | Changqing Li | 2 | update-rc.d: Provide option to permanently disable service, also after upgrade. | enhancement | Medium | | RESOLVED |
| 10416 | Paul Eggleton | chandana kalluri | 5 | Extensible SDK: provide easy means of reconfiguring the kernel | enhancement | Medium+ | | RESOLVED |
| 13059 | Jon Mason | Jon Mason | 5 | Aarch64 builder support | enhancement | High | | RESOLVED |
| 13060 | Jon Mason | Jon Mason | 5 | Arm ptests | enhancement | Medium+ | | RESOLVED |
| 13264 | Tim Orling | Maciej Pijanowski | | Refactor recipetool/devtool to support python3 (only) | enhancement | Medium+ | | RESOLVED |
| 13318 | Richard Purdie | New Comer Bugs | | Add SPDX license headers to all source files for pseudo | enhancement | Medium+ | NEWCOMER | RESOLVED |
| 13327 | Richard Purdie | Richard Purdie | | Extract ptest logs from ptest result files automatically on the autobuilder | enhancement | High | | RESOLVED |
| 11168 | Henry Bruce | Richard Purdie | | multiconfig: use 'mc:' qualifier instead of 'multiconfig:' | enhancement | Medium+ | | RESOLVED |
| 13328 | Richard Purdie | Richard Purdie | | Update pybootchart to work with python3 | enhancement | Medium+ | | RESOLVED |
| 13316 | Richard Purdie | Richard Purdie | | Add SPDX license headers to all python/script files within OE-Core | enhancement | Medium+ | | RESOLVED |
| 13326 | Richard Purdie | Unassigned | | Index the build artefacts directory | enhancement | Medium+ | | RESOLVED |
| | | | 17 | | | | | |
M1 Bugs
| ID | Requester | Assignee | E | Summary (49 tasks) ⇒ | Severity | P | Whiteboard | Status |
|---|
| 13295 | Joshua Watt | Armin Kuster | | openssh ptest results are inconsistent per image | normal | Medium+ | | RESOLVED |
| 13391 | Richard Purdie | Alexander Kanavin | | acl ptest timeout due to perl update | normal | High | | RESOLVED |
| 13363 | Sangeeta Jain | Anuj Mittal | | LogrotateTest (lib/oeqa/runtime/cases/logrotate.py) failed. | minor | Medium+ | Backport to 2.7 | RESOLVED |
| 13396 | Sudhir Sharma | Anuj Mittal | | [QA 2.8 M1 RC1][BSP HW] parselogs.ParseLogsTest.test_parselogs failure on coffeelake | normal | Medium+ | | RESOLVED |
| 13441 | Ross Burton | Anuj Mittal | | Enable CONFIG_RANDOM_TRUST_CPU | normal | Medium+ | | RESOLVED |
| 13248 | Richard Purdie | Apoorv | | New QA process issues | normal | High | | RESOLVED |
| 12923 | Gary | Paul Eggleton | | recipetool create error on empty setup.py | normal | Medium | | RESOLVED |
| 13083 | Ross Burton | Paul Eggleton | | recipetool create doesn't handle Python 3 | normal | Medium+ | | RESOLVED |
| 13168 | Romuald Jeanne | Paul Eggleton | | devtool: cannot add new recipe if reproducible_build class is inherited | normal | Medium+ | Backport 2.5 and 2.6 | RESOLVED |
| 13404 | Alexander Kanavin | Paul Eggleton | | 'devtool upgrade' tracebacks when a recipe includes another recipe with PV in the name | normal | Medium+ | | RESOLVED |
| 13422 | Ross Burton | Paul Eggleton | | recipetool.RecipetoolTests.test_recipetool_load_plugin fails | normal | Medium+ | | RESOLVED |
| 13284 | Armin Kuster | Changqing Li | | Patchtest seems to be off-line and not working | normal | High | | RESOLVED |
| 13416 | | Changqing Li | | gcc 9.1 runtime do_install error | major | Medium+ | | RESOLVED |
| 13356 | Emil Melar | David Reyna | 1 | oe-core.xml in thud branch refers to wrong bitbake version | normal | Medium+ | also backport to Thud (2.6) and Warrior (2.7) | RESOLVED |
| 13382 | Jaewon Lee | Jaewon Lee | | devtool build should also deploy artifact | normal | Medium | | RESOLVED |
| 13377 | Jaewon Lee | Jaewon Lee | | gst-inspect on target is looking for a .so on host | normal | Medium+ | | RESOLVED |
| 13379 | Armin Kuster | Jon Mason | | qemuarm64 runtime syslog test failure in testimage | normal | High | | RESOLVED |
| 12962 | Ross Burton | Joshua Watt | | [meta-mingw] update-alternative warnings | normal | Medium | | RESOLVED |
| 13301 | Joshua Watt | Mariano Lopez | 2 | util-linux ptest results are inconsistent per image | normal | Medium+ | | RESOLVED |
| 13309 | Armin Kuster | Kevin Hao | | [master] qemuppc fails to shutdown intermittently | normal | High | with backport | RESOLVED |
| 13418 | Ross Burton | Robert Yang | | Multilib and update-alternatives conflict | normal | Medium+ | | RESOLVED |
| 13133 | Florian Kauer | Mark Hatle | | Git submodule fetcher fails if last component of path is different than the URL | major | Medium+ | | RESOLVED |
| 13340 | Matthias Schoepfer | Matthias Schoepfer | | python3 build fails when target is mips softfloat | normal | Medium | | RESOLVED |
| 13431 | Ross Burton | Michael Halstead | | Add new Debian Stable (Buster) autobuilder worker | normal | Medium+ | | RESOLVED |
| 13290 | Joshua Watt | Mingli Yu | | elfutils ptest results are inconsistent per image | normal | Medium+ | | RESOLVED |
| 13294 | Joshua Watt | Mingli Yu | | nettle ptest results are inconsistent per image | normal | Medium+ | | RESOLVED |
| 13339 | Nam Nguyen | Mingli Yu | | Ruby 2.4.4 does not run with Apalis iMX6 | normal | Medium+ | | RESOLVED |
| 13243 | Paul Eggleton | New Comer Bugs | | devtool test failure in python scanning code | normal | Medium+ | NEWCOMER | RESOLVED |
| 13258 | Kay Liu | Otavio Salvador | | i.MX6 solox platform CPU hang in the case of using the Linux OS | normal | Medium | | RESOLVED |
| 11776 | Patrick Ohly | Chen Qi | 2 | read-only rootfs + systemd: systemd-tmpfiles fails | normal | Medium+ | | RESOLVED |
| 13061 | Richard Purdie | Chen Qi | | oe-selftest imagefeatures.ImageFeatures.test_image_fstypes (ubifs) | normal | Medium+ | | RESOLVED |
| 13390 | Richard Purdie | Chen Qi | | oe-selftest qemu process handling issue | normal | Medium+ | | RESOLVED |
| 12991 | pravin | Richard Purdie | | [2.6 M4 RC1][Build-Appliance] Bitbake build-appliance-image getting failed during building image due to webkitgtk package | normal | Medium+ | | RESOLVED |
| 13208 | Jaewon Lee | Richard Purdie | | Python2.7 check in oe-buildenv-internal is erroring | major | Medium+ | | RESOLVED |
| 13280 | Denys Dmytriyenko | Richard Purdie | | Race condition in wic to package IMAGE_BOOT_FILES before EXTRA_IMAGEDEPENDS are built | normal | Medium+ | | RESOLVED |
| 13299 | Joshua Watt | Richard Purdie | | quilt ptest results are inconsistent per image | normal | Medium+ | NEWCOMER | RESOLVED |
| 12984 | Joe Slater | Richard Purdie | | Specifying invalid PRSERV_HOST causes bitbake to hang | normal | Medium+ | | RESOLVED |
| 13057 | Paul Eggleton | Ross Burton | | No default network configuration for systemd-networkd | normal | Medium+ | | RESOLVED |
| 13366 | Richard Purdie | Ross Burton | | bluez5 ptest hangs with gcc 9 | normal | Medium+ | | RESOLVED |
| 13072 | Tim Orling | Ross Burton | | vulkan: upstream repo has changed; latest version is 1.1.92.1 | normal | Medium+ | | RESOLVED |
| 13423 | Richard Purdie | Ross Burton | | efivar fails to build on opensuse423 with musl | normal | Medium+ | | RESOLVED |
| 13291 | Joshua Watt | Sakib Sajal | | flex ptest results are inconsistent per image | normal | Medium+ | | RESOLVED |
| 12857 | Kai Ruhnau | Seebs | | Running anything dotnet/CoreCLR based in a devshell doesn't work | normal | Medium+ | | RESOLVED |
| 12873 | Pau Espin Pedrol | Tim Orling | 5 | cpan_build.bbclass is broken (no dependency Module::Build) | normal | Medium+ | Backport to 2.6.x, 2.7.x | RESOLVED |
| 13200 | Tim Orling | Tim Orling | | perl-rdepends.txt missing many dependencies for perl-module-test-more | normal | Medium+ | Backport to 2.7 | RESOLVED |
| 13381 | Tim Orling | Tim Orling | 1 | SCons should migrate to Python 3 | normal | Undecided | | RESOLVED |
| 13293 | Joshua Watt | Unassigned | | libxml-parser-perl ptest results are inconsistent per image | normal | Medium+ | NEWCOMER | RESOLVED |
| 13300 | Joshua Watt | Unassigned | | tcl ptest results are inconsistent per image | normal | Medium+ | NEWCOMER | RESOLVED |
| 12886 | Ferry | He Zhe | | Removal of VM_DEFAULT_KERNEL in Sumo breaks booting initramfs kernel | normal | Medium+ | Move to YP 2.6.2, YP 2.5.3 | RESOLVED |
| | | | 11 | | | | | |
The planed schedule for 2.8 can be found at Yocto_2.8_Schedule
Live Schedule & Execution History
