QA: Difference between revisions

From Yocto Project
Jump to navigationJump to search
No edit summary
 
(99 intermediate revisions by 5 users not shown)
Line 1: Line 1:
See the http://yoctoproject.org/projects/testing-qa-and-autobuilder page on the website for more information on this project.  
== QA Process==
=== Intro ===
The Yocto Project primarily runs QA through its autobuilder. Patches are pre-tested using the 'a-quick' builds on the project autobuilder https://autobuilder.yoctoproject.org/typhoon/#/console usually in master-next or ross/mut branches. XXX-next branches are also used for testing stable branch changes. We aim to only merge patches which pass these tests. A more comprehensive set of tests is available in 'a-full' builds which may also be used. Test results for all builds are saved in json format and can be handled by the 'resultool' script. Build ouput/results can be seen at https://autobuilder.yocto.io/pub/non-release/.


= QA Docs =
For milestone and main project releases, an 'a-full' build is run as normal but the options to save build output, send a QA email and generate a release are selected. This then triggers a more extensive QA process where other teams can contribute to and extend the autobuilder results as follows:
* Test plans
** [[Yocto 2.1 Overall Test Plan]]
** [[Toaster testing plan]]
** [[SDK Testing Plan]]
** [[Distro Testing Plan]]
** [[BSP Test Plan]]


* Test Cases
Test Trigger
** [[Yocto 2.1 Test Cases]]
Test Planning
Test Execution
Test Result Store
Test Monitoring & Reporting
Release Decision


=== Test Trigger ===
Each QA team can subscribe to QA notification email which is sent from the autobuilder (request through Richard Purdie) and also to the yocto mailing list.  The list of notifications is maintained on config.json in the yocto-autobuilder-helper git repository which has a branch per release.


* Test Cycle
=== Test Planning ===
** [[Testing Cycle]]
Once received the QA notification email, each QA team (eg. intel, windriver, etc) will perform planning on what extra tests they plan to run and when they'll send the results back, then reply to the QA notification email with the planned extra tests and estimate of execution time to QA stakeholders (eg. Richard Purdie, Programme Management (Stephen Jolley) and Release Engineering), yocto mailing list and the lead QA team.  Each QA team can refer to OEQA for automated and manual test cases for their planning.


=== Test Execution ===
Each QA team will execute the planned extra tests. To make sure test result from the test execution could fully integrated to the new QA tooling (resulttool for test result management and reporting/regression), execute OEQA automated tests and OEQA manual tests through resulttool (refer https://wiki.yoctoproject.org/wiki/Resulttool#manualexecution).


* QA Automation
=== Test Result Store ===
**[[AutoBuilder]]
Each QA team will store test result to the remote yocto-testresults-contrib git repository using resulttool (refer https://wiki.yoctoproject.org/wiki/Resulttool#store), then send the QA completion email (include new defects information) to both QA stakeholder and the lead QA team.
**[[Distribution Support‎]]
**[[Oe-selftest]]
**[[Image tests]]
**[[Runtime]]
*** add ptest wiki
*** add piglit test wiki
*** add kernel test wiki
***[[LSB]]
***[[LSB Result]]
***[[LTP]]
***[[LTP result]]
***[[POSIX]]
***[[Posix result]]
***[[POSIX-results]]
***[[POSIX History Results]]
***[[LAVA server vs Yocto HW automation testing]]
**[[Automated package upgrade testing]]


*QA Tracking pages
=== Test Result Repo ===
**[[2.1 QA Owned Features]]
**[[2.1 qa owned bugs]]
**[[2.1 qa owned features to verify]]
**[[Old resolved bugs and features]]


*Reporting
After testing is complete, lead QA team will upload test results and test report in yocto-testresults-contrib git repo (https://git.yoctoproject.org/cgit/cgit.cgi/yocto-testresults-contrib/log/?h=intel-yocto-testresults) and send QA release mail. This is a staging repo where test results and test report are kept for tests run by independent test teams.
**[[Bug reporting and Information levels]]
**[[Testopia]]
**[http://errors.yoctoproject.org/Errors/ error report web]


*1.9 QA Tracking
To access test results, clone the repo https://git.yoctoproject.org/cgit/cgit.cgi/yocto-testresults-contrib/ and checkout QA branch "intel-yocto-testresults"
** [[1.9 qa owned features]]
** [[1.9 qa owned BUGS]]


*1.8 QA Tracking pages
For tests run at Autobuilder, test results and reports for release builds can be find at https://autobuilder.yocto.io/pub/releases/ inside release version folder.
** [[1.8 QA Status]]
** [[1.8 qa run history]]
** [[1.8 qa queries]]
** [[1.8 qa owned features]]
** [[1.8 qa owned bugs by OWNER]]
** [[1.8 qa owned bugs by PRODUCT]]


*Performance testing
=== Test Monitoring & Reporting ===
***[[Performance Test]]
QA stakeholder will monitor testing progress from remote yocto-testresults git repository using resulttool (refer https://wiki.yoctoproject.org/wiki/Resulttool#report).  Once every QA team completed the test execution, the lead QA team will create QA test report and regression using resulttool, then store the report and regression into https://wiki.yoctoproject.org/wiki/Main_Page. Send email report to QA stakeholder and public yocto mailing list.


= QA Old Test plans =
=== Release Decision ===
* Yocto 0.9
QA stakeholder will make the final decision for release.
** Yocto 0.9 [[Overall Test Plan]]
Once decision is taken to release a milestone, RE team will take test results from staging repo and put in https://git.yoctoproject.org/cgit/cgit.cgi/yocto-testresults/ git repo under same commit id as poky.
** Yocto 0.9 [[Milestone Test Report]]


* Yocto 1.0
==Current Release QA info==
**[[Yocto 1.0 Overall Test Plan]]
**[[Yocto 1.0.2 Test Report]]


* Yocto 1.1
Tracking info for current release
**[[Yocto 1.1 Overall Test Plan]]
**[[Yocto 1.1 Test Execution Plan]]
**[[Yocto 1.1 Weekly Test Report]]
**[[Yocto 1.1 Milestone Test Report]]
**[[Yocto 1.1.1 Test Report]]


* Yocto 1.2
* For latest release info, go to https://autobuilder.yocto.io/pub/releases/
**[[Yocto 1.2 Overall Test Plan]]
* For latest QA results, go to https://git.yoctoproject.org/cgit/cgit.cgi/yocto-testresults-contrib/log/?h=intel-yocto-testresults
**[[Yocto 1.2 Test Execution Plan]]
**[[Yocto 1.2 Weekly Test Report]]
**[[Yocto 1.2 Milestone Test Report]]


* Yocto 1.3
==Current Release QA Test Plans==
**[[Yocto 1.3 Overall Test Plan]]
**[[Yocto 1.3 Test Execution Plan]]
**[[Yocto 1.3 Weekly Test Report]]
**[[Yocto 1.3 Milestone Test Report]]


* Yocto 1.4
* [[Yocto Project 3.4_Release Test Plan]]
**[[Yocto 1.4 Overall Test Plan]]
* [[BSP Test Plan]]
**[[Yocto 1.4 Milestone Test Report]]
* [https://lists.yoctoproject.org/pipermail/yocto-perf/  Performance Archives ]


* Yocto 1.5
==Test Execution==
**[[Yocto 1.5 Overall Test Plan]]
===Autobuilder===
**[https://wiki.yoctoproject.org/wiki/1.5_qa_run_history Yocto 1.5 Weekly, Full Pass and Milestone Test Report]
**[[1.5 QA Status]]


* Yocto 1.6
The [[AutoBuilder]] is Yocto Project's tool for non-manual test execution, it performs the following functions:
**[[Yocto 1.6 Overall Test Plan]]
**[https://wiki.yoctoproject.org/wiki/1.6_qa_run_history Yocto 1.6 Weekly, Full Pass and Milestone Test Report]
**[[1.6 QA Status]] - live status report for the current weekly build
**[[1.6 qa owned bugs]]


*Yocto 1.7
* A scheduled nightly build and test execution that includes:
**[[1.7 QA Status]]
** That each image created executes the corresponding set of [[image tests|image/run-time tests]]
**[[1.7 qa owned bugs]]
** Specific Autobuilder tasks for running [[oe-selftest|build-time testing]]
**[[1.7 qa run history]]
* A service for on demand testing requests.
**[[1.7 meta-intel qa run history]]


* Yocto 1.8
===Image Testing===
**[[1.8 QA Status]]
In order to execute tests in an image, it is necessary to boot it in either a virtual or a physical target.  
**[[1.8 qa run history]]
**[[1.8 qa queries]]
**[[1.8 qa owned features]]
**[[1.8 qa owned bugs by OWNER]]
**[[1.8 qa owned bugs by PRODUCT]]


'''Testing Images in Virtual Targets''':


The execution in a virtual environment has a nice flow, documented in the [[Image_tests#Enabling_and_running_the_tests|Image Tests Enabling...]] section.
'''Testing Images in Physical Targets''':
For executing tests in physical targets it would be required to:
# Boot the image in the target by following the [http://www.yoctoproject.org/docs/2.2/mega-manual/mega-manual.html#building-an-image-for-hardware building an image for hardware] instructions.
# Run the image tests by following the [https://wiki.yoctoproject.org/wiki/BSP_Test_Plan#Automated_Tests image test] or [http://www.yoctoproject.org/docs/2.2/mega-manual/mega-manual.html#exporting-tests exporting tests] instructions.
===Creating and Adding New Tests===
Tests for a given component can be automated in the AutoBuilder. With that purpose, follow the [[AutoBuilder/Adding Automated Tests|Adding Automated Tests]] to the Autobuilder Guide.
A list of tests that are automated [[QA#List_of_Automated_Tests|can be seen here]].
==Reporting==
*[[Bug reporting and Information levels]]
*[[https://wiki.yoctoproject.org/wiki/Resulttool#report The Test Reporting Tool]]
*[http://errors.yoctoproject.org/Errors/ error report web]
*[[https://git.yoctoproject.org/cgit/cgit.cgi/yocto-testresults-contrib/ Test results report]]
==Performance testing==
[[Performance Test]]
==QA Resources==
*[[QA Master Test Plan]]
*[[Resulttool]]
*[[https://git.yoctoproject.org/cgit/cgit.cgi/yocto-testresults |qa results Git Repository]]
*[[https://git.yoctoproject.org/cgit/cgit.cgi/yocto-testresults-contrib |qa results staging Git Repository]]
= Archive =
You can find the previous QA work by release in the [[QA/Archive|Yocto Project QA Archive]].
= Other Relevant Data=
* [[Yocto Bug Trend]]
* [[Yocto Bug Trend]]
* Compliance Test Result
* Compliance Test Result
Line 126: Line 107:
** [[Posix result]]
** [[Posix result]]
** [[LSB Result]]
** [[LSB Result]]
* [[ADT Testing]]
* [[Regression Test]]
* [[Performance Test]]
* [[Distro Test]]
* [[Distribution Support]]
* [[Distribution Support]]
* [[QA BKM sharing]]
* [[LAVA server vs Yocto HW automation testing]]
* [[Testopia]]
** Note: The LAVA framework usage stopped in favor of testing in the AutoBuilder in early 2016.
 
 
 


[[Category:QA]]
==List of Automated Tests==
* [[Ptest]]
* add piglit test wiki
* [[Kernel]]
*[[LSB]]
*[[LSB Result]]
*[[LTP]]
*[[LTP result]]
*[[POSIX]]
*[[Posix result]]
*[[POSIX-results]]
*[[POSIX History Results]]
*[[Automated package upgrade testing]]

Latest revision as of 05:23, 8 September 2021

QA Process

Intro

The Yocto Project primarily runs QA through its autobuilder. Patches are pre-tested using the 'a-quick' builds on the project autobuilder https://autobuilder.yoctoproject.org/typhoon/#/console usually in master-next or ross/mut branches. XXX-next branches are also used for testing stable branch changes. We aim to only merge patches which pass these tests. A more comprehensive set of tests is available in 'a-full' builds which may also be used. Test results for all builds are saved in json format and can be handled by the 'resultool' script. Build ouput/results can be seen at https://autobuilder.yocto.io/pub/non-release/.

For milestone and main project releases, an 'a-full' build is run as normal but the options to save build output, send a QA email and generate a release are selected. This then triggers a more extensive QA process where other teams can contribute to and extend the autobuilder results as follows:

Test Trigger
Test Planning 
Test Execution
Test Result Store
Test Monitoring & Reporting
Release Decision

Test Trigger

Each QA team can subscribe to QA notification email which is sent from the autobuilder (request through Richard Purdie) and also to the yocto mailing list. The list of notifications is maintained on config.json in the yocto-autobuilder-helper git repository which has a branch per release.

Test Planning

Once received the QA notification email, each QA team (eg. intel, windriver, etc) will perform planning on what extra tests they plan to run and when they'll send the results back, then reply to the QA notification email with the planned extra tests and estimate of execution time to QA stakeholders (eg. Richard Purdie, Programme Management (Stephen Jolley) and Release Engineering), yocto mailing list and the lead QA team. Each QA team can refer to OEQA for automated and manual test cases for their planning.

Test Execution

Each QA team will execute the planned extra tests. To make sure test result from the test execution could fully integrated to the new QA tooling (resulttool for test result management and reporting/regression), execute OEQA automated tests and OEQA manual tests through resulttool (refer https://wiki.yoctoproject.org/wiki/Resulttool#manualexecution).

Test Result Store

Each QA team will store test result to the remote yocto-testresults-contrib git repository using resulttool (refer https://wiki.yoctoproject.org/wiki/Resulttool#store), then send the QA completion email (include new defects information) to both QA stakeholder and the lead QA team.

Test Result Repo

After testing is complete, lead QA team will upload test results and test report in yocto-testresults-contrib git repo (https://git.yoctoproject.org/cgit/cgit.cgi/yocto-testresults-contrib/log/?h=intel-yocto-testresults) and send QA release mail. This is a staging repo where test results and test report are kept for tests run by independent test teams.

To access test results, clone the repo https://git.yoctoproject.org/cgit/cgit.cgi/yocto-testresults-contrib/ and checkout QA branch "intel-yocto-testresults"

For tests run at Autobuilder, test results and reports for release builds can be find at https://autobuilder.yocto.io/pub/releases/ inside release version folder.

Test Monitoring & Reporting

QA stakeholder will monitor testing progress from remote yocto-testresults git repository using resulttool (refer https://wiki.yoctoproject.org/wiki/Resulttool#report). Once every QA team completed the test execution, the lead QA team will create QA test report and regression using resulttool, then store the report and regression into https://wiki.yoctoproject.org/wiki/Main_Page. Send email report to QA stakeholder and public yocto mailing list.

Release Decision

QA stakeholder will make the final decision for release. Once decision is taken to release a milestone, RE team will take test results from staging repo and put in https://git.yoctoproject.org/cgit/cgit.cgi/yocto-testresults/ git repo under same commit id as poky.

Current Release QA info

Tracking info for current release

Current Release QA Test Plans

Test Execution

Autobuilder

The AutoBuilder is Yocto Project's tool for non-manual test execution, it performs the following functions:

  • A scheduled nightly build and test execution that includes:
  • A service for on demand testing requests.

Image Testing

In order to execute tests in an image, it is necessary to boot it in either a virtual or a physical target.

Testing Images in Virtual Targets:

The execution in a virtual environment has a nice flow, documented in the Image Tests Enabling... section.

Testing Images in Physical Targets:

For executing tests in physical targets it would be required to:

  1. Boot the image in the target by following the building an image for hardware instructions.
  2. Run the image tests by following the image test or exporting tests instructions.

Creating and Adding New Tests

Tests for a given component can be automated in the AutoBuilder. With that purpose, follow the Adding Automated Tests to the Autobuilder Guide.

A list of tests that are automated can be seen here.

Reporting

Performance testing

Performance Test

QA Resources

Archive

You can find the previous QA work by release in the Yocto Project QA Archive.

Other Relevant Data

List of Automated Tests