QA-wip: Difference between revisions

From Yocto Project
Jump to navigationJump to search
No edit summary
No edit summary
 
(26 intermediate revisions by the same user not shown)
Line 1: Line 1:
==New QA Process==
== QA Process==
*[[New_QA_process]] - add description here only, no need of separate link
=== Intro ===
Given the QA tooling (resulttool) available to manage QA test results and reporting, here is the QA process. First release to implement this process was 2.7_M3_RC1.


==Current Release QA info==
This QA process consists below:
Test Trigger
Test Planning
Test Execution
Test Result Store
Test Monitoring & Reporting
Release Decision
 
=== Test Trigger ===
Each QA team will subscribe to QA notification email (request through Richard Purdie).  The list of notifications is maintained on config.json in the yocto- autobuilder-helper which has a branch per release.
 
=== Test Planning ===
Once received the QA notification email, each QA team (eg. intel, windriver, etc) will perform planning on what extra tests they plan to run and when they'll send the results back, then reply to the QA notification email with the planned extra tests and estimate of execution time to QA stakeholders (eg. Richard Purdie, Stephen Jolley), yocto mailing list and the lead QA team.  Each QA team can refer to OEQA for automated and manual test cases for their planning.  The lead QA team no longer need to setup Testopia and wiki page. 
 
=== Test Execution ===
Each QA team will execute the planned extra tests. To make sure test result from the test execution could fully integrated to the new QA tooling (resulttool for test result management and reporting/regression), execute OEQA automated tests and OEQA manual tests through resulttool (refer https://wiki.yoctoproject.org/wiki/Resulttool#manualexecution).
 
=== Test Result Store ===
Each QA team will store test result to the remote yocto-testresults-contrib git repository using resulttool (refer https://wiki.yoctoproject.org/wiki/Resulttool#store), then send the QA completion email (include new defects information) to both QA stakeholder and the lead QA team.


Define tracking info for new process - current release
=== Test Result Repo ===


==Current Release QA Trackers== -- move to archive for 2.7
After testing is complete, lead QA team will upload test results and test report in yocto-testresults-contrib git repo (https://git.yoctoproject.org/cgit/cgit.cgi/yocto-testresults-contrib/log/?h=intel-yocto-testresults) and send QA release mail. This is a staging repo where test results and test report are kept for tests run by independent test teams.
*Test Plan
**[[Yocto Project 2.7_Release Test Plan]] --redefine for updated scope


*Live Status
To access test results, clone the repo https://git.yoctoproject.org/cgit/cgit.cgi/yocto-testresults-contrib/ and checkout QA branch "intel-yocto-testresults"
**[[2.7 QA Status]]- no need


*QA Execution History
For tests run at Autobuilder, test results and reports can be find at https://autobuilder.yocto.io/pub/releases/ inside release version folder.
**[[Yocto_Project_2.7_Release_Test_Plan#Live_Schedule_.26_Execution_History | 2.7 qa run history]] - no need


*Bugs that need to be implemented by QA Team
=== Test Monitoring & Reporting ===
**[[2.7_QA_OWNED_BUGS | 2.7 QA Assigned Bugs]] - TBD
QA stakeholder will monitor testing progress from remote yocto-testresults git repository using resulttool (refer https://wiki.yoctoproject.org/wiki/Resulttool#report).  Once every QA team completed the test execution, the lead QA team will create QA test report and regression using resulttool, then store the report and regression into https://wiki.yoctoproject.org/wiki/Main_Page. Send email report to QA stakeholder and public yocto mailing list.


*Bugs that need to be verified by QA Team
=== Release Decision ===
**[[2.7_QA_Bugs_To_Verify]] - TBD
QA stakeholder will make the final decision for release.
<!--
Once decision is taken to release a milestone, RE team will take test results from staging repo and put in https://git.yoctoproject.org/cgit/cgit.cgi/yocto-testresults/ git repo under same commit id as poky.
*Features to verify
**[[2.5 qa_owned features to verify]] - no need


*Features to implement
==Current Release QA info==
**[[2.5 qa owned features]] - no need
-->


*Old Bugs that need to be Verified
Tracking info for current release
**[[Old resolved bugs and features]] - no need


*2.7 Test Run Templates
* For latest release info, go to https://autobuilder.yocto.io/pub/releases/
**[[Yocto_Project_2.7_Release_Test_Plan#Test_Items | Test Run Templates]] - no need
* For latest QA results, go to https://git.yoctoproject.org/cgit/cgit.cgi/yocto-testresults-contrib/log/?h=intel-yocto-testresults


==Current Release QA Test Plans==
==Current Release QA Test Plans==
* [[QA Master Test Plan]] -- keep the link; update page
* [[Yocto Project 2.7_Release Test Plan]] -- TBD
* [[2.7 QA Status]] - no need
* [https://wiki.yoctoproject.org/charts/perf_milestone_GDC/performance_test.html  Performance Charts ] -- TBD
* [[Toaster testing plan]] -- TBD
* [[Extensible SDK Test Plan (eSDK)]] -- TBD
* [[Distro Testing Plan]] -- TBD
* [[BSP Test Plan]] -- TBD
==Current Release QA Test Cases==
* [[Yocto 2.7 Test Cases]] - no need


* [[Yocto Project 2.8_Release Test Plan]]
* [[BSP Test Plan]]
* [https://lists.yoctoproject.org/pipermail/yocto-perf/  Performance Archives ]


==Test Execution== -- TBD
==Test Execution==  
===Autobuilder===
===Autobuilder===


Line 87: Line 89:


==Reporting==
==Reporting==
*[[Bug reporting and Information levels]] -- keep but review
*[[Bug reporting and Information levels]]
*[[Testopia]], the test manager -- no need
*[[https://wiki.yoctoproject.org/wiki/Resulttool#report The Test Reporting Tool]]
*[[QA/Test_Reporting_Tool|The Test Reporting Tool]] -- TBD
*[http://errors.yoctoproject.org/Errors/ error report web]  
*[http://errors.yoctoproject.org/Errors/ error report web] --TBD
*[[https://git.yoctoproject.org/cgit/cgit.cgi/yocto-testresults-contrib/ Test results report]]
*[[QA REPORT TEMPLATE | Wiki of QA Report Template]] -- no need


==Performance testing==
==Performance testing==
[[Performance Test]] -- TBD
[[Performance Test]]


==QA Resources==
==QA Resources==
*[[Rpm's Repository Setup for QA]] -- TBD
*[[QA Master Test Plan]]
*[[Testopia]] -- TBD
*[[Rpm's Repository Setup for QA]]  
*[[Testing Cycle]] -- TBD
*[[Resulttool]]  
*[[qa-tools|qa-tools Git Repository]] -- TBD
*[[Testing Cycle]]  
*[[https://git.yoctoproject.org/cgit/cgit.cgi/yocto-testresults-contrib |qa results Git Repository]]


= Archive =
= Archive =

Latest revision as of 05:46, 21 June 2019

QA Process

Intro

Given the QA tooling (resulttool) available to manage QA test results and reporting, here is the QA process. First release to implement this process was 2.7_M3_RC1.

This QA process consists below:

Test Trigger
Test Planning 
Test Execution
Test Result Store
Test Monitoring & Reporting
Release Decision

Test Trigger

Each QA team will subscribe to QA notification email (request through Richard Purdie). The list of notifications is maintained on config.json in the yocto- autobuilder-helper which has a branch per release.

Test Planning

Once received the QA notification email, each QA team (eg. intel, windriver, etc) will perform planning on what extra tests they plan to run and when they'll send the results back, then reply to the QA notification email with the planned extra tests and estimate of execution time to QA stakeholders (eg. Richard Purdie, Stephen Jolley), yocto mailing list and the lead QA team. Each QA team can refer to OEQA for automated and manual test cases for their planning. The lead QA team no longer need to setup Testopia and wiki page.

Test Execution

Each QA team will execute the planned extra tests. To make sure test result from the test execution could fully integrated to the new QA tooling (resulttool for test result management and reporting/regression), execute OEQA automated tests and OEQA manual tests through resulttool (refer https://wiki.yoctoproject.org/wiki/Resulttool#manualexecution).

Test Result Store

Each QA team will store test result to the remote yocto-testresults-contrib git repository using resulttool (refer https://wiki.yoctoproject.org/wiki/Resulttool#store), then send the QA completion email (include new defects information) to both QA stakeholder and the lead QA team.

Test Result Repo

After testing is complete, lead QA team will upload test results and test report in yocto-testresults-contrib git repo (https://git.yoctoproject.org/cgit/cgit.cgi/yocto-testresults-contrib/log/?h=intel-yocto-testresults) and send QA release mail. This is a staging repo where test results and test report are kept for tests run by independent test teams.

To access test results, clone the repo https://git.yoctoproject.org/cgit/cgit.cgi/yocto-testresults-contrib/ and checkout QA branch "intel-yocto-testresults"

For tests run at Autobuilder, test results and reports can be find at https://autobuilder.yocto.io/pub/releases/ inside release version folder.

Test Monitoring & Reporting

QA stakeholder will monitor testing progress from remote yocto-testresults git repository using resulttool (refer https://wiki.yoctoproject.org/wiki/Resulttool#report). Once every QA team completed the test execution, the lead QA team will create QA test report and regression using resulttool, then store the report and regression into https://wiki.yoctoproject.org/wiki/Main_Page. Send email report to QA stakeholder and public yocto mailing list.

Release Decision

QA stakeholder will make the final decision for release. Once decision is taken to release a milestone, RE team will take test results from staging repo and put in https://git.yoctoproject.org/cgit/cgit.cgi/yocto-testresults/ git repo under same commit id as poky.

Current Release QA info

Tracking info for current release

Current Release QA Test Plans

Test Execution

Autobuilder

The AutoBuilder is Yocto Project's tool for non-manual test execution, it performs the following functions:

  • A scheduled nightly build and test execution that includes:
  • A service for on demand testing requests. (Partially working, feature in progress at request #9880)

Yocto Project QA heavily relies in the Autobuilder thanks to the aforementioned scheduled and on demand test execution features.

Image Testing

In order to execute tests in an image, it is necessary to boot it in either a virtual or a physical target.

Testing Images in Virtual Targets:

The execution in a virtual environment has a nice flow, documented in the Image Tests Enabling... section.

Testing Images in Physical Targets:

For executing tests in physical targets it would be required to:

  1. Boot the image in the target by following the building an image for hardware instructions.
  2. Run the image tests by following the image test or exporting tests instructions.

Setting up Targets with Devauto

Manual instructions for setting up the physical test targets appear in many parts of the Yocto Project documentation (i.e. here). It is easy to setup one target using those instructions but it becomes challenging for the cases where multiple targets have to be prepared or the case where it is required to serialize a changing setup over time, for one: testing on several images using the same target.

Devauto is the Python library and command line interface intended to manage the device automation assets that act upon the target's physical state. Refer to the Devauto documentation for more information about the hardware supported and the library and CLI functionality.

Creating and Adding New Tests

Tests for a given component can be automated in the AutoBuilder. With that purpose, follow the Adding Automated Tests to the Autobuilder Guide.

A list of tests that are automated can be seen here.

Reporting

Performance testing

Performance Test

QA Resources

Archive

You can find the previous QA work by release in the Yocto Project QA Archive.

Other Relevant Data

List of Automated Tests