QA: Difference between revisions

From Yocto Project
Jump to navigationJump to search
(62 intermediate revisions by 3 users not shown)
Line 1: Line 1:
See the http://yoctoproject.org/projects/testing-qa-and-autobuilder page on the website for more information on this project.
==Current Release QA Trackers==


= QA Docs =
*Test Plan
* Test plans
**[[Yocto Project 2.3_Release Test Plan]]
** [[Yocto 2.1 Overall Test Plan]]
** [[Toaster testing plan]]
** [[SDK Testing Plan]]
** [[Distro Testing Plan]]
** [[BSP Test Plan]]


* Test Cases
*Live Status
** [[Yocto 2.1 Test Cases]]
**[[2.3 QA Status]]


*QA Execution History
**[[Yocto_Project_2.3_Release_Test_Plan#Execution_History | 2.3 qa run history]]


* Test Cycle
*Bugs that need to be implemented by QA Team
** [[Testing Cycle]]
**[[2.3_QA_OWNED_BUGS | 2.3 QA Assigned Bugs]]


*Bugs that need to be verified by QA Team
**[[2.3_QA_Bugs_To_Verify]]
<!--
*Features to verify
**[[2.3 qa_owned features to verify]]


* QA Automation
*Features to implement
**[[AutoBuilder]]
**[[2.3 qa owned features]]
**[[Distribution Support‎]]
-->
**[[Oe-selftest]]
**[[Image tests]]
**[[Runtime]]
*** add ptest wiki
*** add piglit test wiki
*** add kernel test wiki
***[[LSB]]
***[[LSB Result]]
***[[LTP]]
***[[LTP result]]
***[[POSIX]]
***[[Posix result]]
***[[POSIX-results]]
***[[POSIX History Results]]
***[[LAVA server vs Yocto HW automation testing]]
**[[Automated package upgrade testing]]


*Toaster
*Old Bugs that need to be Verified
**[[Toaster Backend Testing]]
**[[Old resolved bugs and features]]
**[[Toaster Frontend Testing]]


*Reporting
*2.3 Test Run Templates
**[[Bug reporting and Information levels]]
**[[Yocto_Project_2.3_Release_Test_Plan#Test_Items | Test Run Templates]]
**[[Testopia]]
**[http://errors.yoctoproject.org/Errors/ error report web]


* Bug/Feature Validation
==Current Release QA Test Plans==
** Add Bug Validation wiki
* [[QA Master Test Plan]]
** add Feature Validation wiki
* [[Yocto Project 2.3_Release Test Plan]]
* [[2.3 QA Status]]
* [https://wiki.yoctoproject.org/charts/perf_milestone_GDC/performance_test.html  Performance Charts ]
* [[Toaster testing plan]]
* [[Extensible SDK Test Plan (eSDK)]]
* [[Distro Testing Plan]]
* [[BSP Test Plan]]


*Bug Tracking
==Current Release QA Test Cases==
** [[1.9 qa owned features]]
* [[Yocto 2.3 Test Cases]]
** [[1.9 qa owned BUGS]]


==Test Execution==
===Autobuilder===


** [[1.8 QA Status]]
The [[AutoBuilder]] is Yocto Project's tool for non-manual test execution, it performs the following functions:
** [[1.8 qa run history]]
** [[1.8 qa queries]]
** [[1.8 qa owned features]]
** [[1.8 qa owned bugs by OWNER]]
** [[1.8 qa owned bugs by PRODUCT]]


*Performance testing
* A scheduled nightly build and test execution that includes:
***[[Performance Test]]
** That each image created executes the corresponding set of [[image tests|image/run-time tests]]
** Specific Autobuilder tasks for running [[oe-selftest|build-time testing]]
* A service for on demand testing requests. (Partially working, feature in progress at request [https://bugzilla.yoctoproject.org/show_bug.cgi?id=9080 #9880])


= QA Old Test plans =
Yocto Project QA heavily relies in the Autobuilder thanks to the aforementioned scheduled and on demand test execution features.
* Yocto 0.9
===Image Testing===
** Yocto 0.9 [[Overall Test Plan]]
In order to execute tests in an image, it is necessary to boot it in either a virtual or a physical target.  
** Yocto 0.9 [[Milestone Test Report]]


* Yocto 1.0
'''Testing Images in Virtual Targets''':
**[[Yocto 1.0 Overall Test Plan]]
**[[Yocto 1.0.2 Test Report]]


* Yocto 1.1
The execution in a virtual environment has a nice flow, documented in the [[Image_tests#Enabling_and_running_the_tests|Image Tests Enabling...]] section.  
**[[Yocto 1.1 Overall Test Plan]]
**[[Yocto 1.1 Test Execution Plan]]
**[[Yocto 1.1 Weekly Test Report]]
**[[Yocto 1.1 Milestone Test Report]]
**[[Yocto 1.1.1 Test Report]]


* Yocto 1.2
'''Testing Images in Physical Targets''':
**[[Yocto 1.2 Overall Test Plan]]
**[[Yocto 1.2 Test Execution Plan]]
**[[Yocto 1.2 Weekly Test Report]]
**[[Yocto 1.2 Milestone Test Report]]


* Yocto 1.3
For executing tests in physical targets it would be required to:
**[[Yocto 1.3 Overall Test Plan]]
**[[Yocto 1.3 Test Execution Plan]]
**[[Yocto 1.3 Weekly Test Report]]
**[[Yocto 1.3 Milestone Test Report]]


* Yocto 1.4
# Boot the image in the target by following the [http://www.yoctoproject.org/docs/2.2/mega-manual/mega-manual.html#building-an-image-for-hardware building an image for hardware] instructions.
**[[Yocto 1.4 Overall Test Plan]]
# Run the image tests by following the [http://www.yoctoproject.org/docs/2.2/mega-manual/mega-manual.html#exporting-tests exporting tests] instructions.
**[[Yocto 1.4 Milestone Test Report]]


* Yocto 1.5
===Setting up Targets with Devauto===
**[[Yocto 1.5 Overall Test Plan]]
**[https://wiki.yoctoproject.org/wiki/1.5_qa_run_history Yocto 1.5 Weekly, Full Pass and Milestone Test Report]
**[[1.5 QA Status]]


* Yocto 1.6
Manual instructions for setting up the physical test targets appear in many parts of the Yocto Project documentation (i.e. [http://www.yoctoproject.org/docs/2.2/mega-manual/mega-manual.html#building-an-image-for-hardware here]). It is easy to setup one target using those instructions but it becomes challenging for the cases where multiple targets have to be prepared or the case where it is required to serialize a changing setup over time, for one: testing on several images using the same target.
**[[Yocto 1.6 Overall Test Plan]]
**[https://wiki.yoctoproject.org/wiki/1.6_qa_run_history Yocto 1.6 Weekly, Full Pass and Milestone Test Report]
**[[1.6 QA Status]] - live status report for the current weekly build
**[[1.6 qa owned bugs]]


*Yocto 1.7
[[Devauto]] is the Python library and command line interface intended to manage the device automation assets that act upon the target's physical state. Refer to the [[Devauto]] documentation for more information about the hardware supported and the library and CLI functionality.
**[[1.7 QA Status]]
**[[1.7 qa owned bugs]]
**[[1.7 qa run history]]
**[[1.7 meta-intel qa run history]]


* Yocto 1.8
===Creating and Adding New Tests===
**[[1.8 QA Status]]
**[[1.8 qa run history]]
**[[1.8 qa queries]]
**[[1.8 qa owned features]]
**[[1.8 qa owned bugs by OWNER]]
**[[1.8 qa owned bugs by PRODUCT]]


Tests for a given component can be automated in the AutoBuilder. With that purpose, follow the [[AutoBuilder/Adding Automated Tests|Adding Automated Tests]] to the Autobuilder Guide.


A list of tests that are automated [[QA#List_of_Automated_Tests|can be seen here]].
==Reporting==
*[[Bug reporting and Information levels]]
*[[Testopia]], the test manager
*[[QA/Test_Reporting_Tool|The Test Reporting Tool]]
*[http://errors.yoctoproject.org/Errors/ error report web]
*[[QA REPORT TEMPLATE | Wiki of QA Report Template]]
==Performance testing==
[[Performance Test]]
==QA Resources==
*[[Rpm's Repository Setup for QA]]
*[[Testopia]]
*[[Testing Cycle]]
*[[qa-tools|qa-tools Git Repository]]
= Archive =
You can find the previous QA work by release in the [[QA/Archive|Yocto Project QA Archive]].
= Other Relevant Data=
* [[Yocto Bug Trend]]
* [[Yocto Bug Trend]]
* Compliance Test Result
* Compliance Test Result
Line 134: Line 111:
* [[Distribution Support]]
* [[Distribution Support]]
* [[QA BKM sharing]]
* [[QA BKM sharing]]
* [[Testopia]]
* [[LAVA server vs Yocto HW automation testing]]
 
** Note: The LAVA framework usage stopped in favor of testing in the AutoBuilder in early 2016.
 


==List of Automated Tests==
*[[Distribution Support‎]]
* add ptest wiki
* add piglit test wiki
* add kernel test wiki
*[[LSB]]
*[[LSB Result]]
*[[LTP]]
*[[LTP result]]
*[[POSIX]]
*[[Posix result]]
*[[POSIX-results]]
*[[POSIX History Results]]
*[[Automated package upgrade testing]]


[[Category:QA]]
[[Category:QA]]

Revision as of 16:44, 15 May 2017

Current Release QA Trackers

Current Release QA Test Plans

Current Release QA Test Cases

Test Execution

Autobuilder

The AutoBuilder is Yocto Project's tool for non-manual test execution, it performs the following functions:

  • A scheduled nightly build and test execution that includes:
  • A service for on demand testing requests. (Partially working, feature in progress at request #9880)

Yocto Project QA heavily relies in the Autobuilder thanks to the aforementioned scheduled and on demand test execution features.

Image Testing

In order to execute tests in an image, it is necessary to boot it in either a virtual or a physical target.

Testing Images in Virtual Targets:

The execution in a virtual environment has a nice flow, documented in the Image Tests Enabling... section.

Testing Images in Physical Targets:

For executing tests in physical targets it would be required to:

  1. Boot the image in the target by following the building an image for hardware instructions.
  2. Run the image tests by following the exporting tests instructions.

Setting up Targets with Devauto

Manual instructions for setting up the physical test targets appear in many parts of the Yocto Project documentation (i.e. here). It is easy to setup one target using those instructions but it becomes challenging for the cases where multiple targets have to be prepared or the case where it is required to serialize a changing setup over time, for one: testing on several images using the same target.

Devauto is the Python library and command line interface intended to manage the device automation assets that act upon the target's physical state. Refer to the Devauto documentation for more information about the hardware supported and the library and CLI functionality.

Creating and Adding New Tests

Tests for a given component can be automated in the AutoBuilder. With that purpose, follow the Adding Automated Tests to the Autobuilder Guide.

A list of tests that are automated can be seen here.

Reporting

Performance testing

Performance Test

QA Resources

Archive

You can find the previous QA work by release in the Yocto Project QA Archive.

Other Relevant Data

List of Automated Tests