QA: Difference between revisions

From Yocto Project
Jump to navigationJump to search
(Created page with '==QA== ===Milestone Test Report for Yocto=== ====Fullpass Test Report for M3-20100911 ==== [https://oldwiki.pokylinux.org/share/QA_docs/milestone%20test/M3/Fullpass%20Test%20Rep…')
 
 
(182 intermediate revisions by 18 users not shown)
Line 1: Line 1:
==QA==
== QA Process==
===Milestone Test Report for Yocto===
=== Intro ===
The Yocto Project primarily runs QA through its autobuilder. Patches are pre-tested using the 'a-quick' builds on the project autobuilder https://autobuilder.yoctoproject.org/typhoon/#/console usually in master-next or ross/mut branches. XXX-next branches are also used for testing stable branch changes. We aim to only merge patches which pass these tests. A more comprehensive set of tests is available in 'a-full' builds which may also be used. Test results for all builds are saved in json format and can be handled by the 'resultool' script. Build ouput/results can be seen at https://autobuilder.yocto.io/pub/non-release/.


====Fullpass Test Report for M3-20100911 ====
For milestone and main project releases, an 'a-full' build is run as normal but the options to save build output, send a QA email and generate a release are selected. This then triggers a more extensive QA process where other teams can contribute to and extend the autobuilder results as follows:
[https://oldwiki.pokylinux.org/share/QA_docs/milestone%20test/M3/Fullpass%20Test%20Report%20for%20Yocto%20M3%2020100911%20Build.htm Fullpass Test Report for M3-20100911]


==== Sanity Test Report for M3-20100911 ====
Test Trigger
[https://oldwiki.pokylinux.org/share/QA_docs/milestone%20test/M3/Yocto%20Sanity%20Test%20Report%20for%20M3-20100911.htm Sanity Test Report for M3-20100911]
Test Planning
Test Execution
Test Result Store
Test Monitoring & Reporting
Release Decision


==== Sanity Test Report for M3-20100909 ====
=== Test Trigger ===
[https://oldwiki.pokylinux.org/share/QA_docs/milestone%20test/M3/Yocto%20Sanity%20Test%20Report%20for%20M3-20100909.htm Sanity Test Report for M3-20100911]
Each QA team can subscribe to QA notification email which is sent from the autobuilder (request through Richard Purdie) and also to the yocto mailing list. The list of notifications is maintained on config.json in the yocto-autobuilder-helper git repository which has a branch per release.


==== Partial Sanity Test Report for M3 20100905 ====
=== Test Planning ===
[https://oldwiki.pokylinux.org/share/QA_docs/milestone%20test/M3/Yocto%20Partial%20Sanity%20Test%20Report%20for%20M3%2020100905.htm Partial Test Report for M3 20100905]
Once received the QA notification email, each QA team (eg. intel, windriver, etc) will perform planning on what extra tests they plan to run and when they'll send the results back, then reply to the QA notification email with the planned extra tests and estimate of execution time to QA stakeholders (eg. Richard Purdie, Programme Management (Stephen Jolley) and Release Engineering), yocto mailing list and the lead QA team. Each QA team can refer to OEQA for automated and manual test cases for their planning.


==== Fullpass Test for M2-20100731 ====
=== Test Execution ===
[https://git.pokylinux.org/share/QA_docs/milestone%20test/M2/Fullpass%20Test%20Report%20for%20M2%2020100731.htm Fullpass Test Report for M2-20100731]
Each QA team will execute the planned extra tests. To make sure test result from the test execution could fully integrated to the new QA tooling (resulttool for test result management and reporting/regression), execute OEQA automated tests and OEQA manual tests through resulttool (refer https://wiki.yoctoproject.org/wiki/Resulttool#manualexecution).  
==== M2-20100725 & Bug Update for M2-20100728 ====
I finished testing on QEMU/Netbook for M2-20100725 and verified some bugs with M2-20100728 build. Against M2-20100725 build, I reported 12 new bugs and verified 2 bugs as fixed. With the second build M2-20100728, I verified old critical bugs and reported another 2 new bugs. 2 old bugs are verified as fixed with this build.


==== M3 Test Execution Plan ====
=== Test Result Store ===
[https://oldwiki.pokylinux.org/share/QA_docs/Test%20Plan/M3-Test%20Plan.doc M3 fullpass test plan].
Each QA team will store test result to the remote yocto-testresults-contrib git repository using resulttool (refer https://wiki.yoctoproject.org/wiki/Resulttool#store), then send the QA completion email (include new defects information) to both QA stakeholder and the lead QA team.


==== M4 Test Execution Plan ====
=== Test Result Repo ===
[https://oldwiki.pokylinux.org/share/QA_docs/Test%20Plan/M4-Test_Plan.doc M4 fullpass test plan].


After testing is complete, lead QA team will upload test results and test report in yocto-testresults-contrib git repo (https://git.yoctoproject.org/cgit/cgit.cgi/yocto-testresults-contrib/log/?h=intel-yocto-testresults) and send QA release mail. This is a staging repo where test results and test report are kept for tests run by independent test teams.


To access test results, clone the repo https://git.yoctoproject.org/cgit/cgit.cgi/yocto-testresults-contrib/ and checkout QA branch "intel-yocto-testresults"


There are still 5 open bugs for M2, 4 of them are CRITICAL. And 23 open bugs are marked to M3. As a summary, QEMU targets (x86/arm/mips/ppc) can work without critical bugs. Toolchain has one critical bug. Meanwhile, emenlow/netbook have critical issue with/within X environment.
For tests run at Autobuilder, test results and reports for release builds can be find at https://autobuilder.yocto.io/pub/releases/ inside release version folder.


=== Test Monitoring & Reporting ===
QA stakeholder will monitor testing progress from remote yocto-testresults git repository using resulttool (refer https://wiki.yoctoproject.org/wiki/Resulttool#report).  Once every QA team completed the test execution, the lead QA team will create QA test report and regression using resulttool, then store the report and regression into https://wiki.yoctoproject.org/wiki/Main_Page. Send email report to QA stakeholder and public yocto mailing list.


'''Open Bugs for M2:
=== Release Decision ===
* Wrong qemu binary is used if qemu-native isn't built  --  (Richard, is the issue going to be fixed in M2?)
QA stakeholder will make the final decision for release.
** http://bugzilla.pokylinux.org/show_bug.cgi?id=123
Once decision is taken to release a milestone, RE team will take test results from staging repo and put in https://git.yoctoproject.org/cgit/cgit.cgi/yocto-testresults/ git repo under same commit id as poky.
* Mouse/Keyboard cannot work with poky-image-sato-live on netbook  --  (It should be fixed by Jeff, but mouse/kbd still not work because of bug 171)
** http://bugzilla.pokylinux.org/show_bug.cgi?id=141
* [emenlow] X not bootable of image poky-image-sato-live-emenlow  --  (This bug is with 0728 image)
** http://bugzilla.pokylinux.org/show_bug.cgi?id=159
* [toolchain] Cannot build LTP with ppc/arm toolchain
** http://bugzilla.pokylinux.org/show_bug.cgi?id=161
* [Netbook] Xserver version mismatch makes mouse/keyboard not work
** http://bugzilla.pokylinux.org/show_bug.cgi?id=171


'''Other Open Bugs reported, target for M3 or later:
==Current Release QA info==
* It takes long time to boot qemuppc image at the first time  -- Enhance/Low
** http://bugzilla.pokylinux.org/show_bug.cgi?id=143
* avahi-daemon fail to start with qemumips sdk image  -- Minor/Medium (Target on M3)
** http://bugzilla.pokylinux.org/show_bug.cgi?id=144
* nm-applet segfault with qemux86 sato/sdk image  -- Normal/Medium (Not our focus)
** http://bugzilla.pokylinux.org/show_bug.cgi?id=145
* keystroke combination ctrl-alt-del not work on netbook  -- Normal/Low
** http://bugzilla.pokylinux.org/show_bug.cgi?id=146
* [Netbook] pcmanfm no response after I click File Manager icon  -- Major/Medium (Target on M3)
** http://bugzilla.pokylinux.org/show_bug.cgi?id=148
* [All] There is no way to exit matchbox-keyboard in X  -- Normal/Medium (Target on M3)
** http://bugzilla.pokylinux.org/show_bug.cgi?id=149
* [Netbook] Music Player cannot play WAV format file  -- Major/Undecided
** http://bugzilla.pokylinux.org/show_bug.cgi?id=151
* [emenlow] command reboot cannot make system reboot entirely  -- Medium/Normal (Target on M3)
** http://bugzilla.pokylinux.org/show_bug.cgi?id=160
* [LTP] LTP default test suite cannot fully pass  -- Normal/Low (Target on M4)
** http://bugzilla.pokylinux.org/show_bug.cgi?id=169
* [POSIX] POSIX default test suite cannot fully pass  -- Normal/Low (Target on M4)
** http://bugzilla.pokylinux.org/show_bug.cgi?id=172


'''Verified Fixed Issues:
Tracking info for current release
* qemuppc images cannot boot up
** http://bugzilla.pokylinux.org/show_bug.cgi?id=101
* Some icons broken with qemumips images
** http://bugzilla.pokylinux.org/show_bug.cgi?id=106
* qemumips poky-image-sato/sdk fail to boot  -- (Fixed in 0728 build)
** http://bugzilla.pokylinux.org/show_bug.cgi?id=138
* emenlow live image cannot boot/install  -- (Fixed in 0728 build)
** http://bugzilla.pokylinux.org/show_bug.cgi?id=142


'''Existing Old Issues:
* For latest release info, go to https://autobuilder.yocto.io/pub/releases/
* No mouse action response in X window with qemumips images  -- Major/High (Target on M3)
* For latest QA results, go to https://git.yoctoproject.org/cgit/cgit.cgi/yocto-testresults-contrib/log/?h=intel-yocto-testresults
** http://bugzilla.pokylinux.org/show_bug.cgi?id=99
* qemuarm and qemuppc images cannot shutdown entirely  -- Normal/High (Target on M3)
** http://bugzilla.pokylinux.org/show_bug.cgi?id=100


==== M1 Test Report 2010/7/10 ====
==Current Release QA Test Plans==
'''Summary & Issue List'''


I finished sanitytest for M1 20100710 build. There is 2 new issues found in this testing. qemuppc boot up issue is fixed but qemu will report OOM error when booting qemuppc images. There is no mouse action response after X startup with qemumips images. qemuppc and qemuarm cannot shutdown entirely.
* [[Yocto Project 3.4_Release Test Plan]]
There are totally 4 open bugs in this testing.
* [[BSP Test Plan]]
* [https://lists.yoctoproject.org/pipermail/yocto-perf/  Performance Archives ]


BTW, This time I still build qemu-native from WR's branch (git.pokylinux.org:poky-wr +yocto-kernel-clean).
==Test Execution==
===Autobuilder===


'''Issue List:'''
The [[AutoBuilder]] is Yocto Project's tool for non-manual test execution, it performs the following functions:


* No mouse action response in X window with qemumips images
* A scheduled nightly build and test execution that includes:
** http://bugzilla.pokylinux.org/show_bug.cgi?id=99
** That each image created executes the corresponding set of [[image tests|image/run-time tests]]
* qemuarm and qemuppc images cannot shutdown entirely
** Specific Autobuilder tasks for running [[oe-selftest|build-time testing]]
** http://bugzilla.pokylinux.org/show_bug.cgi?id=100
* A service for on demand testing requests.
* OOM error when booting qemuppc images
** http://bugzilla.pokylinux.org/show_bug.cgi?id=102
* Some icons broken with qemumips images
** http://bugzilla.pokylinux.org/show_bug.cgi?id=106


'''Test Result Matrix:'''
===Image Testing===
In order to execute tests in an image, it is necessary to boot it in either a virtual or a physical target.


''SanityTest Result''
'''Testing Images in Virtual Targets''':
{| border="1"
|-
! Machine
! Image
! Boot
! Network(ping)
! X startup
! SSH Service
! Shutdown
! Comments
|-
! rowspan="2"| qemux86
! poky-image-sato
| style="color:green" | PASS
| style="color:green" | PASS
| style="color:green"| PASS
| style="color:green"| PASS
| style="color:green" | PASS
|
|-
! poky-image-sdk
| style="color:green" | PASS
| style="color:green" | PASS
| style="color:green"| PASS
| style="color:green"| PASS
| style="color:green" | PASS
|
|-
! rowspan="2"| qemuxarm
! poky-image-sato
| style="color:green" | PASS
| style="color:green" | PASS
| style="color:green"| PASS
| style="color:green"| PASS
| style="color:red" | FAIL
|
|-
! poky-image-sdk
| style="color:green" | PASS
| style="color:green" | PASS
| style="color:green"| PASS
| style="color:green"| PASS
| style="color:red" | FAIL
|
|-
! rowspan="2"| qemumips
! poky-image-sato
| style="color:green" | PASS
| style="color:green" | PASS
| style="color:green" | PASS
| style="color:green"| PASS
| style="color:green"| PASS
| style="color:red" | No mouse action response, some icons broken
|-
! poky-image-sdk
| style="color:green" | PASS
| style="color:green" | PASS
| style="color:green" | PASS
| style="color:green"| PASS
| style="color:green"| PASS
| style="color:red" | No mouse action response, some icons broken
|-
! rowspan="2"| qemuppc
! poky-image-sato
| style="color:green" | PASS
| style="color:green" | PASS
| N/A
| style="color:green" | PASS
| style="color:red"| FAIL
|
|-
! poky-image-sdk
| style="color:green" | PASS
| style="color:green" | PASS
| N/A
| style="color:green" | PASS
| style="color:red"| FAIL
|
|}


'''Detail Test Result'''
The execution in a virtual environment has a nice flow, documented in the [[Image_tests#Enabling_and_running_the_tests|Image Tests Enabling...]] section.


Test Result can be found here [https://git.pokylinux.org/share/QA_docs/milestone%20test/M1-Testing-20100710.xls Milestone Test Result for 20100710]
'''Testing Images in Physical Targets''':


'''Testing Env'''
For executing tests in physical targets it would be required to:
: Host: Ubuntu 10.04 LTS
: Kernel: 2.6.32-21-generic-pae
: Poky gitinfo(used to build qemu-native): aa416b6133bd55b5ba4d2fb6f5faef274ba511a0 (git.pokylinux.org:poky-wr +yocto-kernel-clean)
: Build Image Download URL: http://autobuilder.pokylinux.org/milestone/20100710-1/


# Boot the image in the target by following the [http://www.yoctoproject.org/docs/2.2/mega-manual/mega-manual.html#building-an-image-for-hardware building an image for hardware] instructions.
# Run the image tests by following the [https://wiki.yoctoproject.org/wiki/BSP_Test_Plan#Automated_Tests image test] or [http://www.yoctoproject.org/docs/2.2/mega-manual/mega-manual.html#exporting-tests exporting tests] instructions.


==== M1 Test Report 2010/7/8 ====
===Creating and Adding New Tests===
'''Summary & Issue List'''


I finished sanitytest for M1 20100708 build. There are 3 issues found in this testing.
Tests for a given component can be automated in the AutoBuilder. With that purpose, follow the [[AutoBuilder/Adding Automated Tests|Adding Automated Tests]] to the Autobuilder Guide.
The major issue is that there is no cursor in qemuarm and qemumips images. I also tried qemuppc because there is some fixing in latest tree, but qemuppc still fails to startup.
5 issues found in last time testing are fixed. Qemuarm can boot up now. Qemuarm and qemumips can start X now.


BTW, This time I still build qemu-native from WR's branch (git.pokylinux.org:poky-wr +yocto-kernel-clean).
A list of tests that are automated [[QA#List_of_Automated_Tests|can be seen here]].


'''Issue List:'''
==Reporting==
*[[Bug reporting and Information levels]]
*[[https://wiki.yoctoproject.org/wiki/Resulttool#report The Test Reporting Tool]]
*[http://errors.yoctoproject.org/Errors/ error report web]
*[[https://git.yoctoproject.org/cgit/cgit.cgi/yocto-testresults-contrib/ Test results report]]


* No cursor in X window with qemuarm and qemumips images
==Performance testing==
** http://bugzilla.pokylinux.org/show_bug.cgi?id=99
[[Performance Test]]
* qemuarm images cannot shutdown entirely
** http://bugzilla.pokylinux.org/show_bug.cgi?id=100
* qemuppc images cannot boot up
** http://bugzilla.pokylinux.org/show_bug.cgi?id=101


'''Test Result Matrix:'''
==QA Resources==
*[[QA Master Test Plan]]
*[[Resulttool]]
*[[https://git.yoctoproject.org/cgit/cgit.cgi/yocto-testresults |qa results Git Repository]]
*[[https://git.yoctoproject.org/cgit/cgit.cgi/yocto-testresults-contrib |qa results staging Git Repository]]


''SanityTest Result''
= Archive =
{| border="1"
|-
! Machine
! Image
! Boot
! Network(ping)
! X startup
! SSH Service
! Shutdown
! Comments
|-
! rowspan="2"| qemux86
! poky-image-sato
| style="color:green" | PASS
| style="color:green" | PASS
| style="color:green"| PASS
| style="color:green"| PASS
| style="color:green" | PASS
|
|-
! poky-image-sdk
| style="color:green" | PASS
| style="color:green" | PASS
| style="color:green"| PASS
| style="color:green"| PASS
| style="color:green" | PASS
|
|-
! rowspan="2"| qemuxarm
! poky-image-sato
| style="color:green" | PASS
| style="color:green" | PASS
| style="color:green"| PASS
| style="color:green"| PASS
| style="color:red" | FAIL
| style="color:red" | No cursor after X startup
|-
! poky-image-sdk
| style="color:green" | PASS
| style="color:green" | PASS
| style="color:green"| PASS
| style="color:green"| PASS
| style="color:red" | FAIL
| style="color:red" | No cursor after X startup
|-
! rowspan="2"| qemumips
! poky-image-sato
| style="color:green" | PASS
| style="color:green" | PASS
| style="color:green" | PASS
| style="color:green"| PASS
| style="color:green"| PASS
| style="color:red" | No cursor after X startup
|-
! poky-image-sdk
| style="color:green" | PASS
| style="color:green" | PASS
| style="color:green" | PASS
| style="color:green"| PASS
| style="color:green"| PASS
| style="color:red" | No cursor after X startup
|}


'''Detail Test Result'''
You can find the previous QA work by release in the [[QA/Archive|Yocto Project QA Archive]].


Test Result can be found here [https://git.pokylinux.org/share/QA_docs/milestone%20test/M1-Testing-20100708.xls Milestone Test Result for 20100708]
= Other Relevant Data=
* [[Yocto Bug Trend]]
* Compliance Test Result
** [[LTP result]]
** [[Posix result]]
** [[LSB Result]]
* [[Distribution Support]]
* [[LAVA server vs Yocto HW automation testing]]
** Note: The LAVA framework usage stopped in favor of testing in the AutoBuilder in early 2016.


'''Testing Env'''
==List of Automated Tests==
: Host: Ubuntu 10.04 LTS
* [[Ptest]]
: Kernel: 2.6.32-21-generic-pae
* add piglit test wiki
: Poky gitinfo(used to build qemu-native): aa416b6133bd55b5ba4d2fb6f5faef274ba511a0 (git.pokylinux.org:poky-wr +yocto-kernel-clean)
* [[Kernel]]
: Build Image Download URL: http://autobuilder.pokylinux.org/milestone/20100708-1/
*[[LSB]]
 
*[[LSB Result]]
==== M1 Test Report 2010/6/29 ====
*[[LTP]]
'''Summary & Issue List'''
*[[LTP result]]
 
*[[POSIX]]
There are 8 target images be tested, only 3 ones can boot up, while SDK image for qemumips is not built out. Testing for qemuarm and qemuppc is blocked. Network in qemux86 image is not enabled. X failure is still within SDK image. And there is no cursor within qemux86 poky-image-sato. Detail issue list as below.
*[[Posix result]]
 
*[[POSIX-results]]
For BAT testing, LTP and POSIX testing are performed against qemux86 poky-image-sdk and qemumips poky-image-sato. Compared with last time test result, LTP result for qemux86 is same, while POSIX result has one more failure.
*[[POSIX History Results]]
 
*[[Automated package upgrade testing]]
BTW, I build qemu-native by myself from poky WR’s branch (git.pokylinux.org:poky-wr). These qemu startup scripts, like “runqemu”, also come from the branch.
 
'''Issue List:'''
# No network in qemux86 image, (sato/sdk). It should be caused by lack of e1000 module built in WR kernel.
# No cursor after X startup in qemux86 poky-image-sato image.
# Qemux86 poky-image-sdk image fails to start X. (Known issue as Saul pointed)
# Fail to boot up qemuarm images, (sato/sdk). It seems no root partition when booting. See attachment “qemuarm-poky-image.log” for full log when booting QEMU. [https://git.pokylinux.org/share/QA_docs/milestone%20test/log-20100629/qemuarm-poky-image.log qemuarm log]
# Fail to boot up qemuppc images, (sato/sdk). qemu reports “could not load PPC PREP bios ‘powerpc_rom.bin’”. See attachment “qemuppc-poky-image.log” for full log. [https://git.pokylinux.org/share/QA_docs/milestone%20test/log-20100629/qemuppc-poky-image.log qemuppc log]
# By default, script “runqemu” does not start X for qemuarm, qemuppc and qemumips. I find there is no wnidow popup even I remove the X disable option, "-nographic" for qemumips.
 
'''Test Result Matrix:'''
 
''SanityTest Result''
{| border="1"
|-
! Machine
! Image
! Boot
! Network(ping)
! X startup
! SSH Service
! Shutdown
! Comments
|-
! rowspan="2"| qemux86
! poky-image-sato
| style="color:green" | PASS
| style="color:red" | FAIL
| style="color:green"| PASS
| N/T
| style="color:green" | PASS
| style="color:red" | No cursor after X startup with show-cursor option
|-
! poky-image-sdk
| style="color:green" | PASS
| style="color:red" | FAIL
| style="color:red"| FAIL
| N/T
| style="color:green" | PASS
|
|-
! rowspan="2"| qemuxarm
! poky-image-sato
| style="color:red" | FAIL
| style="color:red" | BLOCK
| style="color:red"| BLOCK
| style="color:red"| BLOCK
| style="color:red"| BLOCK
|
|-
! poky-image-sdk
| style="color:red" | FAIL
| style="color:red" | BLOCK
| style="color:red"| BLOCK
| style="color:red"| BLOCK
| style="color:red"| BLOCK
|
|-
! rowspan="2"| qemumips
! poky-image-sato
| style="color:green" | PASS
| style="color:green" | PASS
| N/A
| style="color:green"| PASS
| style="color:green"| PASS
| style="color:red" | option "-nographic" is set in script "poky-qemu-internal" for qemumips. There is no window popup even I remove the option.
|-
! poky-image-sdk
| N/A
| N/A
| N/A
| N/A
| N/A
| style="color:red" | No SDK images for qemumips
|-
! rowspan="2"| qemuppc
! poky-image-sato
| style="color:red" | FAIL
| style="color:red" | BLOCK
| style="color:red" | BLOCK
| style="color:red" | BLOCK
| style="color:red" | BLOCK
|
|-
! poky-image-sdk
| style="color:red" | FAIL
| style="color:red" | BLOCK
| style="color:red" | BLOCK
| style="color:red" | BLOCK
| style="color:red" | BLOCK
|
|}
 
''BAT Result:''
{| border="1"
|-
! Machine
! Image
! LTP(FAIL/TOTAL)
! POSIX(FAIL/UNSUPPORT/TOTAL)
|-
! qemux86
! poky-image-sdk
| style="color:red" | 46/1120
| style="color:red" | 51/99/1644
|-
! qemumips
! poky-image-sato
| style="color:red" | 58/1116
| N/A
|}
 
'''Detail Test Result'''
 
Test Result can be found here [https://git.pokylinux.org/share/QA_docs/milestone%20test/Nightly-M1-Testing-20100629.xls Milestone Test Result for 20100629]
 
'''Testing Env'''
: Host: Ubuntu 10.04 LTS
: Kernel: 2.6.32-21-generic-pae
: Poky gitinfo(used to build qemu-native): 5280cfe8c9cd9c9a880683186c77c24d380978ed (git.pokylinux.org:poky-wr)
: Build Image Download URL: Except for SDK images for qemuppc and qemuarm (from http://autobuilder.pokylinux.org/nightly/20100629-1/), all the other images come from http://autobuilder.pokylinux.org/nightly/20100628-1/.
 
=== Enable Automation Test in Poky ===
Currently we have implemented two automated sanity test cases in Poky. User can follow below steps to enable the automation test:
 
* Make sure your working tree has latest test code. The first sanity test implementation is checked in poky master tree as commit, '''fd1d661b391dfad1edaac937a17c1165f635031a.
* Make sure your machine has package "expect" installed. The package is necessary for testing. We will add a recipe for it later. Currently, you need install it by yourself. On Ubuntu host, you can use apt-get to install expect:
<pre>
$ apt-get install expect
</pre>
* Edit /etc/sudoers to allow the user run sudo without prompted for a password. For example, you can do like following for user "tester":
<pre>
$ cat /etc/sudoers
# /etc/sudoers
#
# This file MUST be edited with the 'visudo' command as root.
#
# See the man page for details on how to write a sudoers file.
#
....
....
....
 
%tester ALL=(ALL) NOPASSWD: NOPASSWD: ALL
</pre>
* Typically, you need a X environment to start QEMU. You can use your desktop's DISPLAY. Or, alternatively, you can install vncserver, and startup a X environment in localhost:x.0. (x can be 1, 2, 3.....n)
* You need set "IMAGETEST" to "qemu" in build/conf/local.conf
<pre>
$ cat local.conf
.....
# Set IMAGETEST to qemu if you want to build testcases and start
# testing in qemu after do_rootfs.
IMAGETEST = "qemu"
.....
</pre>
* Then you can run "bitbake poky-image-xxx" (xxx can be minimal, sato, sdk). After the kernel/image are generated, you need run "bitbake poky-image-xxx -c qemuimagetest". It will trigger automation test and you will get the result after a while. :)
 
===Full Pass Specification===
 
==== M2 Test Execution Plan ====
Since there are still some unclear requirements for Yocto project, currently it’s hard to define test plan to cover all future milestone/weekly testing. According to the suggestion from KangKang, I defined a test execution plan for M2 testing, which only covers tests for M2 requirement.
 
 
The fullpass test plan is here, [https://git.pokylinux.org/share/QA_docs/Test%20Plan/TC_test_plan-2010-08-11.doc M2 fullpass test plan]. For each target, like qemux86, qemuarm, netbook, e-Menlow, etc, I define separate test plan for them, with sub-tests chosen from the fullpass test plan.
 
I setup a testlink page to track test case and test plan status, http://otc-qa.sh.intel.com/testlink_yocto/. You can use username/passwd: guest/guest, to browser test case and test plan.

Latest revision as of 05:23, 8 September 2021

QA Process

Intro

The Yocto Project primarily runs QA through its autobuilder. Patches are pre-tested using the 'a-quick' builds on the project autobuilder https://autobuilder.yoctoproject.org/typhoon/#/console usually in master-next or ross/mut branches. XXX-next branches are also used for testing stable branch changes. We aim to only merge patches which pass these tests. A more comprehensive set of tests is available in 'a-full' builds which may also be used. Test results for all builds are saved in json format and can be handled by the 'resultool' script. Build ouput/results can be seen at https://autobuilder.yocto.io/pub/non-release/.

For milestone and main project releases, an 'a-full' build is run as normal but the options to save build output, send a QA email and generate a release are selected. This then triggers a more extensive QA process where other teams can contribute to and extend the autobuilder results as follows: 

Test Trigger
Test Planning 
Test Execution
Test Result Store
Test Monitoring & Reporting
Release Decision

Test Trigger

Each QA team can subscribe to QA notification email which is sent from the autobuilder (request through Richard Purdie) and also to the yocto mailing list. The list of notifications is maintained on config.json in the yocto-autobuilder-helper git repository which has a branch per release.

Test Planning

Once received the QA notification email, each QA team (eg. intel, windriver, etc) will perform planning on what extra tests they plan to run and when they'll send the results back, then reply to the QA notification email with the planned extra tests and estimate of execution time to QA stakeholders (eg. Richard Purdie, Programme Management (Stephen Jolley) and Release Engineering), yocto mailing list and the lead QA team. Each QA team can refer to OEQA for automated and manual test cases for their planning.

Test Execution

Each QA team will execute the planned extra tests. To make sure test result from the test execution could fully integrated to the new QA tooling (resulttool for test result management and reporting/regression), execute OEQA automated tests and OEQA manual tests through resulttool (refer https://wiki.yoctoproject.org/wiki/Resulttool#manualexecution).

Test Result Store

Each QA team will store test result to the remote yocto-testresults-contrib git repository using resulttool (refer https://wiki.yoctoproject.org/wiki/Resulttool#store), then send the QA completion email (include new defects information) to both QA stakeholder and the lead QA team.

Test Result Repo

After testing is complete, lead QA team will upload test results and test report in yocto-testresults-contrib git repo (https://git.yoctoproject.org/cgit/cgit.cgi/yocto-testresults-contrib/log/?h=intel-yocto-testresults) and send QA release mail. This is a staging repo where test results and test report are kept for tests run by independent test teams.

To access test results, clone the repo https://git.yoctoproject.org/cgit/cgit.cgi/yocto-testresults-contrib/ and checkout QA branch "intel-yocto-testresults"

For tests run at Autobuilder, test results and reports for release builds can be find at https://autobuilder.yocto.io/pub/releases/ inside release version folder.

Test Monitoring & Reporting

QA stakeholder will monitor testing progress from remote yocto-testresults git repository using resulttool (refer https://wiki.yoctoproject.org/wiki/Resulttool#report). Once every QA team completed the test execution, the lead QA team will create QA test report and regression using resulttool, then store the report and regression into https://wiki.yoctoproject.org/wiki/Main_Page. Send email report to QA stakeholder and public yocto mailing list.

Release Decision

QA stakeholder will make the final decision for release. Once decision is taken to release a milestone, RE team will take test results from staging repo and put in https://git.yoctoproject.org/cgit/cgit.cgi/yocto-testresults/ git repo under same commit id as poky.

Current Release QA info

Tracking info for current release

Current Release QA Test Plans

Test Execution

Autobuilder

The AutoBuilder is Yocto Project's tool for non-manual test execution, it performs the following functions:

  • A scheduled nightly build and test execution that includes:
  • A service for on demand testing requests.

Image Testing

In order to execute tests in an image, it is necessary to boot it in either a virtual or a physical target.

Testing Images in Virtual Targets:

The execution in a virtual environment has a nice flow, documented in the Image Tests Enabling... section.

Testing Images in Physical Targets:

For executing tests in physical targets it would be required to:

  1. Boot the image in the target by following the building an image for hardware instructions.
  2. Run the image tests by following the image test or exporting tests instructions.

Creating and Adding New Tests

Tests for a given component can be automated in the AutoBuilder. With that purpose, follow the Adding Automated Tests to the Autobuilder Guide.

A list of tests that are automated can be seen here.

Reporting

Performance testing

Performance Test

QA Resources

Archive

You can find the previous QA work by release in the Yocto Project QA Archive.

Other Relevant Data

List of Automated Tests

Retrieved from "https://wiki.yoctoproject.org/wiki/index.php?title=QA&oldid=84917"