QA: Difference between revisions

From Yocto Project
Jump to navigationJump to search
No edit summary
No edit summary
(82 intermediate revisions by 13 users not shown)
Line 1: Line 1:
===Milestone Test Report for Yocto===
See the http://yoctoproject.org/projects/testing-qa-and-autobuilder page on the website for more information on this project.


====Fullpass Test Report for M3-20100911 ====
= QA Docs =
[https://oldwiki.pokylinux.org/share/QA_docs/milestone%20test/M3/Fullpass%20Test%20Report%20for%20Yocto%20M3%2020100911%20Build.htm Fullpass Test Report for M3-20100911]
* Test plans
** [[Yocto 2.1 Overall Test Plan]]
** [[Toaster testing plan]]
** [[SDK Testing Plan]]
** [[Distro Testing Plan]]
** [[BSP Test Plan]]


==== Sanity Test Report for M3-20100911 ====
* Test Cases
[https://oldwiki.pokylinux.org/share/QA_docs/milestone%20test/M3/Yocto%20Sanity%20Test%20Report%20for%20M3-20100911.htm Sanity Test Report for M3-20100911]
** [[Yocto 2.1 Test Cases]]


==== Sanity Test Report for M3-20100909 ====
[https://oldwiki.pokylinux.org/share/QA_docs/milestone%20test/M3/Yocto%20Sanity%20Test%20Report%20for%20M3-20100909.htm Sanity Test Report for M3-20100911]


==== Partial Sanity Test Report for M3 20100905 ====
* Test Cycle
[https://oldwiki.pokylinux.org/share/QA_docs/milestone%20test/M3/Yocto%20Partial%20Sanity%20Test%20Report%20for%20M3%2020100905.htm Partial Test Report for M3 20100905]
** [[Testing Cycle]]


==== Fullpass Test for M2-20100731 ====
[https://git.pokylinux.org/share/QA_docs/milestone%20test/M2/Fullpass%20Test%20Report%20for%20M2%2020100731.htm Fullpass Test Report for M2-20100731]
==== M2-20100725 & Bug Update for M2-20100728 ====
I finished testing on QEMU/Netbook for M2-20100725 and verified some bugs with M2-20100728 build. Against M2-20100725 build, I reported 12 new bugs and verified 2 bugs as fixed. With the second build M2-20100728, I verified old critical bugs and reported another 2 new bugs. 2 old bugs are verified as fixed with this build.


==== M3 Test Execution Plan ====
* QA Automation
[https://oldwiki.pokylinux.org/share/QA_docs/Test%20Plan/M3-Test%20Plan.doc M3 fullpass test plan].
**[[AutoBuilder]]
**[[Distribution Support‎]]
**[[Oe-selftest]]
**[[Image tests]]
**[[Runtime]]
*** add ptest wiki
*** add piglit test wiki
*** add kernel test wiki
***[[LSB]]
***[[LSB Result]]
***[[LTP]]
***[[LTP result]]
***[[POSIX]]
***[[Posix result]]
***[[POSIX-results]]
***[[POSIX History Results]]
***[[LAVA server vs Yocto HW automation testing]]
**[[Automated package upgrade testing]]


==== M4 Test Execution Plan ====
*QA Tracking pages
[https://oldwiki.pokylinux.org/share/QA_docs/Test%20Plan/M4-Test_Plan.doc M4 fullpass test plan].
**[[2.1 QA Owned Features]]
**[[2.1 qa owned bugs]]
**[[2.1 qa owned features to verify]]
**[[Old resolved bugs and features]]


*Reporting
**[[Bug reporting and Information levels]]
**[[Testopia]]
**[http://errors.yoctoproject.org/Errors/ error report web]


*1.9 QA Tracking
** [[1.9 qa owned features]]
** [[1.9 qa owned BUGS]]


There are still 5 open bugs for M2, 4 of them are CRITICAL. And 23 open bugs are marked to M3. As a summary, QEMU targets (x86/arm/mips/ppc) can work without critical bugs. Toolchain has one critical bug. Meanwhile, emenlow/netbook have critical issue with/within X environment.
*1.8 QA Tracking pages
** [[1.8 QA Status]]
** [[1.8 qa run history]]
** [[1.8 qa queries]]
** [[1.8 qa owned features]]
** [[1.8 qa owned bugs by OWNER]]
** [[1.8 qa owned bugs by PRODUCT]]


*Performance testing
***[[Performance Test]]


'''Open Bugs for M2:
*QA Resources
* Wrong qemu binary is used if qemu-native isn't built  --  (Richard, is the issue going to be fixed in M2?)
***[[Rpm's Repository Setup for QA]]
** http://bugzilla.pokylinux.org/show_bug.cgi?id=123
* Mouse/Keyboard cannot work with poky-image-sato-live on netbook  --  (It should be fixed by Jeff, but mouse/kbd still not work because of bug 171)
** http://bugzilla.pokylinux.org/show_bug.cgi?id=141
* [emenlow] X not bootable of image poky-image-sato-live-emenlow  --  (This bug is with 0728 image)
** http://bugzilla.pokylinux.org/show_bug.cgi?id=159
* [toolchain] Cannot build LTP with ppc/arm toolchain
** http://bugzilla.pokylinux.org/show_bug.cgi?id=161
* [Netbook] Xserver version mismatch makes mouse/keyboard not work
** http://bugzilla.pokylinux.org/show_bug.cgi?id=171


'''Other Open Bugs reported, target for M3 or later:
= QA Old Test plans =
* It takes long time to boot qemuppc image at the first time  -- Enhance/Low
* Yocto 0.9
** http://bugzilla.pokylinux.org/show_bug.cgi?id=143
** Yocto 0.9 [[Overall Test Plan]]
* avahi-daemon fail to start with qemumips sdk image  -- Minor/Medium (Target on M3)
** Yocto 0.9 [[Milestone Test Report]]
** http://bugzilla.pokylinux.org/show_bug.cgi?id=144
* nm-applet segfault with qemux86 sato/sdk image  -- Normal/Medium (Not our focus)
** http://bugzilla.pokylinux.org/show_bug.cgi?id=145
* keystroke combination ctrl-alt-del not work on netbook  -- Normal/Low
** http://bugzilla.pokylinux.org/show_bug.cgi?id=146
* [Netbook] pcmanfm no response after I click File Manager icon  -- Major/Medium (Target on M3)
** http://bugzilla.pokylinux.org/show_bug.cgi?id=148
* [All] There is no way to exit matchbox-keyboard in X  -- Normal/Medium (Target on M3)
** http://bugzilla.pokylinux.org/show_bug.cgi?id=149
* [Netbook] Music Player cannot play WAV format file  -- Major/Undecided
** http://bugzilla.pokylinux.org/show_bug.cgi?id=151
* [emenlow] command reboot cannot make system reboot entirely  -- Medium/Normal (Target on M3)
** http://bugzilla.pokylinux.org/show_bug.cgi?id=160
* [LTP] LTP default test suite cannot fully pass  -- Normal/Low (Target on M4)
** http://bugzilla.pokylinux.org/show_bug.cgi?id=169
* [POSIX] POSIX default test suite cannot fully pass  -- Normal/Low (Target on M4)
** http://bugzilla.pokylinux.org/show_bug.cgi?id=172


'''Verified Fixed Issues:
* Yocto 1.0
* qemuppc images cannot boot up
**[[Yocto 1.0 Overall Test Plan]]
** http://bugzilla.pokylinux.org/show_bug.cgi?id=101
**[[Yocto 1.0.2 Test Report]]
* Some icons broken with qemumips images
** http://bugzilla.pokylinux.org/show_bug.cgi?id=106
* qemumips poky-image-sato/sdk fail to boot  -- (Fixed in 0728 build)
** http://bugzilla.pokylinux.org/show_bug.cgi?id=138
* emenlow live image cannot boot/install  -- (Fixed in 0728 build)
** http://bugzilla.pokylinux.org/show_bug.cgi?id=142


'''Existing Old Issues:
* Yocto 1.1
* No mouse action response in X window with qemumips images  -- Major/High (Target on M3)
**[[Yocto 1.1 Overall Test Plan]]
** http://bugzilla.pokylinux.org/show_bug.cgi?id=99
**[[Yocto 1.1 Test Execution Plan]]
* qemuarm and qemuppc images cannot shutdown entirely  -- Normal/High (Target on M3)
**[[Yocto 1.1 Weekly Test Report]]
** http://bugzilla.pokylinux.org/show_bug.cgi?id=100
**[[Yocto 1.1 Milestone Test Report]]
**[[Yocto 1.1.1 Test Report]]


==== M1 Test Report 2010/7/10 ====
* Yocto 1.2
'''Summary & Issue List'''
**[[Yocto 1.2 Overall Test Plan]]
**[[Yocto 1.2 Test Execution Plan]]
**[[Yocto 1.2 Weekly Test Report]]
**[[Yocto 1.2 Milestone Test Report]]


I finished sanitytest for M1 20100710 build. There is 2 new issues found in this testing. qemuppc boot up issue is fixed but qemu will report OOM error when booting qemuppc images. There is no mouse action response after X startup with qemumips images. qemuppc and qemuarm cannot shutdown entirely.
* Yocto 1.3
There are totally 4 open bugs in this testing.
**[[Yocto 1.3 Overall Test Plan]]
**[[Yocto 1.3 Test Execution Plan]]
**[[Yocto 1.3 Weekly Test Report]]
**[[Yocto 1.3 Milestone Test Report]]


BTW, This time I still build qemu-native from WR's branch (git.pokylinux.org:poky-wr +yocto-kernel-clean).
* Yocto 1.4
**[[Yocto 1.4 Overall Test Plan]]
**[[Yocto 1.4 Milestone Test Report]]


'''Issue List:'''
* Yocto 1.5
**[[Yocto 1.5 Overall Test Plan]]
**[https://wiki.yoctoproject.org/wiki/1.5_qa_run_history Yocto 1.5 Weekly, Full Pass and Milestone Test Report]
**[[1.5 QA Status]]


* No mouse action response in X window with qemumips images
* Yocto 1.6
** http://bugzilla.pokylinux.org/show_bug.cgi?id=99
**[[Yocto 1.6 Overall Test Plan]]
* qemuarm and qemuppc images cannot shutdown entirely
**[https://wiki.yoctoproject.org/wiki/1.6_qa_run_history Yocto 1.6 Weekly, Full Pass and Milestone Test Report]
** http://bugzilla.pokylinux.org/show_bug.cgi?id=100
**[[1.6 QA Status]] - live status report for the current weekly build
* OOM error when booting qemuppc images
**[[1.6 qa owned bugs]]
** http://bugzilla.pokylinux.org/show_bug.cgi?id=102
* Some icons broken with qemumips images
** http://bugzilla.pokylinux.org/show_bug.cgi?id=106


'''Test Result Matrix:'''
*Yocto 1.7
**[[1.7 QA Status]]
**[[1.7 qa owned bugs]]
**[[1.7 qa run history]]
**[[1.7 meta-intel qa run history]]


''SanityTest Result''
* Yocto 1.8
{| border="1"
**[[1.8 QA Status]]
|-
**[[1.8 qa run history]]
! Machine
**[[1.8 qa queries]]
! Image
**[[1.8 qa owned features]]
! Boot
**[[1.8 qa owned bugs by OWNER]]
! Network(ping)
**[[1.8 qa owned bugs by PRODUCT]]
! X startup
! SSH Service
! Shutdown
! Comments
|-
! rowspan="2"| qemux86
! poky-image-sato
| style="color:green" | PASS
| style="color:green" | PASS
| style="color:green"| PASS
| style="color:green"| PASS
| style="color:green" | PASS
|
|-
! poky-image-sdk
| style="color:green" | PASS
| style="color:green" | PASS
| style="color:green"| PASS
| style="color:green"| PASS
| style="color:green" | PASS
|
|-
! rowspan="2"| qemuxarm
! poky-image-sato
| style="color:green" | PASS
| style="color:green" | PASS
| style="color:green"| PASS
| style="color:green"| PASS
| style="color:red" | FAIL
|
|-
! poky-image-sdk
| style="color:green" | PASS
| style="color:green" | PASS
| style="color:green"| PASS
| style="color:green"| PASS
| style="color:red" | FAIL
|
|-
! rowspan="2"| qemumips
! poky-image-sato
| style="color:green" | PASS
| style="color:green" | PASS
| style="color:green" | PASS
| style="color:green"| PASS
| style="color:green"| PASS
| style="color:red" | No mouse action response, some icons broken
|-
! poky-image-sdk
| style="color:green" | PASS
| style="color:green" | PASS
| style="color:green" | PASS
| style="color:green"| PASS
| style="color:green"| PASS
| style="color:red" | No mouse action response, some icons broken
|-
! rowspan="2"| qemuppc
! poky-image-sato
| style="color:green" | PASS
| style="color:green" | PASS
| N/A
| style="color:green" | PASS
| style="color:red"| FAIL
|
|-
! poky-image-sdk
| style="color:green" | PASS
| style="color:green" | PASS
| N/A
| style="color:green" | PASS
| style="color:red"| FAIL
|
|}


'''Detail Test Result'''


Test Result can be found here [https://git.pokylinux.org/share/QA_docs/milestone%20test/M1-Testing-20100710.xls Milestone Test Result for 20100710]
* [[Yocto Bug Trend]]
* Compliance Test Result
** [[LTP result]]
** [[Posix result]]
** [[LSB Result]]
* [[ADT Testing]]
* [[Regression Test]]
* [[Performance Test]]
* [[Distro Test]]
* [[Distribution Support]]
* [[QA BKM sharing]]
* [[Testopia]]


'''Testing Env'''
: Host: Ubuntu 10.04 LTS
: Kernel: 2.6.32-21-generic-pae
: Poky gitinfo(used to build qemu-native): aa416b6133bd55b5ba4d2fb6f5faef274ba511a0 (git.pokylinux.org:poky-wr +yocto-kernel-clean)
: Build Image Download URL: http://autobuilder.pokylinux.org/milestone/20100710-1/




==== M1 Test Report 2010/7/8 ====
'''Summary & Issue List'''


I finished sanitytest for M1 20100708 build. There are 3 issues found in this testing.
[[Category:QA]]
The major issue is that there is no cursor in qemuarm and qemumips images. I also tried qemuppc because there is some fixing in latest tree, but qemuppc still fails to startup.
5 issues found in last time testing are fixed. Qemuarm can boot up now. Qemuarm and qemumips can start X now.
 
BTW, This time I still build qemu-native from WR's branch (git.pokylinux.org:poky-wr +yocto-kernel-clean).
 
'''Issue List:'''
 
* No cursor in X window with qemuarm and qemumips images
** http://bugzilla.pokylinux.org/show_bug.cgi?id=99
* qemuarm images cannot shutdown entirely
** http://bugzilla.pokylinux.org/show_bug.cgi?id=100
* qemuppc images cannot boot up
** http://bugzilla.pokylinux.org/show_bug.cgi?id=101
 
'''Test Result Matrix:'''
 
''SanityTest Result''
{| border="1"
|-
! Machine
! Image
! Boot
! Network(ping)
! X startup
! SSH Service
! Shutdown
! Comments
|-
! rowspan="2"| qemux86
! poky-image-sato
| style="color:green" | PASS
| style="color:green" | PASS
| style="color:green"| PASS
| style="color:green"| PASS
| style="color:green" | PASS
|
|-
! poky-image-sdk
| style="color:green" | PASS
| style="color:green" | PASS
| style="color:green"| PASS
| style="color:green"| PASS
| style="color:green" | PASS
|
|-
! rowspan="2"| qemuxarm
! poky-image-sato
| style="color:green" | PASS
| style="color:green" | PASS
| style="color:green"| PASS
| style="color:green"| PASS
| style="color:red" | FAIL
| style="color:red" | No cursor after X startup
|-
! poky-image-sdk
| style="color:green" | PASS
| style="color:green" | PASS
| style="color:green"| PASS
| style="color:green"| PASS
| style="color:red" | FAIL
| style="color:red" | No cursor after X startup
|-
! rowspan="2"| qemumips
! poky-image-sato
| style="color:green" | PASS
| style="color:green" | PASS
| style="color:green" | PASS
| style="color:green"| PASS
| style="color:green"| PASS
| style="color:red" | No cursor after X startup
|-
! poky-image-sdk
| style="color:green" | PASS
| style="color:green" | PASS
| style="color:green" | PASS
| style="color:green"| PASS
| style="color:green"| PASS
| style="color:red" | No cursor after X startup
|}
 
'''Detail Test Result'''
 
Test Result can be found here [https://git.pokylinux.org/share/QA_docs/milestone%20test/M1-Testing-20100708.xls Milestone Test Result for 20100708]
 
'''Testing Env'''
: Host: Ubuntu 10.04 LTS
: Kernel: 2.6.32-21-generic-pae
: Poky gitinfo(used to build qemu-native): aa416b6133bd55b5ba4d2fb6f5faef274ba511a0 (git.pokylinux.org:poky-wr +yocto-kernel-clean)
: Build Image Download URL: http://autobuilder.pokylinux.org/milestone/20100708-1/
 
==== M1 Test Report 2010/6/29 ====
'''Summary & Issue List'''
 
There are 8 target images be tested, only 3 ones can boot up, while SDK image for qemumips is not built out. Testing for qemuarm and qemuppc is blocked. Network in qemux86 image is not enabled. X failure is still within SDK image. And there is no cursor within qemux86 poky-image-sato. Detail issue list as below.
 
For BAT testing, LTP and POSIX testing are performed against qemux86 poky-image-sdk and qemumips poky-image-sato. Compared with last time test result, LTP result for qemux86 is same, while POSIX result has one more failure.
 
BTW, I build qemu-native by myself from poky WR’s branch (git.pokylinux.org:poky-wr). These qemu startup scripts, like “runqemu”, also come from the branch.
 
'''Issue List:'''
# No network in qemux86 image, (sato/sdk). It should be caused by lack of e1000 module built in WR kernel.
# No cursor after X startup in qemux86 poky-image-sato image.
# Qemux86 poky-image-sdk image fails to start X. (Known issue as Saul pointed)
# Fail to boot up qemuarm images, (sato/sdk). It seems no root partition when booting. See attachment “qemuarm-poky-image.log” for full log when booting QEMU. [https://git.pokylinux.org/share/QA_docs/milestone%20test/log-20100629/qemuarm-poky-image.log qemuarm log]
# Fail to boot up qemuppc images, (sato/sdk). qemu reports “could not load PPC PREP bios ‘powerpc_rom.bin’”. See attachment “qemuppc-poky-image.log” for full log. [https://git.pokylinux.org/share/QA_docs/milestone%20test/log-20100629/qemuppc-poky-image.log qemuppc log]
# By default, script “runqemu” does not start X for qemuarm, qemuppc and qemumips. I find there is no wnidow popup even I remove the X disable option, "-nographic" for qemumips.
 
'''Test Result Matrix:'''
 
''SanityTest Result''
{| border="1"
|-
! Machine
! Image
! Boot
! Network(ping)
! X startup
! SSH Service
! Shutdown
! Comments
|-
! rowspan="2"| qemux86
! poky-image-sato
| style="color:green" | PASS
| style="color:red" | FAIL
| style="color:green"| PASS
| N/T
| style="color:green" | PASS
| style="color:red" | No cursor after X startup with show-cursor option
|-
! poky-image-sdk
| style="color:green" | PASS
| style="color:red" | FAIL
| style="color:red"| FAIL
| N/T
| style="color:green" | PASS
|
|-
! rowspan="2"| qemuxarm
! poky-image-sato
| style="color:red" | FAIL
| style="color:red" | BLOCK
| style="color:red"| BLOCK
| style="color:red"| BLOCK
| style="color:red"| BLOCK
|
|-
! poky-image-sdk
| style="color:red" | FAIL
| style="color:red" | BLOCK
| style="color:red"| BLOCK
| style="color:red"| BLOCK
| style="color:red"| BLOCK
|
|-
! rowspan="2"| qemumips
! poky-image-sato
| style="color:green" | PASS
| style="color:green" | PASS
| N/A
| style="color:green"| PASS
| style="color:green"| PASS
| style="color:red" | option "-nographic" is set in script "poky-qemu-internal" for qemumips. There is no window popup even I remove the option.
|-
! poky-image-sdk
| N/A
| N/A
| N/A
| N/A
| N/A
| style="color:red" | No SDK images for qemumips
|-
! rowspan="2"| qemuppc
! poky-image-sato
| style="color:red" | FAIL
| style="color:red" | BLOCK
| style="color:red" | BLOCK
| style="color:red" | BLOCK
| style="color:red" | BLOCK
|
|-
! poky-image-sdk
| style="color:red" | FAIL
| style="color:red" | BLOCK
| style="color:red" | BLOCK
| style="color:red" | BLOCK
| style="color:red" | BLOCK
|
|}
 
''BAT Result:''
{| border="1"
|-
! Machine
! Image
! LTP(FAIL/TOTAL)
! POSIX(FAIL/UNSUPPORT/TOTAL)
|-
! qemux86
! poky-image-sdk
| style="color:red" | 46/1120
| style="color:red" | 51/99/1644
|-
! qemumips
! poky-image-sato
| style="color:red" | 58/1116
| N/A
|}
 
'''Detail Test Result'''
 
Test Result can be found here [https://git.pokylinux.org/share/QA_docs/milestone%20test/Nightly-M1-Testing-20100629.xls Milestone Test Result for 20100629]
 
'''Testing Env'''
: Host: Ubuntu 10.04 LTS
: Kernel: 2.6.32-21-generic-pae
: Poky gitinfo(used to build qemu-native): 5280cfe8c9cd9c9a880683186c77c24d380978ed (git.pokylinux.org:poky-wr)
: Build Image Download URL: Except for SDK images for qemuppc and qemuarm (from http://autobuilder.pokylinux.org/nightly/20100629-1/), all the other images come from http://autobuilder.pokylinux.org/nightly/20100628-1/.
 
=== Enable Automation Test in Poky ===
Currently we have implemented two automated sanity test cases in Poky. User can follow below steps to enable the automation test:
 
* Make sure your working tree has latest test code. The first sanity test implementation is checked in poky master tree as commit, '''fd1d661b391dfad1edaac937a17c1165f635031a.
* Make sure your machine has package "expect" installed. The package is necessary for testing. We will add a recipe for it later. Currently, you need install it by yourself. On Ubuntu host, you can use apt-get to install expect:
<pre>
$ apt-get install expect
</pre>
* Edit /etc/sudoers to allow the user run sudo without prompted for a password. For example, you can do like following for user "tester":
<pre>
$ cat /etc/sudoers
# /etc/sudoers
#
# This file MUST be edited with the 'visudo' command as root.
#
# See the man page for details on how to write a sudoers file.
#
....
....
....
 
%tester ALL=(ALL) NOPASSWD: NOPASSWD: ALL
</pre>
* Typically, you need a X environment to start QEMU. You can use your desktop's DISPLAY. Or, alternatively, you can install vncserver, and startup a X environment in localhost:x.0. (x can be 1, 2, 3.....n)
* You need set "IMAGETEST" to "qemu" in build/conf/local.conf
<pre>
$ cat local.conf
.....
# Set IMAGETEST to qemu if you want to build testcases and start
# testing in qemu after do_rootfs.
IMAGETEST = "qemu"
.....
</pre>
* Then you can run "bitbake poky-image-xxx" (xxx can be minimal, sato, sdk). After the kernel/image are generated, you need run "bitbake poky-image-xxx -c qemuimagetest". It will trigger automation test and you will get the result after a while. :)
 
===Full Pass Specification===
 
==== M2 Test Execution Plan ====
Since there are still some unclear requirements for Yocto project, currently it’s hard to define test plan to cover all future milestone/weekly testing. According to the suggestion from KangKang, I defined a test execution plan for M2 testing, which only covers tests for M2 requirement.
 
 
The fullpass test plan is here, [https://git.pokylinux.org/share/QA_docs/Test%20Plan/TC_test_plan-2010-08-11.doc M2 fullpass test plan]. For each target, like qemux86, qemuarm, netbook, e-Menlow, etc, I define separate test plan for them, with sub-tests chosen from the fullpass test plan.
 
I setup a testlink page to track test case and test plan status, http://otc-qa.sh.intel.com/testlink_yocto/. You can use username/passwd: guest/guest, to browser test case and test plan.

Revision as of 20:25, 20 May 2016

See the http://yoctoproject.org/projects/testing-qa-and-autobuilder page on the website for more information on this project.

QA Docs



QA Old Test plans