QA: Difference between revisions
No edit summary |
No edit summary |
||
| Line 1: | Line 1: | ||
* [[Overall Test Plan]] | |||
* [[Milestone Test Report]] | |||
===Milestone Test Report for Yocto=== | ===Milestone Test Report for Yocto=== | ||
Revision as of 22:17, 22 October 2010
Milestone Test Report for Yocto
Fullpass Test Report for M3-20100911
Fullpass Test Report for M3-20100911
Sanity Test Report for M3-20100911
Sanity Test Report for M3-20100911
Sanity Test Report for M3-20100909
Sanity Test Report for M3-20100911
Partial Sanity Test Report for M3 20100905
Partial Test Report for M3 20100905
Fullpass Test for M2-20100731
Fullpass Test Report for M2-20100731
M2-20100725 & Bug Update for M2-20100728
I finished testing on QEMU/Netbook for M2-20100725 and verified some bugs with M2-20100728 build. Against M2-20100725 build, I reported 12 new bugs and verified 2 bugs as fixed. With the second build M2-20100728, I verified old critical bugs and reported another 2 new bugs. 2 old bugs are verified as fixed with this build.
M3 Test Execution Plan
M4 Test Execution Plan
There are still 5 open bugs for M2, 4 of them are CRITICAL. And 23 open bugs are marked to M3. As a summary, QEMU targets (x86/arm/mips/ppc) can work without critical bugs. Toolchain has one critical bug. Meanwhile, emenlow/netbook have critical issue with/within X environment.
Open Bugs for M2:
- Wrong qemu binary is used if qemu-native isn't built -- (Richard, is the issue going to be fixed in M2?)
- Mouse/Keyboard cannot work with poky-image-sato-live on netbook -- (It should be fixed by Jeff, but mouse/kbd still not work because of bug 171)
- [emenlow] X not bootable of image poky-image-sato-live-emenlow -- (This bug is with 0728 image)
- [toolchain] Cannot build LTP with ppc/arm toolchain
- [Netbook] Xserver version mismatch makes mouse/keyboard not work
Other Open Bugs reported, target for M3 or later:
- It takes long time to boot qemuppc image at the first time -- Enhance/Low
- avahi-daemon fail to start with qemumips sdk image -- Minor/Medium (Target on M3)
- nm-applet segfault with qemux86 sato/sdk image -- Normal/Medium (Not our focus)
- keystroke combination ctrl-alt-del not work on netbook -- Normal/Low
- [Netbook] pcmanfm no response after I click File Manager icon -- Major/Medium (Target on M3)
- [All] There is no way to exit matchbox-keyboard in X -- Normal/Medium (Target on M3)
- [Netbook] Music Player cannot play WAV format file -- Major/Undecided
- [emenlow] command reboot cannot make system reboot entirely -- Medium/Normal (Target on M3)
- [LTP] LTP default test suite cannot fully pass -- Normal/Low (Target on M4)
- [POSIX] POSIX default test suite cannot fully pass -- Normal/Low (Target on M4)
Verified Fixed Issues:
- qemuppc images cannot boot up
- Some icons broken with qemumips images
- qemumips poky-image-sato/sdk fail to boot -- (Fixed in 0728 build)
- emenlow live image cannot boot/install -- (Fixed in 0728 build)
Existing Old Issues:
- No mouse action response in X window with qemumips images -- Major/High (Target on M3)
- qemuarm and qemuppc images cannot shutdown entirely -- Normal/High (Target on M3)
M1 Test Report 2010/7/10
Summary & Issue List
I finished sanitytest for M1 20100710 build. There is 2 new issues found in this testing. qemuppc boot up issue is fixed but qemu will report OOM error when booting qemuppc images. There is no mouse action response after X startup with qemumips images. qemuppc and qemuarm cannot shutdown entirely. There are totally 4 open bugs in this testing.
BTW, This time I still build qemu-native from WR's branch (git.pokylinux.org:poky-wr +yocto-kernel-clean).
Issue List:
- No mouse action response in X window with qemumips images
- qemuarm and qemuppc images cannot shutdown entirely
- OOM error when booting qemuppc images
- Some icons broken with qemumips images
Test Result Matrix:
SanityTest Result
| Machine | Image | Boot | Network(ping) | X startup | SSH Service | Shutdown | Comments |
|---|---|---|---|---|---|---|---|
| qemux86 | poky-image-sato | PASS | PASS | PASS | PASS | PASS | |
| poky-image-sdk | PASS | PASS | PASS | PASS | PASS | ||
| qemuxarm | poky-image-sato | PASS | PASS | PASS | PASS | FAIL | |
| poky-image-sdk | PASS | PASS | PASS | PASS | FAIL | ||
| qemumips | poky-image-sato | PASS | PASS | PASS | PASS | PASS | No mouse action response, some icons broken |
| poky-image-sdk | PASS | PASS | PASS | PASS | PASS | No mouse action response, some icons broken | |
| qemuppc | poky-image-sato | PASS | PASS | N/A | PASS | FAIL | |
| poky-image-sdk | PASS | PASS | N/A | PASS | FAIL |
Detail Test Result
Test Result can be found here Milestone Test Result for 20100710
Testing Env
- Host: Ubuntu 10.04 LTS
- Kernel: 2.6.32-21-generic-pae
- Poky gitinfo(used to build qemu-native): aa416b6133bd55b5ba4d2fb6f5faef274ba511a0 (git.pokylinux.org:poky-wr +yocto-kernel-clean)
- Build Image Download URL: http://autobuilder.pokylinux.org/milestone/20100710-1/
M1 Test Report 2010/7/8
Summary & Issue List
I finished sanitytest for M1 20100708 build. There are 3 issues found in this testing. The major issue is that there is no cursor in qemuarm and qemumips images. I also tried qemuppc because there is some fixing in latest tree, but qemuppc still fails to startup. 5 issues found in last time testing are fixed. Qemuarm can boot up now. Qemuarm and qemumips can start X now.
BTW, This time I still build qemu-native from WR's branch (git.pokylinux.org:poky-wr +yocto-kernel-clean).
Issue List:
- No cursor in X window with qemuarm and qemumips images
- qemuarm images cannot shutdown entirely
- qemuppc images cannot boot up
Test Result Matrix:
SanityTest Result
| Machine | Image | Boot | Network(ping) | X startup | SSH Service | Shutdown | Comments |
|---|---|---|---|---|---|---|---|
| qemux86 | poky-image-sato | PASS | PASS | PASS | PASS | PASS | |
| poky-image-sdk | PASS | PASS | PASS | PASS | PASS | ||
| qemuxarm | poky-image-sato | PASS | PASS | PASS | PASS | FAIL | No cursor after X startup |
| poky-image-sdk | PASS | PASS | PASS | PASS | FAIL | No cursor after X startup | |
| qemumips | poky-image-sato | PASS | PASS | PASS | PASS | PASS | No cursor after X startup |
| poky-image-sdk | PASS | PASS | PASS | PASS | PASS | No cursor after X startup |
Detail Test Result
Test Result can be found here Milestone Test Result for 20100708
Testing Env
- Host: Ubuntu 10.04 LTS
- Kernel: 2.6.32-21-generic-pae
- Poky gitinfo(used to build qemu-native): aa416b6133bd55b5ba4d2fb6f5faef274ba511a0 (git.pokylinux.org:poky-wr +yocto-kernel-clean)
- Build Image Download URL: http://autobuilder.pokylinux.org/milestone/20100708-1/
M1 Test Report 2010/6/29
Summary & Issue List
There are 8 target images be tested, only 3 ones can boot up, while SDK image for qemumips is not built out. Testing for qemuarm and qemuppc is blocked. Network in qemux86 image is not enabled. X failure is still within SDK image. And there is no cursor within qemux86 poky-image-sato. Detail issue list as below.
For BAT testing, LTP and POSIX testing are performed against qemux86 poky-image-sdk and qemumips poky-image-sato. Compared with last time test result, LTP result for qemux86 is same, while POSIX result has one more failure.
BTW, I build qemu-native by myself from poky WR’s branch (git.pokylinux.org:poky-wr). These qemu startup scripts, like “runqemu”, also come from the branch.
Issue List:
- No network in qemux86 image, (sato/sdk). It should be caused by lack of e1000 module built in WR kernel.
- No cursor after X startup in qemux86 poky-image-sato image.
- Qemux86 poky-image-sdk image fails to start X. (Known issue as Saul pointed)
- Fail to boot up qemuarm images, (sato/sdk). It seems no root partition when booting. See attachment “qemuarm-poky-image.log” for full log when booting QEMU. qemuarm log
- Fail to boot up qemuppc images, (sato/sdk). qemu reports “could not load PPC PREP bios ‘powerpc_rom.bin’”. See attachment “qemuppc-poky-image.log” for full log. qemuppc log
- By default, script “runqemu” does not start X for qemuarm, qemuppc and qemumips. I find there is no wnidow popup even I remove the X disable option, "-nographic" for qemumips.
Test Result Matrix:
SanityTest Result
| Machine | Image | Boot | Network(ping) | X startup | SSH Service | Shutdown | Comments |
|---|---|---|---|---|---|---|---|
| qemux86 | poky-image-sato | PASS | FAIL | PASS | N/T | PASS | No cursor after X startup with show-cursor option |
| poky-image-sdk | PASS | FAIL | FAIL | N/T | PASS | ||
| qemuxarm | poky-image-sato | FAIL | BLOCK | BLOCK | BLOCK | BLOCK | |
| poky-image-sdk | FAIL | BLOCK | BLOCK | BLOCK | BLOCK | ||
| qemumips | poky-image-sato | PASS | PASS | N/A | PASS | PASS | option "-nographic" is set in script "poky-qemu-internal" for qemumips. There is no window popup even I remove the option. |
| poky-image-sdk | N/A | N/A | N/A | N/A | N/A | No SDK images for qemumips | |
| qemuppc | poky-image-sato | FAIL | BLOCK | BLOCK | BLOCK | BLOCK | |
| poky-image-sdk | FAIL | BLOCK | BLOCK | BLOCK | BLOCK |
BAT Result:
| Machine | Image | LTP(FAIL/TOTAL) | POSIX(FAIL/UNSUPPORT/TOTAL) |
|---|---|---|---|
| qemux86 | poky-image-sdk | 46/1120 | 51/99/1644 |
| qemumips | poky-image-sato | 58/1116 | N/A |
Detail Test Result
Test Result can be found here Milestone Test Result for 20100629
Testing Env
- Host: Ubuntu 10.04 LTS
- Kernel: 2.6.32-21-generic-pae
- Poky gitinfo(used to build qemu-native): 5280cfe8c9cd9c9a880683186c77c24d380978ed (git.pokylinux.org:poky-wr)
- Build Image Download URL: Except for SDK images for qemuppc and qemuarm (from http://autobuilder.pokylinux.org/nightly/20100629-1/), all the other images come from http://autobuilder.pokylinux.org/nightly/20100628-1/.
Enable Automation Test in Poky
Currently we have implemented two automated sanity test cases in Poky. User can follow below steps to enable the automation test:
- Make sure your working tree has latest test code. The first sanity test implementation is checked in poky master tree as commit, fd1d661b391dfad1edaac937a17c1165f635031a.
- Make sure your machine has package "expect" installed. The package is necessary for testing. We will add a recipe for it later. Currently, you need install it by yourself. On Ubuntu host, you can use apt-get to install expect:
$ apt-get install expect
- Edit /etc/sudoers to allow the user run sudo without prompted for a password. For example, you can do like following for user "tester":
$ cat /etc/sudoers # /etc/sudoers # # This file MUST be edited with the 'visudo' command as root. # # See the man page for details on how to write a sudoers file. # .... .... .... %tester ALL=(ALL) NOPASSWD: NOPASSWD: ALL
- Typically, you need a X environment to start QEMU. You can use your desktop's DISPLAY. Or, alternatively, you can install vncserver, and startup a X environment in localhost:x.0. (x can be 1, 2, 3.....n)
- You need set "IMAGETEST" to "qemu" in build/conf/local.conf
$ cat local.conf ..... # Set IMAGETEST to qemu if you want to build testcases and start # testing in qemu after do_rootfs. IMAGETEST = "qemu" .....
- Then you can run "bitbake poky-image-xxx" (xxx can be minimal, sato, sdk). After the kernel/image are generated, you need run "bitbake poky-image-xxx -c qemuimagetest". It will trigger automation test and you will get the result after a while. :)
Full Pass Specification
M2 Test Execution Plan
Since there are still some unclear requirements for Yocto project, currently it’s hard to define test plan to cover all future milestone/weekly testing. According to the suggestion from KangKang, I defined a test execution plan for M2 testing, which only covers tests for M2 requirement.
The fullpass test plan is here, M2 fullpass test plan. For each target, like qemux86, qemuarm, netbook, e-Menlow, etc, I define separate test plan for them, with sub-tests chosen from the fullpass test plan.
I setup a testlink page to track test case and test plan status, http://otc-qa.sh.intel.com/testlink_yocto/. You can use username/passwd: guest/guest, to browser test case and test plan.
