Wiki Test Report Format
From Yocto Project
Jump to navigationJump to search
This page is to define Test Report Format for Yocto
Test Summary
Here is a test result summary for a build.
Test Result Matrix
A general Test Result Summary and a detailed Test Result Summary should be provided here.
Example
Test Result Summary | ||||
---|---|---|---|---|
Component | Target | Status | Comments | |
BSP | SugarBay | GOOD | Zypper install/remove has issue; 3D game running failed; | |
CrownBay | BLOCK | emgd crownbay build failed; matchbox-panel segfault; S3 fail on B0 board | ||
JasperForest | GOOD | The only issue is zypper install/remove bug | ||
Blacksand | Buggy | mouse/keyboard do not work in X window; zypper install/remove not work well | ||
Beagleboard | GOOD | Only one RTC issue exists | ||
Routerstationpro | GOOD | Only zypper install has issue | ||
Mpc8315e-rdb | GOOD | Only zypper install has issue | ||
QEMU | qemux86 | Buggy | Only zypper install/remove has issue | |
qemux86-64 | Buggy | Only zypper install/remove has issue | ||
qemuarm | Buggy | zypper could not work well | ||
qemuppc | Buggy | zypper could not work well | ||
qemumips | Buggy | Only zypper install has issue | ||
ADT | BLOCK | i686 toolchain not copied to sharing folder; unfs does not work with qemuppc; | ||
Core Build System | BUGGY | sstate and non-gplv3 build do not work correctly | ||
Stress | GOOD | Sugarbay pass 18 hours stress and Jasperforest pass 12 hours stress | ||
Compliance | GOOD | Sugarbay pass LTP and POSIX test | ||
Power/Performance | GOOD | |||
Note:
BLOCK | Critical bugs, more than 50% test cases are blocked |
GOOD | Only Normal, Minor or Enhancement bugs, less than 10% test cases failed |
BUGGY | Normal, Major and Critical bugs, more than 10% test cases failed |
Detailed Test Result for each component | |||||
---|---|---|---|---|---|
Target | Total TCs | Not Run | Passed | Failed | Not testable (Blocked) |
Sugarbay Sato-SDK | 64 | 0 | 60 | 1 (bug 1101) | 3 (bug 1112, 883) |
Crownbay Sato-SDK | 60 | 0 | 55 | 1 (bug 1099) | 4 |
Jasperforest LSB-SDK | 34 | 0 | 34 | 0 | 0 |
n450 Sato-SDK | 60 | 0 | 52 | 2 (bug 613, 1099) | 6 (bug 613) |
eMenlow Sato-SDK | 60 | 0 | 48 | 2 (bug 1099, 503) | 6 (bug 503) |
Routerstationpro Sato-SDK | 29 | 0 | 29 | 0 | 0 |
Mpc8315e-rdb Sato-SDK | 30 | 0 | 26 | 4 (bug 766) | 0 |
Qemux86-64 Sato | 28 | 0 | 28 | 0 | 0 |
Qemux86-64 Sato-SDK | 31 | 0 | 31 | 0 | 0 |
Qemux86 Sato | 28 | 0 | 28 | 0 | 0 |
Qemux86 Sato-SDK | 31 | 0 | 31 | 0 | 0 |
Qemumips Sato | 26 | 0 | 26 | 0 | 0 |
Qemumips Sato-SDK | 32 | 0 | 32 | 0 | 0 |
Qemuppc Sato | 20 | 0 | 19 | 0 | 1 (bug 414) |
Qemuppc Sato-SDK | 26 | 0 | 25 | 0 | 1 (bug 414) |
Qemuarm Sato | 26 | 0 | 26 | 0 | 0 |
Qemuarm Sato-SDK | 32 | 0 | 32 | 0 | 0 |
Core Build System | 8 | 0 | 7 | 1 (bug 1133) | 0 |
ADT | 6 | 0 | 5 | 1 (bug 414) | 0 |
Power/Performance | 4 | 0 | 4 | 0 | 0 |
Stress | 2 | 0 | 2 | 0 | 0 |
Total | 637 | 0 | 600 | 12 | 21 |
- You can check the detailed test result in attachment for each target.
- The failed/blocked case number is listed with failed cases’ bug number.
Commit Information
This section is to provide tree/branch commit information for testing.
Example:
Tree/Branch: Poky/1.1_M1
Poky Commit: 4ff7af11ef69849ef9c16f585eae58ac920b222b
Meta Branch: 1.1_M1
Meta Commit: fc719f0cd6530ce15148b4aa274f1644b461b298
Issue Summary
This section shows a list of open bugs for each components.
Example
Component | Bug Number | Target Milestone | |
---|---|---|---|
System & Core OS | Installation & Boot | Bug 1006: [mpc8315e-rdb & routerstationpro] minimal,sato,sato-sdk images boot failed (nightly build 20110423-1) | 1.1 |
Bug 1099: [blacksand/emenlow/crownbay] grub install failed with sato-sdk live image | 1.1 M2 | ||
Power Mangement | Bug 613: [Blacksand]system cannot enter S3 standby mode | 1.1 | |
Bug 503: [emenlow] system cannot enter S3 standby mode | 1.1 | ||
Others | Bug 766: [mpc8315e-rdb] USB does not work on mpc8315e-rdb | 1.0 M4 | |
Bug 1101: keyboard pop up when clicking terminal icon | 1.0 M2 | ||
Bug 1107: Open or add person in contacts may segfault on sugarbay/blacksand | 1.1 M2 | ||
Bug 1097: The mouse can not click accurate to the icons in qemumips X | 1.1 M2 | ||
Graphics | Bug 1112: [sugarbay] mesa-demos should be added into sato image for graphics testing | 1.1 M2 | |
Bug 883: [Sugarbay] GFX 3D game test fail to run with sugarbay BSP image due to libsdl.so | 1.1 | ||
Multimedia | Works well in sugarbay/n450/eMenlow/crownbay | ||
Core Build System | New! Bug 1136: do_menuconfig for linux-yocto needs ${B} exists | ||
New! Bug 1133: No provider for core-image-minimal/basic for non-GPLv3 build | 1.1 | M2 | |
1.1 M2 | |||
Compliance | Compliance testing is finished, but result is still under analysis. LTP test result is on https://wiki.pokylinux.org/wiki/LTP_result | ||
POSIX test result is on https://wiki.pokylinux.org/wiki/Posix_result | |||
A new LTP failure is found and reported as bug 1138: [LTP] Some growfiles cases failed with Yocto 1.1 M1 build. | |||
Stress | Helltest and Crashme could pass 24 hours testing | ||
ADT | Bug 414: [PPC] kernel panic when booting poky-image-sdk-qemuppc through UNFS | 1.1 M3 | |