Wiki Test Report Format
From Yocto Project
(Redirected from Test Report Format)
Jump to navigationJump to search
This page is to define Test Report Format for Yocto
Test Summary
Here is a test result summary for a build.
Test Result Matrix
A general Test Result Summary and a detailed Test Result Summary should be provided here.
Example
| Test Result Summary | ||||
|---|---|---|---|---|
| Component | Target | Status | Comments | |
| BSP | SugarBay | GOOD | Zypper install/remove has issue; 3D game running failed; | |
| CrownBay | BLOCK | emgd crownbay build failed; matchbox-panel segfault; S3 fail on B0 board | ||
| JasperForest | GOOD | The only issue is zypper install/remove bug | ||
| Blacksand | Buggy | mouse/keyboard do not work in X window; zypper install/remove not work well | ||
| Beagleboard | GOOD | Only one RTC issue exists | ||
| Routerstationpro | GOOD | Only zypper install has issue | ||
| Mpc8315e-rdb | GOOD | Only zypper install has issue | ||
| QEMU | qemux86 | Buggy | Only zypper install/remove has issue | |
| qemux86-64 | Buggy | Only zypper install/remove has issue | ||
| qemuarm | Buggy | zypper could not work well | ||
| qemuppc | Buggy | zypper could not work well | ||
| qemumips | Buggy | Only zypper install has issue | ||
| ADT | BLOCK | i686 toolchain not copied to sharing folder; unfs does not work with qemuppc; | ||
| Core Build System | BUGGY | sstate and non-gplv3 build do not work correctly | ||
| Stress | GOOD | Sugarbay pass 18 hours stress and Jasperforest pass 12 hours stress | ||
| Compliance | GOOD | Sugarbay pass LTP and POSIX test | ||
| Power/Performance | GOOD | |||
Note:
| BLOCK | Critical bugs, more than 50% test cases are blocked |
| GOOD | Only Normal, Minor or Enhancement bugs, less than 10% test cases failed |
| BUGGY | Normal, Major and Critical bugs, more than 10% test cases failed |
| Detailed Test Result for each component | |||||
|---|---|---|---|---|---|
| Target | Total TCs | Not Run | Passed | Failed | Not testable (Blocked) |
| Sugarbay Sato-SDK | 64 | 0 | 60 | 1 (bug 1101) | 3 (bug 1112, 883) |
| Crownbay Sato-SDK | 60 | 0 | 55 | 1 (bug 1099) | 4 |
| Jasperforest LSB-SDK | 34 | 0 | 34 | 0 | 0 |
| n450 Sato-SDK | 60 | 0 | 52 | 2 (bug 613, 1099) | 6 (bug 613) |
| eMenlow Sato-SDK | 60 | 0 | 48 | 2 (bug 1099, 503) | 6 (bug 503) |
| Routerstationpro Sato-SDK | 29 | 0 | 29 | 0 | 0 |
| Mpc8315e-rdb Sato-SDK | 30 | 0 | 26 | 4 (bug 766) | 0 |
| Qemux86-64 Sato | 28 | 0 | 28 | 0 | 0 |
| Qemux86-64 Sato-SDK | 31 | 0 | 31 | 0 | 0 |
| Qemux86 Sato | 28 | 0 | 28 | 0 | 0 |
| Qemux86 Sato-SDK | 31 | 0 | 31 | 0 | 0 |
| Qemumips Sato | 26 | 0 | 26 | 0 | 0 |
| Qemumips Sato-SDK | 32 | 0 | 32 | 0 | 0 |
| Qemuppc Sato | 20 | 0 | 19 | 0 | 1 (bug 414) |
| Qemuppc Sato-SDK | 26 | 0 | 25 | 0 | 1 (bug 414) |
| Qemuarm Sato | 26 | 0 | 26 | 0 | 0 |
| Qemuarm Sato-SDK | 32 | 0 | 32 | 0 | 0 |
| Core Build System | 8 | 0 | 7 | 1 (bug 1133) | 0 |
| ADT | 6 | 0 | 5 | 1 (bug 414) | 0 |
| Power/Performance | 4 | 0 | 4 | 0 | 0 |
| Stress | 2 | 0 | 2 | 0 | 0 |
| Total | 637 | 0 | 600 | 12 | 21 |
- You can check the detailed test result in attachment for each target.
- The failed/blocked case number is listed with failed cases’ bug number.
Commit Information
This section is to provide tree/branch commit information for testing.
Example:
Tree/Branch: Poky/1.1_M1
Poky Commit: 4ff7af11ef69849ef9c16f585eae58ac920b222b
Meta Branch: 1.1_M1
Meta Commit: fc719f0cd6530ce15148b4aa274f1644b461b298
Issue Summary
This section shows a list of open bugs for each components.
Example
| Component | Bug Number | Target Milestone | |
|---|---|---|---|
| System & Core OS | Installation & Boot | Bug 1006: [mpc8315e-rdb & routerstationpro] minimal,sato,sato-sdk images boot failed (nightly build 20110423-1) | 1.1 |
| Bug 1099: [blacksand/emenlow/crownbay] grub install failed with sato-sdk live image | 1.1 M2 | ||
| Power Mangement | Bug 613: [Blacksand]system cannot enter S3 standby mode | 1.1 | |
| Bug 503: [emenlow] system cannot enter S3 standby mode | 1.1 | ||
| Others | Bug 766: [mpc8315e-rdb] USB does not work on mpc8315e-rdb | 1.0 M4 | |
| Bug 1101: keyboard pop up when clicking terminal icon | 1.0 M2 | ||
| Bug 1107: Open or add person in contacts may segfault on sugarbay/blacksand | 1.1 M2 | ||
| Bug 1097: The mouse can not click accurate to the icons in qemumips X | 1.1 M2 | ||
| Graphics | Bug 1112: [sugarbay] mesa-demos should be added into sato image for graphics testing | 1.1 M2 | |
| Bug 883: [Sugarbay] GFX 3D game test fail to run with sugarbay BSP image due to libsdl.so | 1.1 | ||
| Multimedia | Works well in sugarbay/n450/eMenlow/crownbay | ||
| Core Build System | New! Bug 1136: do_menuconfig for linux-yocto needs ${B} exists | 1.1 M2 | |
| New! Bug 1133: No provider for core-image-minimal/basic for non-GPLv3 build | 1.1 M2 | ||
| Compliance | Compliance testing is finished, but result is still under analysis. LTP test result is on https://wiki.pokylinux.org/wiki/LTP_result POSIX test result is on https://wiki.pokylinux.org/wiki/Posix_result A new LTP failure is found and reported as bug 1138: [LTP] Some growfiles cases failed with Yocto 1.1 M1 build. | ||
| Stress | Helltest and Crashme could pass 24 hours testing | ||
| ADT | Bug 414: [PPC] kernel panic when booting poky-image-sdk-qemuppc through UNFS | 1.1 M3 | |
