Wiki Test Report Format: Difference between revisions

From Yocto Project
Jump to navigationJump to search
m (Benjamin moved page Test Report Format to Wiki Test Report Format)
 
(80 intermediate revisions by one other user not shown)
Line 7: Line 7:
''A general Test Result Summary and a detailed Test Result Summary should be provided here.
''A general Test Result Summary and a detailed Test Result Summary should be provided here.


===== Example =====
'''Example'''
{| border="1" {{table}}
{| border="1" {{table}}
! colspan="5" align="center" style="background:#f0f0f0;"|'''Test Result Summary'''
! colspan="5" align="center" style="background:#f0f0f0;"|'''Test Result Summary'''
Line 53: Line 53:
|}
|}


'''Detailed Test Result Summary'''
'''Note:'''
'''Image Test Result'''
{| {{table}}
|-
| bgcolor="red" | BLOCK || Critical bugs, more than 50% test cases are blocked
|-
| bgcolor="green" | GOOD ||Only Normal, Minor or Enhancement bugs, less than 10% test cases failed
|-
| bgcolor="orange" | BUGGY ||Normal, Major and Critical bugs, more than 10% test cases failed
|-
|
|}
 
{| border="1" {{table}}
{| border="1" {{table}}
| align="center" style="background:#f0f0f0;"|'''Test Target'''
! colspan="6" align="center" style="background:#f0f0f0;"|'''Detailed Test Result for each component'''
| align="center" style="background:#f0f0f0;"|'''System & Core OS'''
|-
| align="center" style="background:#f0f0f0;"|'''SDK'''
| align="center" style="background:#f0f0f0;"|'''Target'''
| align="center" style="background:#f0f0f0;"|'''Compliance'''
| align="center" style="background:#f0f0f0;"|'''Total TCs'''
| align="center" style="background:#f0f0f0;"|'''Stress'''
| align="center" style="background:#f0f0f0;"|'''Not Run'''
| align="center" style="background:#f0f0f0;"|'''Power/Performance'''
| align="center" style="background:#f0f0f0;"|'''Passed'''
| align="center" style="background:#f0f0f0;"|'''Graphics'''
| align="center" style="background:#f0f0f0;"|'''Failed'''
| align="center" style="background:#f0f0f0;"|'''Mulitimedia'''
| align="center" style="background:#f0f0f0;"|'''Not testable (Blocked)'''
|-
| '''Sugarbay Sato-SDK'''||64||0|| <span style="color:green"> 60 </span>|| <span style="color:red"> 1 (bug 1101) </span>|| <span style="color:red"> 3 (bug 1112, 883) </span>
|-
| '''Crownbay Sato-SDK'''||60||0||<span style="color:green"> 55 </span>||<span style="color:red"> 1 (bug 1099) </span>||<span style="color:red"> 4 </span>
|-
|-
| '''qemux86'''||bgcolor="orange"| Buggy||bgcolor="green" |GOOD|| || || || ||
| '''Jasperforest LSB-SDK'''||34||0||<span style="color:green"> 34 </span> || <span style="color:red"> 0 </span>|| <span style="color:red"> 0 </span>
|-
|-
| '''qemux86-64'''||bgcolor="orange"| Buggy||bgcolor="green" |GOOD|| || || || ||
| '''n450 Sato-SDK'''||60||0||<span style="color:green">52</span>||<span style="color:red">2 (bug 613, 1099)</span>||<span style="color:red">6 (bug 613) </span>
|-
|-
| '''qemuarm'''||bgcolor="orange"| Buggy||bgcolor="green" |GOOD|| || || || ||
| '''eMenlow Sato-SDK'''||60||0||<span style="color:green">48</span>||<span style="color:red">2 (bug 1099, 503)</span>||<span style="color:red">6 (bug 503) </span>
|-
|-
| '''qemumips'''||bgcolor="orange"| Buggy||bgcolor="green" |GOOD|| || || || ||
| '''Routerstationpro Sato-SDK'''||29||0||<span style="color:green"> 29 </span>||<span style="color:red">0</span>||<span style="color:red">0 </span>
|-
|-
| '''qemuppc'''||bgcolor="orange"| Buggy||bgcolor="green" |GOOD|| || || || ||
| '''Mpc8315e-rdb Sato-SDK'''||30||0||<span style="color:green">26</span>||<span style="color:red">4 (bug 766)</span>||<span style="color:red">0</span>
|-
|-
| '''SugarBay'''||bgcolor="green" |GOOD|| ||bgcolor="orange"| Buggy||bgcolor="green" |GOOD||bgcolor="green" |GOOD (May add Link to P&P result page)||bgcolor="green" |GOOD||bgcolor="green" |GOOD
| '''Qemux86-64 Sato'''||28||0||<span style="color:green">28</span>||<span style="color:red">0</span>||<span style="color:red">0</span>
|-
|-
| '''CrownBay'''||bgcolor="green" |GOOD|| ||bgcolor="green" |GOOD|| || ||bgcolor="green" |GOOD||bgcolor="green" |GOOD
| '''Qemux86-64 Sato-SDK'''||31||0||<span style="color:green">31</span>||<span style="color:red">0</span>||<span style="color:red">0</span>
|-
|-
| '''CrownBay noemgd'''||bgcolor="green" |GOOD || || || || || ||
| '''Qemux86 Sato'''||28||0||<span style="color:green">28</span>||<span style="color:red">0</span>||<span style="color:red">0</span>
|-
|-
| '''FishRiver Island 2'''||bgcolor="green" |GOOD|| ||bgcolor="orange" | Buggy|| ||bgcolor="green" |GOOD(May add Link to P&P result page)||bgcolor="green" |GOOD||bgcolor="green" |GOOD
| '''Qemux86 Sato-SDK'''||31||0||<span style="color:green">31</span>||<span style="color:red">0</span>||<span style="color:red">0</span>
|-
|-
| '''HuronRiver'''||bgcolor="red" |BLOCK|| || || ||bgcolor="red" |BLOCK||bgcolor="red" |BLOCK||bgcolor="red" |BLOCK
| '''Qemumips Sato'''||26||0||<span style="color:green">26</span>||<span style="color:red">0</span>||<span style="color:red">0</span>
|-
|-
| '''Jasperforest'''||bgcolor="red" |BLOCK|| ||bgcolor="red" |BLOCK||bgcolor="green" |GOOD|| || ||
| '''Qemumips Sato-SDK'''||32||0||<span style="color:green">32</span>||<span style="color:red">0</span>||<span style="color:red">0</span>
|-
|-
| '''Blacksand'''||bgcolor="orange"| Buggy|| || || || || ||bgcolor="orange" | Buggy
| '''Qemuppc Sato'''||20||0||<span style="color:green">19</span>||<span style="color:red">0</span>||<span style="color:red">1 (bug 414)</span>
|-
|-
| '''eMenlow'''||bgcolor="green" |GOOD|| || || || || ||bgcolor="green" |GOOD
| '''Qemuppc Sato-SDK'''||26||0||<span style="color:green">25</span>||<span style="color:red">0</span>||<span style="color:red">1 (bug 414) </span>
|-
|-
|  
| '''Qemuarm Sato'''||26||0||<span style="color:green">26</span>||<span style="color:red">0</span>||<span style="color:red">0</span>
|}
|-
'''Poky Test Result'''
| '''Qemuarm Sato-SDK'''||32||0||<span style="color:green">32</span>||<span style="color:red">0</span>||<span style="color:red">0</span>
{| {{table}}
|-
| align="center" style="background:#f0f0f0;"|'''Test Target'''
| '''Core Build System'''||8||0||<span style="color:green">7</span>||<span style="color:red">1 (bug 1133)</span>||<span style="color:red">0</span>
| align="center" style="background:#f0f0f0;"|'''Functionality'''
|-
| align="center" style="background:#f0f0f0;"|'''Performance'''
| '''ADT'''||6||0||<span style="color:green">5</span>||<span style="color:red">1 (bug 414)</span>||<span style="color:red">0</span>
| align="center" style="background:#f0f0f0;"|'''Distribution Support'''
|-
|-
| '''Poky'''||bgcolor="orange"|Buggy||bgcolor="green" |GOOD||bgcolor="green" |GOOD
| '''Power/Performance'''||4||0||<span style="color:green">4</span>||<span style="color:red">0</span>||<span style="color:red">0</span>
|-
|-
|  
| '''Stress'''||2||0||<span style="color:green">2</span>||<span style="color:red">0</span>||<span style="color:red">0</span>
|}
'''Documentation Test Result'''
{| {{table}}
| align="center" style="background:#f0f0f0;"|''''''
| align="center" style="background:#f0f0f0;"|'''Quick Start Doc'''
|-
|-
| '''Documentation'''||bgcolor="green" |GOOD
| '''Total'''||637||0||<span style="color:green">600</span>||<span style="color:red">12</span>||<span style="color:red">21</span>
|-
|-
|  
|  
|}
|}


* You can check the detailed test result in attachment for each target.
* The failed/blocked case number is listed with failed cases’ bug number.


'''Note:'''
==== Commit Information ====
{| {{table}}
This section is to provide tree/branch commit information for testing.
 
'''Example:'''
 
Tree/Branch: Poky/1.1_M1
 
Poky Commit: 4ff7af11ef69849ef9c16f585eae58ac920b222b
 
Meta Branch: 1.1_M1
 
Meta Commit: fc719f0cd6530ce15148b4aa274f1644b461b298
 
==== Issue Summary ====
This section shows a list of open bugs for each components.
 
'''Example'''
{| border="1" {{table}}
! colspan="2" align="center" style="background:#f0f0f0;"|'''Component'''
| align="center" style="background:#f0f0f0;"|'''Bug Number'''
| align="center" style="background:#f0f0f0;"|'''Target Milestone'''
|-
| rowspan="8" | '''System & Core OS'''
| rowspan="2" align="center" | '''Installation & Boot'''
| Bug 1006: [mpc8315e-rdb & routerstationpro] minimal,sato,sato-sdk images boot failed (nightly build 20110423-1) || 1.1
|-
| Bug 1099: [blacksand/emenlow/crownbay] grub install failed with sato-sdk live image || 1.1 M2
|-
| rowspan="2" align="center" | '''Power Mangement'''
| Bug 613: [Blacksand]system cannot enter S3 standby mode||1.1
|-
| Bug 503: [emenlow] system cannot enter S3 standby mode ||1.1
|-
| rowspan="4" align="center" | '''Others'''
| Bug 766: [mpc8315e-rdb] USB does not work on mpc8315e-rdb||1.0 M4
|-
| Bug 1101: keyboard pop up when clicking terminal icon||1.0 M2
|-
| Bug 1107: Open or add person in contacts may segfault on sugarbay/blacksand||1.1 M2
|-
| Bug 1097: The mouse can not click accurate to the icons in qemumips X||1.1 M2
|-
| colspan="2" rowspan="2" align="center" | '''Graphics'''
| Bug 1112: [sugarbay] mesa-demos should be added into sato image for graphics testing||1.1 M2
|-
| Bug 883: [Sugarbay] GFX 3D game test fail to run with sugarbay BSP image due to libsdl.so ||1.1
|-
| colspan="2" align="center | '''Multimedia'''||Works well in sugarbay/n450/eMenlow/crownbay||　
|-
| colspan="2" rowspan="2" align="center" | '''Core Build System'''
| <span style="color:red">New!</span> Bug 1136: do_menuconfig for linux-yocto needs ${B} exists || 1.1 M2
|-
| <span style="color:red">New!</span> Bug 1133: No provider for core-image-minimal/basic for non-GPLv3 build||1.1 M2
|-
|-
| bgcolor="red" | BLOCK || Critical bugs, more than 50% test cases are blocked
| colspan="2" align="center | '''Compliance'''||Compliance testing is finished, but result is still under analysis. LTP test result is on https://wiki.pokylinux.org/wiki/LTP_result &nbsp; POSIX test result is on https://wiki.pokylinux.org/wiki/Posix_result &nbsp;  A new LTP failure is found and reported as bug 1138: [LTP] Some growfiles cases failed with Yocto 1.1 M1 build.||　
|-
|-
| bgcolor="green" | GOOD ||Only Normal, Minor or Enhancement bugs, less than 10% test cases failed
| colspan="2" align="center | '''Stress'''||Helltest and Crashme could pass 24 hours testing||　
|-
|-
| bgcolor="orange" | BUGGY ||Normal, Major and Critical bugs, more than 10% test cases failed
| colspan="2" align="center | '''ADT''' ||Bug 414: [PPC] kernel panic when booting poky-image-sdk-qemuppc through UNFS||1.1 M3
|-
|-
|  
|  
|}
|}

Latest revision as of 20:13, 10 November 2016

This page is to define Test Report Format for Yocto

Test Summary

Here is a test result summary for a build.

Test Result Matrix

A general Test Result Summary and a detailed Test Result Summary should be provided here.

Example

Test Result Summary
Component Target Status Comments
BSP SugarBay GOOD Zypper install/remove has issue; 3D game running failed;
　 CrownBay BLOCK emgd crownbay build failed; matchbox-panel segfault; S3 fail on B0 board
　 JasperForest GOOD The only issue is zypper install/remove bug
　 Blacksand Buggy mouse/keyboard do not work in X window; zypper install/remove not work well
　 Beagleboard GOOD Only one RTC issue exists
　 Routerstationpro GOOD Only zypper install has issue
　 Mpc8315e-rdb GOOD Only zypper install has issue
QEMU qemux86 Buggy Only zypper install/remove has issue
　 qemux86-64 Buggy Only zypper install/remove has issue
　 qemuarm Buggy zypper could not work well
　 qemuppc Buggy zypper could not work well
　 qemumips Buggy Only zypper install has issue
ADT 　 BLOCK i686 toolchain not copied to sharing folder; unfs does not work with qemuppc;
Core Build System 　 BUGGY sstate and non-gplv3 build do not work correctly
Stress 　 GOOD Sugarbay pass 18 hours stress and Jasperforest pass 12 hours stress
Compliance 　 GOOD Sugarbay pass LTP and POSIX test
Power/Performance 　 GOOD 　

Note:

BLOCK Critical bugs, more than 50% test cases are blocked
GOOD Only Normal, Minor or Enhancement bugs, less than 10% test cases failed
BUGGY Normal, Major and Critical bugs, more than 10% test cases failed
Detailed Test Result for each component
Target Total TCs Not Run Passed Failed Not testable (Blocked)
Sugarbay Sato-SDK 64 0 60 1 (bug 1101) 3 (bug 1112, 883)
Crownbay Sato-SDK 60 0 55 1 (bug 1099) 4
Jasperforest LSB-SDK 34 0 34 0 0
n450 Sato-SDK 60 0 52 2 (bug 613, 1099) 6 (bug 613)
eMenlow Sato-SDK 60 0 48 2 (bug 1099, 503) 6 (bug 503)
Routerstationpro Sato-SDK 29 0 29 0 0
Mpc8315e-rdb Sato-SDK 30 0 26 4 (bug 766) 0
Qemux86-64 Sato 28 0 28 0 0
Qemux86-64 Sato-SDK 31 0 31 0 0
Qemux86 Sato 28 0 28 0 0
Qemux86 Sato-SDK 31 0 31 0 0
Qemumips Sato 26 0 26 0 0
Qemumips Sato-SDK 32 0 32 0 0
Qemuppc Sato 20 0 19 0 1 (bug 414)
Qemuppc Sato-SDK 26 0 25 0 1 (bug 414)
Qemuarm Sato 26 0 26 0 0
Qemuarm Sato-SDK 32 0 32 0 0
Core Build System 8 0 7 1 (bug 1133) 0
ADT 6 0 5 1 (bug 414) 0
Power/Performance 4 0 4 0 0
Stress 2 0 2 0 0
Total 637 0 600 12 21
  • You can check the detailed test result in attachment for each target.
  • The failed/blocked case number is listed with failed cases’ bug number.

Commit Information

This section is to provide tree/branch commit information for testing.

Example:

Tree/Branch: Poky/1.1_M1

Poky Commit: 4ff7af11ef69849ef9c16f585eae58ac920b222b

Meta Branch: 1.1_M1

Meta Commit: fc719f0cd6530ce15148b4aa274f1644b461b298

Issue Summary

This section shows a list of open bugs for each components.

Example

Component Bug Number Target Milestone
System & Core OS Installation & Boot Bug 1006: [mpc8315e-rdb & routerstationpro] minimal,sato,sato-sdk images boot failed (nightly build 20110423-1) 1.1
Bug 1099: [blacksand/emenlow/crownbay] grub install failed with sato-sdk live image 1.1 M2
Power Mangement Bug 613: [Blacksand]system cannot enter S3 standby mode 1.1
Bug 503: [emenlow] system cannot enter S3 standby mode 1.1
Others Bug 766: [mpc8315e-rdb] USB does not work on mpc8315e-rdb 1.0 M4
Bug 1101: keyboard pop up when clicking terminal icon 1.0 M2
Bug 1107: Open or add person in contacts may segfault on sugarbay/blacksand 1.1 M2
Bug 1097: The mouse can not click accurate to the icons in qemumips X 1.1 M2
Graphics Bug 1112: [sugarbay] mesa-demos should be added into sato image for graphics testing 1.1 M2
Bug 883: [Sugarbay] GFX 3D game test fail to run with sugarbay BSP image due to libsdl.so 1.1
Multimedia Works well in sugarbay/n450/eMenlow/crownbay 　
Core Build System New! Bug 1136: do_menuconfig for linux-yocto needs ${B} exists 1.1 M2
New! Bug 1133: No provider for core-image-minimal/basic for non-GPLv3 build 1.1 M2
Compliance Compliance testing is finished, but result is still under analysis. LTP test result is on https://wiki.pokylinux.org/wiki/LTP_result   POSIX test result is on https://wiki.pokylinux.org/wiki/Posix_result   A new LTP failure is found and reported as bug 1138: [LTP] Some growfiles cases failed with Yocto 1.1 M1 build. 　
Stress Helltest and Crashme could pass 24 hours testing 　
ADT Bug 414: [PPC] kernel panic when booting poky-image-sdk-qemuppc through UNFS 1.1 M3
Retrieved from "https://wiki.yoctoproject.org/wiki/index.php?title=Wiki_Test_Report_Format&oldid=21184"