Wiki Test Report Format

This page is to define Test Report Format for Yocto

Test Summary

Here is a test result summary for a build.

Test Result Matrix

A general Test Result Summary and a detailed Test Result Summary should be provided here.

Example

Test Result Summary
Component	Target	Status	Comments
BSP	SugarBay	GOOD	Zypper install/remove has issue; 3D game running failed;
	CrownBay	BLOCK	emgd crownbay build failed; matchbox-panel segfault; S3 fail on B0 board
	JasperForest	GOOD	The only issue is zypper install/remove bug
	Blacksand	Buggy	mouse/keyboard do not work in X window; zypper install/remove not work well
	Beagleboard	GOOD	Only one RTC issue exists
	Routerstationpro	GOOD	Only zypper install has issue
	Mpc8315e-rdb	GOOD	Only zypper install has issue
QEMU	qemux86	Buggy	Only zypper install/remove has issue
	qemux86-64	Buggy	Only zypper install/remove has issue
	qemuarm	Buggy	zypper could not work well
	qemuppc	Buggy	zypper could not work well
	qemumips	Buggy	Only zypper install has issue
ADT		BLOCK	i686 toolchain not copied to sharing folder; unfs does not work with qemuppc;
Core Build System		BUGGY	sstate and non-gplv3 build do not work correctly
Stress		GOOD	Sugarbay pass 18 hours stress and Jasperforest pass 12 hours stress
Compliance		GOOD	Sugarbay pass LTP and POSIX test
Power/Performance		GOOD

Note:

BLOCK	Critical bugs, more than 50% test cases are blocked
GOOD	Only Normal, Minor or Enhancement bugs, less than 10% test cases failed
BUGGY	Normal, Major and Critical bugs, more than 10% test cases failed

Detailed Test Result for each component
Target	Total TCs	Not Run	Passed	Failed	Not testable (Blocked)
Sugarbay Sato-SDK	64	0	60	1 (bug 1101)	3 (bug 1112, 883)
Crownbay Sato-SDK	60	0	55	1 (bug 1099)	4
Jasperforest LSB-SDK	34	0	34	0	0
n450 Sato-SDK	60	0	52	2 (bug 613, 1099)	6 (bug 613)
eMenlow Sato-SDK	60	0	48	2 (bug 1099, 503)	6 (bug 503)
Routerstationpro Sato-SDK	29	0	29	0	0
Mpc8315e-rdb Sato-SDK	30	0	26	4 (bug 766)	0
Qemux86-64 Sato	28	0	28	0	0
Qemux86-64 Sato-SDK	31	0	31	0	0
Qemux86 Sato	28	0	28	0	0
Qemux86 Sato-SDK	31	0	31	0	0
Qemumips Sato	26	0	26	0	0
Qemumips Sato-SDK	32	0	32	0	0
Qemuppc Sato	20	0	19	0	1 (bug 414)
Qemuppc Sato-SDK	26	0	25	0	1 (bug 414)
Qemuarm Sato	26	0	26	0	0
Qemuarm Sato-SDK	32	0	32	0	0
Core Build System	8	0	7	1 (bug 1133)	0
ADT	6	0	5	1 (bug 414)	0
Power/Performance	4	0	4	0	0
Stress	2	0	2	0	0
Total	637	0	600	12	21

• You can check the detailed test result in attachment for each target.
• The failed/blocked case number is listed with failed cases’ bug number.

Commit Information

This section is to provide tree/branch commit information for testing.

Example:

Tree/Branch: Poky/1.1_M1

Poky Commit: 4ff7af11ef69849ef9c16f585eae58ac920b222b

Meta Branch: 1.1_M1

Meta Commit: fc719f0cd6530ce15148b4aa274f1644b461b298

Issue Summary

This section shows a list of open bugs for each components.

Example

Component		Bug Number	Target Milestone
System & Core OS	Installation & Boot	Bug 1006: [mpc8315e-rdb & routerstationpro] minimal,sato,sato-sdk images boot failed (nightly build 20110423-1)	1.1
		Bug 1099: [blacksand/emenlow/crownbay] grub install failed with sato-sdk live image	1.1 M2
	Power Mangement	Bug 613: [Blacksand]system cannot enter S3 standby mode	1.1
		Bug 503: [emenlow] system cannot enter S3 standby mode	1.1
	Others	Bug 766: [mpc8315e-rdb] USB does not work on mpc8315e-rdb	1.0 M4
		Bug 1101: keyboard pop up when clicking terminal icon	1.0 M2
		Bug 1107: Open or add person in contacts may segfault on sugarbay/blacksand	1.1 M2
		Bug 1097: The mouse can not click accurate to the icons in qemumips X	1.1 M2
Graphics		Bug 1112: [sugarbay] mesa-demos should be added into sato image for graphics testing	1.1 M2
		Bug 883: [Sugarbay] GFX 3D game test fail to run with sugarbay BSP image due to libsdl.so	1.1
Multimedia		Works well in sugarbay/n450/eMenlow/crownbay
Core Build System		New! Bug 1136: do_menuconfig for linux-yocto needs ${B} exists	1.1 M2
		New! Bug 1133: No provider for core-image-minimal/basic for non-GPLv3 build	1.1 M2
Compliance		Compliance testing is finished, but result is still under analysis. LTP test result is on https://wiki.pokylinux.org/wiki/LTP_result   POSIX test result is on https://wiki.pokylinux.org/wiki/Posix_result   A new LTP failure is found and reported as bug 1138: [LTP] Some growfiles cases failed with Yocto 1.1 M1 build.
Stress		Helltest and Crashme could pass 24 hours testing
ADT		Bug 414: [PPC] kernel panic when booting poky-image-sdk-qemuppc through UNFS	1.1 M3