QA REPORT TEMPLATE

TITLE : WW## - YYYY-MM-DD - Full Test Cycle 2.x Mx rcx

{{#vardefine:adt_fedora_i686| }} {{#vardefine:adt_fedora_x86-64| }} {{#vardefine:adt_ubuntu_x86-64| }} {{#vardefine:beaglebone_black| }} {{#vardefine:edgerouter| }} {{#vardefine:genericx86| }} {{#vardefine:genericx86_64| }} {{#vardefine:mpc8315e-rdb| }} {{#vardefine:p1022ds-rdb| }} {{#vardefine:qemux86| }} {{#vardefine:qemuarm| }} {{#vardefine:qemumips| }} {{#vardefine:qemuppc| }} {{#vardefine:qemux86-64| }} {{#vardefine:eclipseplugin| }} {{#vardefine:meta-yocto| }} {{#vardefine:eSDK| }} {{#vardefine:toaster| }} {{#vardefine:oe-core| }} {{#vardefine:distro| }} {{#vardefine:bitbake| }} {{#vardefine:runtime| }} {{#vardefine:build-appliance| }} {{#vardefine:kernel| }} {{#vardefine:AutBuild-Testing| }} {{#vardefine:CROPS| }}

Testing data

 Test type:
 Branch:
 Build name:
 poky commit:
 Autobuilder images repository:
 Distributions tested: 
 -
 -    

Test Run Summary Report

AutBuild-Testing

{{#testopia:

run_id=0, {{#var:bitbake}}, {{#var:oe-core}}, {{#var:meta-yocto}}, {{#var:kernel}}, {{#var:eSDK}}, {{#var:nuc}}, {{#var:atom-pc}}, {{#var:genericx86}}, {{#var:genericx86_64}}, {{#var:beaglebone_black}}, {{#var:edgerouter}}, {{#var:mpc8315e-rdb}}, {{#var:p1022ds-rdb}}, {{#var:qemux86}}, {{#var:qemux86-64}}, {{#var:qemuarm}}, {{#var:qemumips}}, {{#var:qemuppc}}, {{#var:runtime}}, {{#var:build-appliance}}, {{#var:toaster}}, {{#var:adt_opensuse_123_x86-64}}, {{#var:adt_fedora18_i686}}, {{#var:adt_fedora19_x86-64}}, {{#var:adt_fedora19_i686}}, {{#var:adt_fedora_i686}}, {{#var:adt_ubuntu_x86-64}}, {{#var:adt_ubuntu_1310_x86-64}}, {{#var:eclipseplugin}}, {{#var:distro}}, {{#var:AutBuild-Testing}}, {{#var:CROPS}}

report_id=concatstats}} {{#testopia:

chart=bstatus

report_id=bugs

charttype=bar

sortable=true

nudechart=true

chartpos=top }} {{#testopia:

run_id=0, {{#var:bitbake}}, {{#var:oe-core}}, {{#var:meta-yocto}}, {{#var:kernel}}, {{#var:eSDK}}, {{#var:nuc}}, {{#var:atom-pc}}, {{#var:genericx86}}, {{#var:genericx86_64}}, {{#var:beaglebone_black}}, {{#var:edgerouter}}, {{#var:mpc8315e-rdb}}, {{#var:p1022ds-rdb}}, {{#var:qemux86}}, {{#var:qemux86-64}}, {{#var:qemuarm}}, {{#var:qemumips}}, {{#var:qemuppc}}, {{#var:runtime}}, {{#var:build-appliance}}, {{#var:toaster}}, {{#var:adt_opensuse_123_x86-64}}, {{#var:adt_fedora18_i686}}, {{#var:adt_fedora19_x86-64}}, {{#var:adt_fedora19_i686}}, {{#var:adt_fedora_i686}}, {{#var:adt_ubuntu_x86-64}}, {{#var:eclipseplugin}}, {{#var:distro}}, {{#var:AutBuild-Testing}}, {{#var:CROPS}}

charttype=bar

report_id=status}}

Bugs Found during QA Test

New issues Found during QA Test & Community

Meta-yocto
1	Should install stripped-down gnome-icon-theme	High	Ross Burton - Disabled		---	RESOLVED	FIXED	---	2005-10-12

High and M+ issues Found during QA Test (NOT NEW)

Meta-yocto
1	Should install stripped-down gnome-icon-theme	High	Ross Burton - Disabled		---	RESOLVED	FIXED	---	2005-10-12

Other Bugs found during QA Test (NOT NEW)

---
1	Should install stripped-down gnome-icon-theme	High	Ross Burton - Disabled		---	RESOLVED	FIXED	---	2005-10-12

Current community Open Bugs - Medium+/High

Overview of the open bugs opened against current milestone or release which are Medium+ or High and are not targeted for the next milestones

ID	P	Summary	Assignee	QA	Milestone	Status	cf_regression_type	Created
		0
BitBake
10984	Medium+	[Test Case 1058] bitbake can use alternate download methods than git	Richard Purdie	oscar lopez arandas	2.3 M3	CLOSED	---	2017-01-27
		1

Bug Trend

An overview of this week's Yocto Bug Trend charts can be accessed on the following page

Build Performance Test

Detailed description on how performance test are run can be found on Performance Test wiki. The below charts are displaying the build performance trend for 2.1 M4 to 2.3 M1. Performance execution has 2 stages, one executed form 1.6 to 2.1 in RO and another one started on 2.1 from GDC, there are difference between HW used in both location hence the numbers are different.

• RO Charts

• GDC Charts

Below chars are showing the latest performance results taken from current milestone

Performance Charts

NOTE: During the 2.3 M2 QA cycle period, Fedora machine was under debugging activities to investigate more on bug 10884 hence only results for Ubuntu machine are displayed on the charts.

Performance Charts

For Bitbake core-image-sato with rm_work

Building the Virtual kernel

Building the rootfs

Building core-image-sato

eSDK Deploy time

pTest for genericx86-64 on NUC

open bugs

ID	Summary (12 tasks) ⇒	P	Assignee	QA	Milestone	Status	Resolution	cf_regression_type	Created
OE-Core
5697	busybox: ptest cases failed	Medium	Changqing Li		2.7 M2	RESOLVED	FIXED	---	2013-12-31
5695	python: ptest cases failed	Medium	Ross Burton		2.6	RESOLVED	FIXED	---	2013-12-31
10841	pTest : Various test cases are failing on "dbus-test" module	Medium	Ross Burton		2.5	RESOLVED	DUPLICATE	---	2016-12-21
10842	testrunner test timeout can lead to garbage output	Medium	Aníbal Limón		2.3 M4	VERIFIED	FIXED	---	2016-12-21
10840	pTest- openssl: test cases "test_bn" and "test_cms" are failing	Medium+	Alexander Kanavin		2.4 M1	VERIFIED	FIXED	---	2016-12-21
Runtime Testing
9055	pTest - acl (NUC) failed tested on NUC 5425WYK for YP 2.1_M2.rc1	Medium	Robert Yang	Francisco Pedraza	2.4	RESOLVED	FIXED	Regression (Used to work)	2016-02-04
9042	pTest - strace fails on NUC 5425WYK for YP 2.1_M2.rc1	Medium	Changqing Li		2.7 M2	RESOLVED	FIXED	Regression (Used to work)	2016-02-03
8832	pTest - bash tests fail for 2.1_M1.rc1	Medium	Changqing Li		2.7 M2	RESOLVED	WORKSFORME	---	2015-12-17
8831	pTest - busybox tests fail for 2.1_M1.rc1	Medium	dengke		Future	RESOLVED	DUPLICATE	---	2015-12-17
8828	pTest - lttng-tools tests fail	Medium	Richard Purdie	yeoh ee peng	4.99	RESOLVED	FIXED	Regression (Used to work)	2015-12-17
9372	Ptest for acl hang because of lack of two user bin and daemon	Medium	Chen Qi	oscar lopez arandas	2.4	RESOLVED	FIXED	---	2016-04-01
8471	ptest - some valgrind ptests fail on 2.4 M1.rc1	Medium	Maxin B. John		4.99	RESOLVED	DUPLICATE	Regression (Used to work)	2015-10-09

results

< Back
Ran on a NUC and compared with commit 573c646d4cc62dcd0c230381df4940bdf314d495

FULL LOG

Failures

% increased

% decreased

Results

Package Name	Tests ran	Passed	Failed	Passrate	Last Passrate
acl	380	354	26	93.20%	93.20%
attr	143	143	0	100.00%	100.00%
bash	76	72	4	94.70%	94.70%
bluez5	25	25	0	100.00%	100.00%
busybox	518	509	9	98.30%	98.60%
bzip2	6	6	0	100.00%	100.00%
dbus-test	15	4	11	26.70%	26.70%
diffstat	440	440	0	100.00%	100.00%
diffutils	17	17	0	100.00%	100.00%
e2fsprogs	1	1	0	100.00%	0.00%
gawk	298	297	1	99.70%	99.70%
gdbm	22	22	0	100.00%	100.00%
gdk-pixbuf	18	18	0	100.00%	100.00%
glib-2.0	250	250	0	100.00%	97.90%
kbd	7	7	0	100.00%	100.00%
libpcre	3	3	0	100.00%	100.00%
libxml2	85	82	3	96.50%	96.50%
lttng-tools	2057	1988	69	96.60%	99.80%
nettle	88	87	1	98.90%	98.90%
openssh	56	56	0	100.00%	100.00%
openssl	52	52	0	100.00%	96.20%
pango	12	12	0	100.00%	100.00%
perl	2286	2285	1	100.00%	100.00%
perl5	2286	2285	1	100.00%	100.00%
python	32058	31932	126	99.60%	99.60%
quilt	57	57	0	100.00%	98.20%
strace	278	273	5	98.20%	99.60%
tcl	148	147	1	99.30%	100.00%
util-linux	109	107	2	98.20%	98.50%
valgrind	553	2	551	0.40%	0.00%
zlib	2	2	0	100.00%	100.00%

Compliance Report

POSIX results

General results for POSIX execution can be found POSIX_result and greater details on each architecture can be found POSIX_details

Below you will see the last three execution results for comparison:

Yocto 2.# RC3 build

Status	genericx86-64	beaglebone	edgerouter	mpc8315e-rdb
failures	##	##	##	##

Yocto 2.# RC2 build

Status	genericx86-64	beaglebone	edgerouter	mpc8315e-rdb
failures	##	##	##	##

Yocto 2.# RC1 build

Status	genericx86-64	beaglebone	edgerouter	mpc8315e-rdb
failures	##	##	##	##

LTP results

General results for LTP can be found here LTP_results and for greater information click on each architecture to be redirected to their details.

Yocto 2.# RC3 build

Status	genericx86-64	beaglebone	edgerouter	mpc8315e-rdb
total	####	####	####	####
failures	##	##	##	##

Yocto 2.# RC2 build

Status	genericx86-64	beaglebone	edgerouter	mpc8315e-rdb
total	####	####	####	####
failures	##	##	##	##

Yocto 2.# RC1 build

Status	genericx86-64	beaglebone	edgerouter	mpc8315e-rdb
total	####	####	####	####
failures	##	##	##	##

Detailed Report

Distro Testing

Status {{#testopia: |run_id={{#var:distro}} |showhide=true |hidden=true |report_id=status |charttype=pie3}}

Bugs {{#testopia: |run_id={{#var:distro}} |report_id=bugs |chart=bstatus,prio,severity |charttype=pie,pie,pie |showhide=true |hidden=true}}

Agenda {{#testopia: |run_id={{#var:distro}} |report_id=agenda |showhide=true |hidden=true |reportoptions=withdependency=false}}

Core Build System

Bitbake

Status {{#testopia: |run_id={{#var:bitbake}} |showhide=true |hidden=true |report_id=status |charttype=pie3}}

Bugs {{#testopia: |run_id={{#var:bitbake}} |report_id=bugs |chart=bstatus,prio,severity |charttype=pie,pie,pie |showhide=true |hidden=true}}

Agenda {{#testopia: |run_id={{#var:bitbake}} |report_id=agenda |showhide=true |hidden=true |reportoptions=withdependency=false}}

Meta-Yocyo

Status {{#testopia: |run_id={{#var:meta-yocto}} |showhide=true |hidden=true |report_id=status |charttype=pie3}}

Bugs {{#testopia: |run_id={{#var:meta-yocto}} |report_id=bugs |chart=bstatus,prio,severity |charttype=pie,pie,pie |showhide=true |hidden=true}}

Agenda {{#testopia: |run_id={{#var:meta-yocto}} |report_id=agenda |showhide=true |hidden=true |reportoptions=withdependency=false}}

OE-Core

Status {{#testopia: |run_id={{#var:oe-core}} |showhide=true |hidden=true |report_id=status |charttype=pie3}}

Bugs {{#testopia: |run_id={{#var:oe-core}} |report_id=bugs |chart=bstatus,prio,severity |charttype=pie,pie,pie |showhide=true |hidden=true}}

Agenda {{#testopia: |run_id={{#var:oe-core}} |report_id=agenda |showhide=true |hidden=true |reportoptions=withdependency=false}}

eSDK

Status {{#testopia: |run_id={{#var:eSDK}} |showhide=true |hidden=true |report_id=status |charttype=pie3}}

Bugs {{#testopia: |run_id={{#var:eSDK}} |report_id=bugs |chart=bstatus,prio,severity |charttype=pie,pie,pie |showhide=true |hidden=true}}

Agenda {{#testopia: |run_id={{#var:eSDK}} |report_id=agenda |showhide=true |hidden=true |reportoptions=withdependency=false}}

BSP

genericx86: core-image-sato-sdk

Status {{#testopia: |run_id={{#var:genericx86}} |showhide=true |hidden=true |report_id=status |charttype=pie3}}

Bugs {{#testopia: |run_id={{#var:genericx86}} |report_id=bugs |chart=bstatus,prio,severity |charttype=pie,pie,pie |showhide=true |hidden=true}}

Agenda {{#testopia: |run_id={{#var:genericx86}} |report_id=agenda |showhide=true |hidden=true |reportoptions=withdependency=false}}

genericx86_64: core-image-sato-sdk

Status {{#testopia: |run_id={{#var:genericx86_64}} |showhide=true |hidden=true |report_id=status |charttype=pie3}}

Bugs {{#testopia: |run_id={{#var:genericx86_64}} |report_id=bugs |chart=bstatus,prio,severity |charttype=pie,pie,pie |showhide=true |hidden=true}}

Agenda {{#testopia: |run_id={{#var:genericx86_64}} |report_id=agenda |showhide=true |hidden=true |reportoptions=withdependency=false}}

Beaglebone Black: core-image-sato-sdk

Status {{#testopia: |run_id={{#var:beaglebone_black}} |showhide=true |hidden=true |report_id=status |charttype=pie3}}

Bugs {{#testopia: |run_id={{#var:beaglebone_black}} |report_id=bugs |chart=bstatus,prio,severity |charttype=pie,pie,pie |showhide=true |hidden=true}}

Agenda {{#testopia: |run_id={{#var:beaglebone_black}} |report_id=agenda |showhide=true |hidden=true |reportoptions=withdependency=false}}

EdgeRouter: core-image-sato-sdk

Status {{#testopia: |run_id={{#var:edgerouter}} |showhide=true |hidden=true |report_id=status |charttype=pie3}}

Bugs {{#testopia: |run_id={{#var:edgerouter}} |report_id=bugs |chart=bstatus,prio,severity |charttype=pie,pie,pie |showhide=true |hidden=true}}

Agenda {{#testopia: |run_id={{#var:edgerouter}} |report_id=agenda |showhide=true |hidden=true |reportoptions=withdependency=false}}

MPC8315e-rdb: core-image-sato-sdk

Status {{#testopia: |run_id={{#var:mpc8315e-rdb}} |showhide=true |hidden=true |report_id=status |charttype=pie3}}

Bugs {{#testopia: |run_id={{#var:mpc8315e-rdb}} |report_id=bugs |chart=bstatus,prio,severity |charttype=pie,pie,pie |showhide=true |hidden=true}}

Agenda {{#testopia: |run_id={{#var:mpc8315e-rdb}} |report_id=agenda |showhide=true |hidden=true |reportoptions=withdependency=false}}

P1022ds-rdb: core-image-sato-sdk

Status {{#testopia: |run_id={{#var:p1022ds-rdb}} |showhide=true |hidden=true |report_id=status |charttype=pie3}}

Bugs {{#testopia: |run_id={{#var:p1022ds-rdb}} |report_id=bugs |chart=bstatus,prio,severity |charttype=pie,pie,pie |showhide=true |hidden=true}}

Agenda {{#testopia: |run_id={{#var:p1022ds-rdb}} |report_id=agenda |showhide=true |hidden=true |reportoptions=withdependency=false}}

Qemu

qemux86: core-image-sato-sdk

Status {{#testopia: |run_id={{#var:qemux86}} |showhide=true |hidden=true |report_id=status |charttype=pie3}}

Bugs {{#testopia: |run_id={{#var:qemux86}} |report_id=bugs |chart=bstatus,prio,severity |charttype=pie,pie,pie |showhide=true |hidden=true}}

Agenda {{#testopia: |run_id={{#var:qemux86}} |report_id=agenda |showhide=true |hidden=true |reportoptions=withdependency=false}}

qemux86-64: core-image-sato-sdk

Status {{#testopia: |run_id={{#var:qemux86-64}} |showhide=true |hidden=true |report_id=status |charttype=pie3}}

Bugs {{#testopia: |run_id={{#var:qemux86-64}} |report_id=bugs |chart=bstatus,prio,severity |charttype=pie,pie,pie |showhide=true |hidden=true}}

Agenda {{#testopia: |run_id={{#var:qemux86-64}} |report_id=agenda |showhide=true |hidden=true |reportoptions=withdependency=false}}

qemuarm: core-image-sato-sdk

Status {{#testopia: |run_id={{#var:qemuarm}} |showhide=true |hidden=true |report_id=status |charttype=pie3}}

Bugs {{#testopia: |run_id={{#var:qemuarm}} |report_id=bugs |chart=bstatus,prio,severity |charttype=pie,pie,pie |showhide=true |hidden=true}}

Agenda {{#testopia: |run_id={{#var:qemuarm}} |report_id=agenda |showhide=true |hidden=true |reportoptions=withdependency=false}}

qemumips: core-image-sato-sdk

Status {{#testopia: |run_id={{#var:qemumips}} |showhide=true |hidden=true |report_id=status |charttype=pie3}}

Bugs {{#testopia: |run_id={{#var:qemumips}} |report_id=bugs |chart=bstatus,prio,severity |charttype=pie,pie,pie |showhide=true |hidden=true}}

Agenda {{#testopia: |run_id={{#var:qemumips}} |report_id=agenda |showhide=true |hidden=true |reportoptions=withdependency=false}}

qemuppc: core-image-sato-sdk

Status {{#testopia: |run_id={{#var:qemuppc}} |showhide=true |hidden=true |report_id=status |charttype=pie3}}

Bugs {{#testopia: |run_id={{#var:qemuppc}} |report_id=bugs |chart=bstatus,prio,severity |charttype=pie,pie,pie |showhide=true |hidden=true}}

Agenda {{#testopia: |run_id={{#var:qemuppc}} |report_id=agenda |showhide=true |hidden=true |reportoptions=withdependency=false}}

Toaster

Status {{#testopia: |run_id={{#var:toaster}} |showhide=true |hidden=true |report_id=status |charttype=pie3}}

Bugs {{#testopia: |run_id={{#var:toaster}} |report_id=bugs |chart=bstatus,prio,severity |charttype=pie,pie,pie |showhide=true |hidden=true}}

Agenda {{#testopia: |run_id={{#var:toaster}} |report_id=agenda |showhide=true |hidden=true |reportoptions=withdependency=false}}

Runtime Testing

Status {{#testopia: |run_id={{#var:runtime}} |showhide=true |hidden=true |report_id=status |charttype=pie3}}

Bugs {{#testopia: |run_id={{#var:runtime}} |report_id=bugs |chart=bstatus,prio,severity |charttype=pie,pie,pie |showhide=true |hidden=true}}

Agenda {{#testopia: |run_id={{#var:runtime}} |report_id=agenda |showhide=true |hidden=true |reportoptions=withdependency=false}}

Detailed result at: Build performance metrics

Build-appliance

Status {{#testopia: |run_id={{#var:build-appliance}} |showhide=true |hidden=true |report_id=status |charttype=pie3}}

Bugs {{#testopia: |run_id={{#var:build-appliance}} |report_id=bugs |chart=bstatus,prio,severity |charttype=pie,pie,pie |showhide=true |hidden=true}}

Agenda {{#testopia: |run_id={{#var:build-appliance}} |report_id=agenda |showhide=true |hidden=true |reportoptions=withdependency=false}}

ADT

ADT on: Fedora i686

Status {{#testopia: |run_id={{#var:adt_fedora_i686}} |showhide=true |hidden=true |report_id=status |charttype=pie3}}

Bugs {{#testopia: |run_id={{#var:adt_fedora_i686}} |report_id=bugs |chart=bstatus,prio,severity |charttype=pie,pie,pie |showhide=true |hidden=true}}

Agenda {{#testopia: |run_id={{#var:adt_fedora_i686}} |report_id=agenda |showhide=true |hidden=true |reportoptions=withdependency=false}}

ADT on: Fedora i686

Status {{#testopia: |run_id={{#var:adt_fedora_x86-64}} |showhide=true |hidden=true |report_id=status |charttype=pie3}}

Bugs {{#testopia: |run_id={{#var:adt_fedora_x86-64}} |report_id=bugs |chart=bstatus,prio,severity |charttype=pie,pie,pie |showhide=true |hidden=true}}

Agenda {{#testopia: |run_id={{#var:adt_fedora_x86-64}} |report_id=agenda |showhide=true |hidden=true |reportoptions=withdependency=false}}

ADT on: Ubuntu x86-64

Status {{#testopia: |run_id={{#var:adt_ubuntu_x86-64}} |showhide=true |hidden=true |report_id=status |charttype=pie3}}

Bugs {{#testopia: |run_id={{#var:adt_ubuntu_x86-64}} |report_id=bugs |chart=bstatus,prio,severity |charttype=pie,pie,pie |showhide=true |hidden=true}}

Agenda {{#testopia: |run_id={{#var:adt_ubuntu_x86-64}} |report_id=agenda |showhide=true |hidden=true |reportoptions=withdependency=false}}

Eclipse Plugin

Status {{#testopia: |run_id={{#var:eclipseplugin}} |showhide=true |hidden=true |report_id=status |charttype=pie3}}

Bugs {{#testopia: |run_id={{#var:eclipseplugin}} |report_id=bugs |chart=bstatus,prio,severity |charttype=pie,pie,pie |showhide=true |hidden=true}}

Agenda {{#testopia: |run_id={{#var:eclipseplugin}} |report_id=agenda |showhide=true |hidden=true |reportoptions=withdependency=false}}

Kernel

Status {{#testopia: |run_id={{#var:kernel}} |showhide=true |hidden=true |report_id=status |charttype=pie3}}

Bugs {{#testopia: |run_id={{#var:kernel}} |report_id=bugs |chart=bstatus,prio,severity |charttype=pie,pie,pie |showhide=true |hidden=true}}

Agenda {{#testopia: |run_id={{#var:kernel}} |report_id=agenda |showhide=true |hidden=true |reportoptions=withdependency=false}}