WW06 - 2017-02-03 - Full Test Cycle 2.3 M2 rc3

From Yocto Project
Revision as of 14:52, 8 February 2017 by Jose G Perez Carranza (talk | contribs) (Created page with "{| {{#vardefine:adt_fedora_i686| }} {{#vardefine:adt_fedora_x86-64| }} {{#vardefine:adt_ubuntu_x86-64| }} {{#vardefine:beaglebone_black| }} {{#vardefine:edgerouter|...")
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to navigationJump to search
{{#vardefine:adt_fedora_i686| }} {{#vardefine:adt_fedora_x86-64| }} {{#vardefine:adt_ubuntu_x86-64| }} {{#vardefine:beaglebone_black| }} {{#vardefine:edgerouter| }} {{#vardefine:genericx86| 6696, 6704, 6710, 6707}} {{#vardefine:genericx86_64| 6703, 6684, 6702, 6709, 6708, 6706, 6705}} {{#vardefine:mpc8315e-rdb| }} {{#vardefine:p1022ds-rdb| }} {{#vardefine:qemux86| 6685, 6689}} {{#vardefine:qemuarm| }} {{#vardefine:qemumips| }} {{#vardefine:qemuppc| }} {{#vardefine:qemux86-64| 6686, 6688}} {{#vardefine:eclipseplugin| 6715}} {{#vardefine:meta-yocto| 6682}} {{#vardefine:eSDK| 6711, 6712}} {{#vardefine:toaster| 6677, 6678}} {{#vardefine:oe-core| 6679, 6693, 6691, 6692, 6680, 6690, 6694, 6681, 6695}} {{#vardefine:distro| 6697}} {{#vardefine:bitbake| 6687}} {{#vardefine:runtime| }} {{#vardefine:build-appliance| }} {{#vardefine:kernel| 6713}}

Testing data

 Test type:  Full Test Cycle
 Branch: master
 Build name: 2.3 M2 rc1
 poky commit: 62d7d4130202d8ede16abf9e7d779361ca70847e
 Autobuilder images repository:  http://autobuilder.yoctoproject.org/pub/releases/yocto-2.3_M2.rc1/
 Distributions tested: 
 -Ubuntu 16.04
 -Ubuntu 14 04 
 -OpenSuse 421
 -Opensuse 13.2  
 -Debian 8
 -Centos 7
 -Fedora 23
 -Fedora 24

Test Run Summary Report

NOTE: There were missing Test Runs due the lack of resources for the execution.

{{#testopia: run_id=0, {{#var:bitbake}}, {{#var:oe-core}}, {{#var:meta-yocto}}, {{#var:kernel}}, {{#var:eSDK}}, {{#var:nuc}}, {{#var:atom-pc}}, {{#var:genericx86}}, {{#var:genericx86_64}}, {{#var:beaglebone_black}}, {{#var:edgerouter}}, {{#var:mpc8315e-rdb}}, {{#var:p1022ds-rdb}}, {{#var:qemux86}}, {{#var:qemux86-64}}, {{#var:qemuarm}}, {{#var:qemumips}}, {{#var:qemuppc}}, {{#var:runtime}}, {{#var:build-appliance}}, {{#var:toaster}}, {{#var:adt_opensuse_123_x86-64}}, {{#var:adt_fedora18_i686}}, {{#var:adt_fedora19_x86-64}}, {{#var:adt_fedora19_i686}}, {{#var:adt_fedora_i686}}, {{#var:adt_ubuntu_x86-64}}, {{#var:adt_ubuntu_1310_x86-64}}, {{#var:eclipseplugin}}, {{#var:distro}} report_id=concatstats}}


{{#testopia:

chart=bstatus report_id=bugs charttype=bar sortable=true nudechart=true chartpos=top

}}


{{#testopia:

run_id=0, {{#var:bitbake}}, {{#var:oe-core}}, {{#var:meta-yocto}}, {{#var:kernel}}, {{#var:eSDK}}, {{#var:nuc}}, {{#var:atom-pc}}, {{#var:genericx86}}, {{#var:genericx86_64}}, {{#var:beaglebone_black}}, {{#var:edgerouter}}, {{#var:mpc8315e-rdb}}, {{#var:p1022ds-rdb}}, {{#var:qemux86}}, {{#var:qemux86-64}}, {{#var:qemuarm}}, {{#var:qemumips}}, {{#var:qemuppc}}, {{#var:runtime}}, {{#var:build-appliance}}, {{#var:toaster}}, {{#var:adt_opensuse_123_x86-64}}, {{#var:adt_fedora18_i686}}, {{#var:adt_fedora19_x86-64}}, {{#var:adt_fedora19_i686}}, {{#var:adt_fedora_i686}}, {{#var:adt_ubuntu_x86-64}}, {{#var:eclipseplugin}}, {{#var:distro}} charttype=bar report_id=status}}


Bugs Found during QA Test

New issues Found during QA Test & Community

IDSummary (5 tasks) ⇒PAssigneeQAMilestoneStatusResolutioncf_regression_typeCreated
BitBake
10983[Test Case 810] Avoid cache reload for incremental development using a backgrounded bitbake serverMedium+Richard Purdie 2.3 M4RESOLVEDFIXED---2017-01-27
BSPs
10964testimage: Test suite is not running when images are downloaded from AutobuilderHighMariano LopezJose Perez C2.3 M2VERIFIEDFIXED---2017-01-25
Meta-yocto
10970pybootchartgui.py has not been migrated to python3MediumLeonardo Sandoval Gonzalez 2.3 M3RESOLVEDWONTFIX---2017-01-25
OE-Core
10968selftest: devtool and recipetool fails on load plugin test.Medium+Jose Perez C 2.3 M3VERIFIEDNOTABUG---2017-01-25
10990[crosstap] systemtap error on core-image-minimal with tools-profileMedium+Saul Wold 2.3 M3VERIFIEDFIXED---2017-01-30


High and M+ issues Found during QA Test (NOT NEW)

"None"


Other Bugs found during QA Test (NOT NEW)

---
10846Toaster: [Test Case 1095] Filter doesn't show a dialogueMediumDavid ReynaJose Perez C2.4VERIFIEDDUPLICATE---2016-12-22

Current community Open Bugs - Medium+/High

Overview of the open bugs opened against current milestone or release which are Medium+ or High and are not targeted for the next milestones

IDPSummaryAssigneeQAMilestoneStatuscf_regression_typeCreated
  0      

Bug Trend

An overview of this week's Yocto Bug Trend charts can be accessed on the following page

Build Performance Test

Detailed description on how performance test are run can be found on Performance Test wiki. The below charts are displaying the build performance trend for 2.1 M4 to 2.3 M1. Performance execution has 2 stages, one executed form 1.6 to 2.1 in RO and another one started on 2.1 from GDC, there are difference between HW used in both location hence the numbers are different.

Below chars are showing the latest performance results taken from current milestone

Performance Charts

NOTE: During the 2.3 M2 QA cycle period, Fedora machine was under debugging activities to investigate more on bug 10884 hence only results for Ubuntu machine are displayed on the charts.

Performance Charts

For Bitbake core-image-sato with rm_work

Ww05-2k17 - rmwork.png

Building the Virtual kernel

Ww05-2k17 - kernel.png

Building the rootfs

Ww05-2k17 - rootfs.png

Building core-image-sato

Ww05-2k17 - sato.png

eSDK Deploy time

Ww05-2k17 - esdk.png


pTest for genericx86-64 on NUC

open bugs

IDSummary (12 tasks) ⇒PAssigneeQAMilestoneStatusResolutioncf_regression_typeCreated
OE-Core
5695python: ptest cases failedMediumRoss Burton 2.6RESOLVEDFIXED---2013-12-31
10841pTest : Various test cases are failing on "dbus-test" moduleMediumRoss Burton 2.5RESOLVEDDUPLICATE---2016-12-21
5697busybox: ptest cases failedMediumChangqing Li 2.7 M2RESOLVEDFIXED---2013-12-31
10842testrunner test timeout can lead to garbage outputMediumAníbal Limón 2.3 M4VERIFIEDFIXED---2016-12-21
10840pTest- openssl: test cases "test_bn" and "test_cms" are failingMedium+Alexander Kanavin 2.4 M1VERIFIEDFIXED---2016-12-21
Runtime Testing
8832pTest - bash tests fail for 2.1_M1.rc1MediumChangqing Li 2.7 M2RESOLVEDWORKSFORME---2015-12-17
8831pTest - busybox tests fail for 2.1_M1.rc1Mediumdengke FutureRESOLVEDDUPLICATE---2015-12-17
8828pTest - lttng-tools tests failMediumRichard Purdieyeoh ee peng4.99RESOLVEDFIXEDRegression (Used to work)2015-12-17
9372Ptest for acl hang because of lack of two user bin and daemonMediumChen Qioscar lopez arandas2.4RESOLVEDFIXED---2016-04-01
8471ptest - some valgrind ptests fail on 2.4 M1.rc1MediumMaxin B. John 4.99RESOLVEDDUPLICATERegression (Used to work)2015-10-09
9055pTest - acl (NUC) failed tested on NUC 5425WYK for YP 2.1_M2.rc1MediumRobert YangFrancisco Pedraza2.4RESOLVEDFIXEDRegression (Used to work)2016-02-04
9042pTest - strace fails on NUC 5425WYK for YP 2.1_M2.rc1MediumChangqing Li 2.7 M2RESOLVEDFIXEDRegression (Used to work)2016-02-03

results

< Back
Ran on a NUC and compared with commit 573c646d4cc62dcd0c230381df4940bdf314d495

FULL LOG


Failures
% increased
% decreased


Results
Package Name
Tests ran
Passed
Failed
Passrate
Last Passrate
acl 380 354 26 93.20% 93.20%
attr 143 143 0 100.00% 100.00%
bash 76 72 4 94.70% 94.70%
bluez5 25 25 0 100.00% 100.00%
busybox 518 509 9 98.30% 98.60%
bzip2 6 6 0 100.00% 100.00%
dbus-test 15 4 11 26.70% 26.70%
diffstat 440 440 0 100.00% 100.00%
diffutils 17 17 0 100.00% 100.00%
e2fsprogs 1 1 0 100.00% 0.00%
gawk 298 297 1 99.70% 99.70%
gdbm 22 22 0 100.00% 100.00%
gdk-pixbuf 18 18 0 100.00% 100.00%
glib-2.0 250 250 0 100.00% 97.90%
kbd 7 7 0 100.00% 100.00%
libpcre 3 3 0 100.00% 100.00%
libxml2 85 82 3 96.50% 96.50%
lttng-tools 2057 1988 69 96.60% 99.80%
nettle 88 87 1 98.90% 98.90%
openssh 56 56 0 100.00% 100.00%
openssl 52 52 0 100.00% 96.20%
pango 12 12 0 100.00% 100.00%
perl 2286 2285 1 100.00% 100.00%
perl5 2286 2285 1 100.00% 100.00%
python 32058 31932 126 99.60% 99.60%
quilt 57 57 0 100.00% 98.20%
strace 278 273 5 98.20% 99.60%
tcl 148 147 1 99.30% 100.00%
util-linux 109 107 2 98.20% 98.50%
valgrind 553 2 551 0.40% 0.00%
zlib 2 2 0 100.00% 100.00%

Detailed Report

Distro Testing

Status {{#testopia: |run_id={{#var:distro}} |showhide=true |hidden=true |report_id=status |charttype=pie3}}

Bugs {{#testopia: |run_id={{#var:distro}} |report_id=bugs |chart=bstatus,prio,severity |charttype=pie,pie,pie |showhide=true |hidden=true}}

Agenda {{#testopia: |run_id={{#var:distro}} |report_id=agenda |showhide=true |hidden=true |reportoptions=withdependency=false}}

Core Build System

Bitbake

Status {{#testopia: |run_id={{#var:bitbake}} |showhide=true |hidden=true |report_id=status |charttype=pie3}}

Bugs {{#testopia: |run_id={{#var:bitbake}} |report_id=bugs |chart=bstatus,prio,severity |charttype=pie,pie,pie |showhide=true |hidden=true}}

Agenda {{#testopia: |run_id={{#var:bitbake}} |report_id=agenda |showhide=true |hidden=true |reportoptions=withdependency=false}}

Meta-Yocyo

Status {{#testopia: |run_id={{#var:meta-yocto}} |showhide=true |hidden=true |report_id=status |charttype=pie3}}

Bugs {{#testopia: |run_id={{#var:meta-yocto}} |report_id=bugs |chart=bstatus,prio,severity |charttype=pie,pie,pie |showhide=true |hidden=true}}

Agenda {{#testopia: |run_id={{#var:meta-yocto}} |report_id=agenda |showhide=true |hidden=true |reportoptions=withdependency=false}}

OE-Core

Status {{#testopia: |run_id={{#var:oe-core}} |showhide=true |hidden=true |report_id=status |charttype=pie3}}

Bugs {{#testopia: |run_id={{#var:oe-core}} |report_id=bugs |chart=bstatus,prio,severity |charttype=pie,pie,pie |showhide=true |hidden=true}}

Agenda {{#testopia: |run_id={{#var:oe-core}} |report_id=agenda |showhide=true |hidden=true |reportoptions=withdependency=false}}

eSDK

Status {{#testopia: |run_id={{#var:eSDK}} |showhide=true |hidden=true |report_id=status |charttype=pie3}}

Bugs {{#testopia: |run_id={{#var:eSDK}} |report_id=bugs |chart=bstatus,prio,severity |charttype=pie,pie,pie |showhide=true |hidden=true}}

Agenda {{#testopia: |run_id={{#var:eSDK}} |report_id=agenda |showhide=true |hidden=true |reportoptions=withdependency=false}}

BSP

genericx86: core-image-sato-sdk

Status {{#testopia: |run_id={{#var:genericx86}} |showhide=true |hidden=true |report_id=status |charttype=pie3}}

Bugs {{#testopia: |run_id={{#var:genericx86}} |report_id=bugs |chart=bstatus,prio,severity |charttype=pie,pie,pie |showhide=true |hidden=true}}

Agenda {{#testopia: |run_id={{#var:genericx86}} |report_id=agenda |showhide=true |hidden=true |reportoptions=withdependency=false}}

genericx86_64: core-image-sato-sdk

Status {{#testopia: |run_id={{#var:genericx86_64}} |showhide=true |hidden=true |report_id=status |charttype=pie3}}

Bugs {{#testopia: |run_id={{#var:genericx86_64}} |report_id=bugs |chart=bstatus,prio,severity |charttype=pie,pie,pie |showhide=true |hidden=true}}

Agenda {{#testopia: |run_id={{#var:genericx86_64}} |report_id=agenda |showhide=true |hidden=true |reportoptions=withdependency=false}}

Beaglebone Black: core-image-sato-sdk

Status {{#testopia: |run_id={{#var:beaglebone_black}} |showhide=true |hidden=true |report_id=status |charttype=pie3}}

Bugs {{#testopia: |run_id={{#var:beaglebone_black}} |report_id=bugs |chart=bstatus,prio,severity |charttype=pie,pie,pie |showhide=true |hidden=true}}

Agenda {{#testopia: |run_id={{#var:beaglebone_black}} |report_id=agenda |showhide=true |hidden=true |reportoptions=withdependency=false}}

EdgeRouter: core-image-sato-sdk

Status {{#testopia: |run_id={{#var:edgerouter}} |showhide=true |hidden=true |report_id=status |charttype=pie3}}

Bugs {{#testopia: |run_id={{#var:edgerouter}} |report_id=bugs |chart=bstatus,prio,severity |charttype=pie,pie,pie |showhide=true |hidden=true}}

Agenda {{#testopia: |run_id={{#var:edgerouter}} |report_id=agenda |showhide=true |hidden=true |reportoptions=withdependency=false}}

MPC8315e-rdb: core-image-sato-sdk

Status {{#testopia: |run_id={{#var:mpc8315e-rdb}} |showhide=true |hidden=true |report_id=status |charttype=pie3}}

Bugs {{#testopia: |run_id={{#var:mpc8315e-rdb}} |report_id=bugs |chart=bstatus,prio,severity |charttype=pie,pie,pie |showhide=true |hidden=true}}

Agenda {{#testopia: |run_id={{#var:mpc8315e-rdb}} |report_id=agenda |showhide=true |hidden=true |reportoptions=withdependency=false}}

P1022ds-rdb: core-image-sato-sdk

Status {{#testopia: |run_id={{#var:p1022ds-rdb}} |showhide=true |hidden=true |report_id=status |charttype=pie3}}

Bugs {{#testopia: |run_id={{#var:p1022ds-rdb}} |report_id=bugs |chart=bstatus,prio,severity |charttype=pie,pie,pie |showhide=true |hidden=true}}

Agenda {{#testopia: |run_id={{#var:p1022ds-rdb}} |report_id=agenda |showhide=true |hidden=true |reportoptions=withdependency=false}}


Qemu

qemux86: core-image-sato-sdk

Status {{#testopia: |run_id={{#var:qemux86}} |showhide=true |hidden=true |report_id=status |charttype=pie3}}

Bugs {{#testopia: |run_id={{#var:qemux86}} |report_id=bugs |chart=bstatus,prio,severity |charttype=pie,pie,pie |showhide=true |hidden=true}}

Agenda {{#testopia: |run_id={{#var:qemux86}} |report_id=agenda |showhide=true |hidden=true |reportoptions=withdependency=false}}

qemux86-64: core-image-sato-sdk

Status {{#testopia: |run_id={{#var:qemux86-64}} |showhide=true |hidden=true |report_id=status |charttype=pie3}}

Bugs {{#testopia: |run_id={{#var:qemux86-64}} |report_id=bugs |chart=bstatus,prio,severity |charttype=pie,pie,pie |showhide=true |hidden=true}}

Agenda {{#testopia: |run_id={{#var:qemux86-64}} |report_id=agenda |showhide=true |hidden=true |reportoptions=withdependency=false}}

qemuarm: core-image-sato-sdk

Status {{#testopia: |run_id={{#var:qemuarm}} |showhide=true |hidden=true |report_id=status |charttype=pie3}}

Bugs {{#testopia: |run_id={{#var:qemuarm}} |report_id=bugs |chart=bstatus,prio,severity |charttype=pie,pie,pie |showhide=true |hidden=true}}

Agenda {{#testopia: |run_id={{#var:qemuarm}} |report_id=agenda |showhide=true |hidden=true |reportoptions=withdependency=false}}

qemumips: core-image-sato-sdk

Status {{#testopia: |run_id={{#var:qemumips}} |showhide=true |hidden=true |report_id=status |charttype=pie3}}

Bugs {{#testopia: |run_id={{#var:qemumips}} |report_id=bugs |chart=bstatus,prio,severity |charttype=pie,pie,pie |showhide=true |hidden=true}}

Agenda {{#testopia: |run_id={{#var:qemumips}} |report_id=agenda |showhide=true |hidden=true |reportoptions=withdependency=false}}

qemuppc: core-image-sato-sdk

Status {{#testopia: |run_id={{#var:qemuppc}} |showhide=true |hidden=true |report_id=status |charttype=pie3}}

Bugs {{#testopia: |run_id={{#var:qemuppc}} |report_id=bugs |chart=bstatus,prio,severity |charttype=pie,pie,pie |showhide=true |hidden=true}}

Agenda {{#testopia: |run_id={{#var:qemuppc}} |report_id=agenda |showhide=true |hidden=true |reportoptions=withdependency=false}}

Toaster

Status {{#testopia: |run_id={{#var:toaster}} |showhide=true |hidden=true |report_id=status |charttype=pie3}}

Bugs {{#testopia: |run_id={{#var:toaster}} |report_id=bugs |chart=bstatus,prio,severity |charttype=pie,pie,pie |showhide=true |hidden=true}}

Agenda {{#testopia: |run_id={{#var:toaster}} |report_id=agenda |showhide=true |hidden=true |reportoptions=withdependency=false}}

Runtime Testing

Status {{#testopia: |run_id={{#var:runtime}} |showhide=true |hidden=true |report_id=status |charttype=pie3}}

Bugs {{#testopia: |run_id={{#var:runtime}} |report_id=bugs |chart=bstatus,prio,severity |charttype=pie,pie,pie |showhide=true |hidden=true}}

Agenda {{#testopia: |run_id={{#var:runtime}} |report_id=agenda |showhide=true |hidden=true |reportoptions=withdependency=false}}

Detailed result at: Build performance metrics

Build-appliance

Status {{#testopia: |run_id={{#var:build-appliance}} |showhide=true |hidden=true |report_id=status |charttype=pie3}}

Bugs {{#testopia: |run_id={{#var:build-appliance}} |report_id=bugs |chart=bstatus,prio,severity |charttype=pie,pie,pie |showhide=true |hidden=true}}

Agenda {{#testopia: |run_id={{#var:build-appliance}} |report_id=agenda |showhide=true |hidden=true |reportoptions=withdependency=false}}

ADT

ADT on: Fedora i686

Status {{#testopia: |run_id={{#var:adt_fedora_i686}} |showhide=true |hidden=true |report_id=status |charttype=pie3}}

Bugs {{#testopia: |run_id={{#var:adt_fedora_i686}} |report_id=bugs |chart=bstatus,prio,severity |charttype=pie,pie,pie |showhide=true |hidden=true}}

Agenda {{#testopia: |run_id={{#var:adt_fedora_i686}} |report_id=agenda |showhide=true |hidden=true |reportoptions=withdependency=false}}

ADT on: Fedora i686

Status {{#testopia: |run_id={{#var:adt_fedora_x86-64}} |showhide=true |hidden=true |report_id=status |charttype=pie3}}

Bugs {{#testopia: |run_id={{#var:adt_fedora_x86-64}} |report_id=bugs |chart=bstatus,prio,severity |charttype=pie,pie,pie |showhide=true |hidden=true}}

Agenda {{#testopia: |run_id={{#var:adt_fedora_x86-64}} |report_id=agenda |showhide=true |hidden=true |reportoptions=withdependency=false}}

ADT on: Ubuntu x86-64

Status {{#testopia: |run_id={{#var:adt_ubuntu_x86-64}} |showhide=true |hidden=true |report_id=status |charttype=pie3}}

Bugs {{#testopia: |run_id={{#var:adt_ubuntu_x86-64}} |report_id=bugs |chart=bstatus,prio,severity |charttype=pie,pie,pie |showhide=true |hidden=true}}

Agenda {{#testopia: |run_id={{#var:adt_ubuntu_x86-64}} |report_id=agenda |showhide=true |hidden=true |reportoptions=withdependency=false}}

Eclipse Plugin

Status {{#testopia: |run_id={{#var:eclipseplugin}} |showhide=true |hidden=true |report_id=status |charttype=pie3}}

Bugs {{#testopia: |run_id={{#var:eclipseplugin}} |report_id=bugs |chart=bstatus,prio,severity |charttype=pie,pie,pie |showhide=true |hidden=true}}

Agenda {{#testopia: |run_id={{#var:eclipseplugin}} |report_id=agenda |showhide=true |hidden=true |reportoptions=withdependency=false}}

Kernel

Status {{#testopia: |run_id={{#var:kernel}} |showhide=true |hidden=true |report_id=status |charttype=pie3}}

Bugs {{#testopia: |run_id={{#var:kernel}} |report_id=bugs |chart=bstatus,prio,severity |charttype=pie,pie,pie |showhide=true |hidden=true}}

Agenda {{#testopia: |run_id={{#var:kernel}} |report_id=agenda |showhide=true |hidden=true |reportoptions=withdependency=false}}

Retrieved from "https://wiki.yoctoproject.org/wiki/index.php?title=WW06_-_2017-02-03_-_Full_Test_Cycle_2.3_M2_rc3&oldid=23726"