WW43 - 2017-10-23 - Full Test Cycle 2.4 RC3

From Yocto Project
Jump to: navigation, search

Contents

Testing data

 
 Test type: Full Pass Test
 Branch: rocko
 Build name: yocto-2.4.rc3
 poky commit: 65d23bd7986615fdfb0f1717b615534a2a14ab80
 Autobuilder images repository: http://autobuilder.yoctoproject.org/pub/releases/yocto-2.4.rc3
 Distributions tested: 

      CentOs 7.1
      Debian 8
      Debian 9
      Fedora 25
      Fedora 26
      OpenSuse 422
      Ubuntu 16.04
      Ubuntu 17.04

  
Testing Summary Report
Test RunTest PlanEnvironmentPassedOther issuesFailedFailing bugsBlockedBlocking bugsComplete
8391 8392 eSDK: master branchDebian 8100.0%none0.0%none0.0%none100.0%
8388 CROPS: master branchCentos 7.1100.0%none0.0%none0.0%none100.0%
8410 CROPS: master branchuser-preference (any supported environment)100.0%none0.0%none0.0%none100.0%
8348 8349 ADT: master branchFedora 26 i686100.0%11994 0.0%none0.0%none100.0%
8350 8351 ADT: master branchUbuntu 16.04 x86_64100.0%none0.0%none0.0%none100.0%
8394 Kernel: master branchSuSE 42.2100.0%none0.0%none0.0%none100.0%
8390 Eclipse-Plugin: master branchNeon - Fedora 2485.7%none14.3%12270 0.0%none100.0%
8389 Eclipse-Plugin: master branchOxygen - Fedora2685.7%none14.3%12270 0.0%none100.0%
8395 Meta-yocto: master branchUbuntu 16.04 x86_64100.0%none0.0%none0.0%none100.0%
8405 Build-Appliance: master branchVMWare Player on Ubuntu100.0%none0.0%none0.0%none100.0%
8406 8407 Toaster: master branchDebian 4.3.0-0.bpo.1-amd6496.2%12259 12260 3.8%12261 12262 12263 0.0%none100.0%
8396 OE-Core: master branchCentOS 7.1 x86_64100.0%none0.0%none0.0%none100.0%
8397 OE-Core: master branchDebian 8 x86_64100.0%none0.0%none0.0%none100.0%
8411 OE-Core: master branchDebian 9 x86_64100.0%none0.0%none0.0%none100.0%
8398 OE-Core: master branchFedora 25 x86_64100.0%none0.0%none0.0%none100.0%
8404 OE-Core: master branchFedora 26 x86_6499.6%none0.4%12185 0.0%none100.0%
8400 OE-Core: master branchOpenSuse 42.3 x86_64100.0%none0.0%none0.0%none100.0%
8402 OE-Core: master branchUbuntu 16.04 x86_64100.0%none0.0%none0.0%none100.0%
8403 OE-Core: master branchUbuntu 17.04 x86-64100.0%none0.0%none0.0%none100.0%
8352 BitBake: master branchDebian 8100.0%none0.0%none0.0%none100.0%
8355 8383 BSP/QEMU: master branchBeaglebone Black98.5%none1.5%12131 0.0%none100.0%
8356 8384 BSP/QEMU: master branchEdgeRouter100.0%none0.0%none0.0%none100.0%
8376 BSP/QEMU: master branchgenericx86 - on MMAX32bit100.0%none0.0%none0.0%none100.0%
8377 BSP/QEMU: master branchgenericx86-64 on MMAX64100.0%none0.0%none0.0%none100.0%
8378 BSP/QEMU: master branchgenericx86-64 on NUC97.8%none2.2%12187 0.0%none100.0%
8386 BSP/QEMU: master branchgenericx86-64-lsb on MMAX64100.0%none0.0%none0.0%none100.0%
8385 BSP/QEMU: master branchgenericx86-64-lsb on NUC100.0%none0.0%none0.0%none100.0%
8379 BSP/QEMU: master branchgenericx86-64-WIC on MMAX64100.0%none0.0%none0.0%none100.0%
8380 BSP/QEMU: master branchgenericx86-64-WIC on NUC97.8%none2.2%12187 0.0%none100.0%
8354 BSP/QEMU: master branchGenericx86-64_NUC95.1%none2.4%12187 2.4%12187 100.0%
8387 BSP/QEMU: master branchgenericx86-lsb on MMAX32bit100.0%none0.0%none0.0%none100.0%
8381 BSP/QEMU: master branchgenericx86-WIC on MMAX32100.0%none0.0%none0.0%none100.0%
8353 BSP/QEMU: master branchGenericx86_MMAX32100.0%none0.0%none0.0%none100.0%
8357 8382 BSP/QEMU: master branchMPC8315e-rdb100.0%none0.0%none0.0%none100.0%
8358 8365 BSP/QEMU: master branchqemu-x86100.0%none0.0%none0.0%none100.0%
8359 8366 BSP/QEMU: master branchqemuarm100.0%none0.0%none0.0%none100.0%
8360 8367 BSP/QEMU: master branchqemuarm64100.0%none0.0%none0.0%none100.0%
8361 8368 BSP/QEMU: master branchqemumips100.0%none0.0%none0.0%none100.0%
8362 8369 BSP/QEMU: master branchqemumips64100.0%none0.0%none0.0%none100.0%
8363 8370 BSP/QEMU: master branchqemuppc100.0%none0.0%none0.0%none100.0%
8364 BSP/QEMU: master branchqemux86-64100.0%none0.0%none0.0%none100.0%
8371 BSP/QEMU: master branchqemux86-64,100.0%none0.0%none0.0%none100.0%
8372 BSP/QEMU: master branchqemux86-64-WIC,100.0%none0.0%none0.0%none100.0%
8374 BSP/QEMU: master branchqemux86-lsb100.0%none0.0%none0.0%none100.0%
8373 BSP/QEMU: master branchqemux86-WIC100.0%none0.0%none0.0%none100.0%
8375 BSP/QEMU: master branchqemux86_64-lsb100.0%none0.0%none0.0%none100.0%
8393 Runtime: master branchVMWare Player on Ubuntu88.9%none11.1%12240 0.0%none100.0%
8408 Package-Management-Issues: master branchAny Supported Distro100.0%none0.0%none0.0%none100.0%
Total: 48


Bug Status


Test Case Status Report
StatusCount
BLOCKED1
FAILED11
PASSED3826
PAUSED1
RUNNING1
Total: 3840

Bugs Found during QA Test

New issues Found during QA Test & Community

IDSummary (6 tasks) PAssigneeQAMilestoneStatusResolutioncf_regression_typeCreated
OE-Core
12270[rocko] [Test Case 1534] runqemu fails using esdk in nfs modeMedium+Robert Yang 2.5 M2RESOLVEDFIXED---2017-10-24
Toaster
12259[toaster] when building 3 or more images left hand summary is not shown correctlyMediumDavid Reyna 3.99NEW---2017-10-20
12260[toaster]why arent the meta-qt3 and meta-qt4 present in the compatible layers for pokyMediumDavid Reyna 3.99NEW---2017-10-20
12261[Toaster] Importing layers, confirmation label is misleadingLowDavid Reyna 3.99NEW---2017-10-20
12262[Toaster] Layer details page: optional layers cant be blank when importing a layerMedium+David Reyna 3.99NEW---2017-10-20
12263[Toaster] project layers can be duplicatedMediumDavid Reyna 3.99NEW---2017-10-20


High and M+ issuesFound during QA Test (NOT NEW)

IDSummary (3 tasks) PAssigneeQAMilestoneStatusResolutioncf_regression_typeCreated
BSPs
12131Kernel traceback when reboot beaglebone blackMedium+Kevin HaoYi Zhao2.5 M3VERIFIEDFIXEDRegression (Used to work)2017-09-27
eSDK
12185devtool_modify_native test failure in ross/rockoMedium+Paul EggletonJavier Zepeda2.5 M1VERIFIEDFIXED---2017-10-04
General Runtime
11994Graphical qemu doesn't work over remote X on Fedora 26Medium+Mingli YuYi Zhao3.99RESOLVEDOBSOLETERegression (Used to work)2017-08-29

Other Bugs found during QA Test (NOT NEW)

---
12187not audio- play (wav) with HDMI, 2.4MediumAnuj Mittal 3.99RESOLVEDFIXED---2017-10-04

Current community Open Bugs - Medium+/High

Overview of the open bugs opened against current milestoneor release which are Medium+ or High and are not targetedfor the next milestones

IDPSummaryAssigneeQAMilestoneStatuscf_regression_typeCreated
  0      

Bug Trend

An overview of thisweek's Yocto Bug Trend charts can be accessed on thefollowing [1]

Build Performance Test

Detailed description on how performance test is run can be found on the Performance Test wiki. The below charts are displaying the build performance trend for 2.1M4 to 2.4 RC3. Performance execution has had 2 stages, one executed from 1.6 to 2.1 in RO and executed from 2.1 to 2.4 in GDC and are differences between HW used in both locations, hence the numbers are different.

Below chars are showing the latestperformance results taken from the current milestone.

Performance Charts

Results

Ww43-2k17 - table.png

For Bitbake core-image-sato withrm_work

Ww43-2k17 - rmwork.png

Building the Virtual kernel

Ww43-2k17 - kernel.png

Building the rootfs

Ww43-2k17 - rootfs.png

Building core-image-sato

Ww43-2k17 - sato.png

eSDK Deploy time

Ww43-2k17 - esdk.png

pTest for genericx86-64 on NUC

open bugs

IDSummary (27 tasks) PAssigneeQAMilestoneStatusResolutioncf_regression_typeCreated
Build Testing
12181QA: Archive ptest results in gitMedium+yeoh ee peng 3.99RESOLVEDFIXED---2017-10-04
12215ptest-runner libpcre one failedMediumJuro Bystricky 2.5RESOLVEDFIXED---2017-10-10
12229ptest-runner FAIL: run-lastpipe, 2.4_M4_rc2MediumChangqing Li 2.7RESOLVEDFIXED---2017-10-16
12230ptest-runner FAIL: run-new-exp, 2.4_M4_rc2MediumJuro Bystricky 2.5RESOLVEDWONTFIX---2017-10-16
12231ptest-bash cases failed in 2.4_M4_rc2, [FAIL: run-vredir]MediumChangqing Li 2.7RESOLVEDFIXED---2017-10-16
Manual Testing
11942Tidy up ptest compliance scriptsMedium+yeoh ee peng 3.99RESOLVEDFIXED---2017-08-16
OE-Core
6707Add ptest for curlMediumUnassignedApoorvFutureIN PROGRESS REVIEW---2014-09-11
10266Add ptest for gnutlsMediumUnassigned 3.99NEW---2016-09-13
11674core-image-sato-sdk-ptest build error for cpio imageMediumRoss Burton (Arm) 3.99NEW---2017-06-15
5605Add ptest for ffmpegLowTony Tascioglu FutureNEW---2013-11-27
10841pTest : Various test cases are failing on "dbus-test" moduleMediumRoss Burton (Arm) 2.5RESOLVEDDUPLICATE---2016-12-21
10953util-linux installs libtool wrapper scripts instead of binaries for ptestMediumJuro Bystricky 2.5 M1RESOLVEDFIXED---2017-01-23
11024lttng-tools ships libtool wrappers as ptest binariesMediumdengke 2.7RESOLVEDFIXED---2017-02-07
5695python: ptest cases failedMediumRoss Burton (Arm) 2.6RESOLVEDFIXED---2013-12-31
5697busybox: ptest cases failedMediumChangqing Li 2.7 M2RESOLVEDFIXED---2013-12-31
5698bash: ptest cases failedMediumYi Zhao FutureRESOLVEDDUPLICATE---2013-12-31
11998sed-ptest missingMediumRoss Burton (Arm) 2.5RESOLVEDFIXED---2017-08-29
Program Management
8064ptest fails tracker bugMediumRoss Burton (Arm) 2.5RESOLVEDOBSOLETE---2015-07-28
Recipe Reporting System (RRS)
8650RRS: track ptest-enabled and potentially ptest-enabled statusMediumUnassignedyeoh ee peng3.99NEW---2015-11-05
Runtime Testing
11811ptest-busybox cases failedMediumJuro Bystricky 2.5 M4RESOLVEDNOTABUG---2017-07-19
11813ptest- openssh cases failedMediumArmin Kuster 3.99RESOLVEDFIXED---2017-07-19
6737Extend automated ptest testing to work with opkgUndecidedSonia Delgadoyeoh ee pengFutureRESOLVEDOBSOLETE---2014-09-18
8471ptest - some valgrind ptests fail on 2.4 M1.rc1MediumMaxin B. John 3.99RESOLVEDDUPLICATERegression (Used to work)2015-10-09
8828pTest - lttng-tools tests failMediumRichard Purdieyeoh ee peng3.99RESOLVEDFIXEDRegression (Used to work)2015-12-17
8832pTest - bash tests fail for 2.1_M1.rc1MediumChangqing Li 2.7 M2RESOLVEDWORKSFORME---2015-12-17
9042pTest - strace fails on NUC 5425WYK for YP 2.1_M2.rc1MediumChangqing Li 2.7 M2RESOLVEDFIXEDRegression (Used to work)2016-02-03
11812ptest- libxml2 cases failedMediumJuro Bystricky 2.4VERIFIEDFIXED---2017-07-19

results

< Archive


Ran on a NUC and compared with 2.4_M4_rc2

FULL_LOG

Failures
 % increased
 % decreased
Module not executed
Results
Package Name Tests ran Passed Failed Passrate Last Passrate
acl 380 380 0 100.0 100.0
attr 143 143 0 100.0 100.0
bash 79 76 3 96.2 96.2
bluez5 7 7 0 100.0 100.0
busybox 518 509 9 98.26 98.26
bzip2 6 6 0 100.0 100.0
dbus-test 15 4 11 26.67 26.67
diffstat 440 440 0 100.0 100.0
diffutils 19 19 0 100.0 100.0
e2fsprogs 309 265 44 85.76 85.76
flex 114 114 0 100.0 100.0
gawk 298 297 1 99.66 99.66
gdbm 25 25 0 100.0 100.0
gdk-pixbuf 20 20 0 100.0 100.0
glib-2.0 253 253 0 100.0 100.0
gzip 18 18 0 100.0 100.0
kbd 7 7 0 100.0 100.0
libpcre 3 2 1 66.67 66.67
libxml2 133 133 0 100.0 100.0
lttng-tools 2057 1988 69 96.65 96.65
nettle 88 87 1 98.86 98.86
openssh 58 58 0 100.0 100.0
openssl 52 52 0 100.0 100.0
pango 12 12 0 100.0 100.0
perl 2287 2286 1 99.96 99.96
perl5 2287 2286 1 99.96 99.96
python 32234 32108 126 99.61 99.61
quilt 57 57 0 100.0 100.0
slang 97 97 0 100.0 100.0
strace 356 355 1 99.72 99.72
tcl 148 146 2 98.65 98.65
util-linux 123 102 21 82.93 82.93
valgrind 559 299 260 53.49 53.49
zlib 2 2 0 100.0 100.0


Compliance Report

POSIX results

General results for POSIX execution can be found POSIX_result and greater details on each architecture can be found POSIX_details

Bellow you will see the last three execution results for comparison:

Yocto 2.4 RC3 build

Status genericx86-64 beaglebone edgerouter mpc8315e-rdb
failures 46 45 47 45

Yocto 2.4 RC2 build

Status genericx86-64 beaglebone edgerouter mpc8315e-rdb
failures 46 45 45

Yocto 2.4 RC1 build

Status genericx86-64 beaglebone edgerouter mpc8315e-rdb
failures 46 45 47 45

LTP results

General results for LTP can be found here LTP_results and for greater information click on each architecture to be redirected to their details.

Yocto 2.4 RC3 build

Status genericx86-64 beaglebone edgerouter mpc8315e-rdb
total 1391 1353 1353 1353
failures 26 18 29

Yocto 2.4 RC2 build

Status genericx86-64 beaglebone edgerouter mpc8315e-rdb
total 1391 1353 1353 1353
failures 27 17 30


Yocto 2.4 RC1 build

Status genericx86-64 beaglebone edgerouter mpc8315e-rdb
total 1391 1353 1353 1353
failures 27 18 34 31

Detailed Report

ADT

Status

Bugs

Agenda

Status

Bugs

**No results**

Agenda

Status

Bugs

**No results**

Agenda

Status

Bugs

**No results**

Agenda

BitBake

Status

Bugs

**No results**

Agenda

BSPs

Status

Bugs

**No results**

Agenda

Status

Bugs

Agenda

Status

Bugs

Agenda

Status

Bugs

**No results**

Agenda

Status

Bugs

**No results**

Agenda

Status

Bugs

**No results**

Agenda

Status

Bugs

**No results**

Agenda

Status

Bugs

**No results**

Agenda

Status

Bugs

**No results**

Agenda

Status

Bugs

**No results**

Agenda

Status

Bugs

**No results**

Agenda

Status

Bugs

**No results**

Agenda

Status

Bugs

**No results**

Agenda

Status

Bugs

**No results**

Agenda

Status

Bugs

**No results**

Agenda

Status

Bugs

**No results**

Agenda

Status

Bugs

**No results**

Agenda

Status

Bugs

**No results**

Agenda

Status

Bugs

**No results**

Agenda

Status

Bugs

**No results**

Agenda

Status

Bugs

**No results**

Agenda

Status

Bugs

**No results**

Agenda

Status

Bugs

**No results**

Agenda

Status

Bugs

**No results**

Agenda

Status

Bugs

**No results**

Agenda

Status

Bugs

Agenda

Status

Bugs

**No results**

Agenda

Status

Bugs

Agenda

Status

Bugs

**No results**

Agenda

Status

Bugs

**No results**

Agenda

Status

Bugs

**No results**

Agenda

Status

Bugs

**No results**

Agenda

Status

Bugs

**No results**

Agenda

Status

Bugs

**No results**

Agenda

Status

Bugs

**No results**

Agenda

CROPS

Status

Bugs

**No results**

Agenda

Eclipse Plugin

Status

Bugs

Agenda

Status

Bugs

Agenda

eSDK

Status

Bugs

**No results**

Agenda

Status

Bugs

**No results**

Agenda

General Runtime

Status

Bugs

Agenda

Kernel

Status

Bugs

**No results**

Agenda

Meta-yocto

Status

Bugs

**No results**

Agenda

OE-Core

Status

Bugs

**No results**

Agenda

Status

Bugs

**No results**

Agenda

Status

Bugs

**No results**

Agenda