WW06 - 2018-02-08 - Full Point Release Test Cycle 2.2.3 rc2: Difference between revisions
No edit summary |
No edit summary |
||
(6 intermediate revisions by the same user not shown) | |||
Line 100: | Line 100: | ||
|sort=ID | |sort=ID | ||
|group=product | |group=product | ||
|id= | |id=12521,12519,12520 | ||
}} | }} | ||
<br> | <br> | ||
Line 119: | Line 119: | ||
|sort=priority | |sort=priority | ||
|group=cf_regression_type,product | |group=cf_regression_type,product | ||
|id= | |id=12240 | ||
}} | }} | ||
Line 142: | Line 142: | ||
= Build Performance Test = | = Build Performance Test = | ||
Detailed description on how performance test is run canbe found on the [[Performance Test]] wiki. | Detailed description on how performance test is run canbe found on the [[Performance Test]] wiki. | ||
The belowcharts are displaying the build performance trend for 2.1M4 to 2.3 M3. Performance execution has had 2 stages, one executed from 1.6 to 2.1 in RO and executed from 1 | The belowcharts are displaying the build performance trend for 2.1M4 to 2.3 M3. Performance execution has had 2 stages, one executed from 1.6 to 2.1 in RO and executed from from 2.1 to 2.5 m1 rc3 in GDC and are differences between HW used in bothlocations, hence the numbers are different. | ||
* [https://wiki.yoctoproject.org/charts/perf_milestone/performance_test.html RO Charts] | * [https://wiki.yoctoproject.org/charts/perf_milestone/performance_test.html RO Charts] | ||
Line 150: | Line 150: | ||
== Performance Charts== | == Performance Charts== | ||
Note: Longer than expected eSDK time on Fedora machine were identified to be related to the machine itself. | |||
Will rerun performance tests on new machines from Linux Foundation. | |||
[[File:Ww06-2k18 - table.png | 600px]] | |||
[[File: | |||
=pTest for genericx86-64 on NUC= | =pTest for genericx86-64 on NUC= | ||
Line 185: | Line 169: | ||
== results == | == results == | ||
[https://wiki.yoctoproject.org/wiki/Ptest/archive < Archive] | |||
<br />Ran on a NUC and compared with[[Ptest 60402978fe3648bf560b3386c6e9dd661cdf2083| 2.2.2.rc2]] | |||
[https://wiki.yoctoproject.org/wiki/images/d/df/Ptest_2.2.3_rc2.txt < FULL_LOG] | |||
{| style="border-spacing:0;" | |||
| style="background-color:#ff8080;border:0.05pt solid #000000;padding:0.0382in;"| Failures | |||
|- | |||
| style="background-color:#99ff66;border-top:none;border-bottom:0.05ptsolid #000000;border-left:0.05pt solid #000000;border-right:0.05ptsolid #000000;padding:0.0201in;"| % increased | |||
|- | |||
| style="background-color:#c5000b;border-top:none;border-bottom:0.05ptsolid #000000;border-left:0.05pt solid #000000;border-right:0.05ptsolid #000000;padding:0.0201in;color:#ffffff"| % decreased | |||
|- | |||
| style="background-color:gray;border-top:none;border-bottom:0.05ptsolid #000000;border-left:0.05pt solid #000000;border-right:0.05ptsolid #000000;padding:0.0201in;color:#000000"| Module not executed | |||
|} | |||
{| class="wikitable" | |||
|+Results | |||
|- | |||
| align=center style="background-color:#ccffff;border-top:0.05pt solid #000000;border-bottom:0.5pt solid#000000;border-left:0.05pt solid #000000;border-right:0.5ptsolid #000000;padding:0.0201in;"| '''Package Name''' | |||
| align=center style="background-color:#ccffff;border-top:0.05pt solid #000000;border-bottom:0.5pt solid#000000;border-left:0.05pt solid #000000;border-right:0.5ptsolid #000000;padding:0.0201in;"| '''Tests ran''' | |||
| align=center style="background-color:#ccffff;border-top:0.05pt solid #000000;border-bottom:0.5pt solid#000000;border-left:0.05pt solid #000000;border-right:0.5ptsolid #000000;padding:0.0201in;"| '''Passed''' | |||
| align=center style="background-color:#ccffff;border-top:0.05pt solid #000000;border-bottom:0.5pt solid#000000;border-left:0.05pt solid #000000;border-right:0.5ptsolid #000000;padding:0.0201in;"| '''Failed''' | |||
| align=center style="background-color:#ccffff;border-top:0.05pt solid #000000;border-bottom:0.5pt solid#000000;border-left:0.05pt solid #000000;border-right:0.5ptsolid #000000;padding:0.0201in;"| '''Passrate''' | |||
| align=center style="background-color:#ccffff;border-top:0.05pt solid #000000;border-bottom:0.5pt solid#000000;border-left:0.05pt solid #000000;border-right:0.5ptsolid #000000;padding:0.0201in;"| '''Last Passrate''' | |||
|- | |||
| align=left style="background-color:trasnparentborder-color:#000000"| acl | |||
| align=center style="background-color:transparent;border-color:#000000"|380 | |||
| align=center style="background-color:transparent;border-color:#000000"|354 | |||
| align=center style="background-color:#ff8080;border-color:#000000"|26 | |||
| align=center style="background-color:transparent;border-color:#000000;color:#000000"|93.16 | |||
| align=center style="background-color:transparent;border-color:#000000;color:#000000"|93.16 | |||
|- | |||
| align=left style="background-color:trasnparentborder-color:#000000"| attr | |||
| align=center style="background-color:transparent;border-color:#000000"|143 | |||
| align=center style="background-color:transparent;border-color:#000000"|143 | |||
| align=center style="background-color:transparent;border-color:#000000"|0 | |||
| align=center style="background-color:transparent;border-color:#000000;color:#000000"|100.0 | |||
| align=center style="background-color:transparent;border-color:#000000;color:#000000"|100.0 | |||
|- | |||
| align=left style="background-color:trasnparentborder-color:#000000"| bash | |||
| align=center style="background-color:transparent;border-color:#000000"|76 | |||
| align=center style="background-color:transparent;border-color:#000000"|72 | |||
| align=center style="background-color:#ff8080;border-color:#000000"|4 | |||
| align=center style="background-color:transparent;border-color:#000000;color:#000000"|94.74 | |||
| align=center style="background-color:transparent;border-color:#000000;color:#000000"|94.74 | |||
|- | |||
| align=left style="background-color:trasnparentborder-color:#000000"| bluez5 | |||
| align=center style="background-color:transparent;border-color:#000000"|25 | |||
| align=center style="background-color:transparent;border-color:#000000"|25 | |||
| align=center style="background-color:transparent;border-color:#000000"|0 | |||
| align=center style="background-color:transparent;border-color:#000000;color:#000000"|100.0 | |||
| align=center style="background-color:transparent;border-color:#000000;color:#000000"|100.0 | |||
|- | |||
| align=left style="background-color:trasnparentborder-color:#000000"| busybox | |||
| align=center style="background-color:transparent;border-color:#000000"|518 | |||
| align=center style="background-color:transparent;border-color:#000000"|509 | |||
| align=center style="background-color:#ff8080;border-color:#000000"|9 | |||
| align=center style="background-color:transparent;border-color:#000000;color:#000000"|98.26 | |||
| align=center style="background-color:transparent;border-color:#000000;color:#000000"|98.26 | |||
|- | |||
| align=left style="background-color:trasnparentborder-color:#000000"| bzip2 | |||
| align=center style="background-color:transparent;border-color:#000000"|6 | |||
| align=center style="background-color:transparent;border-color:#000000"|6 | |||
| align=center style="background-color:transparent;border-color:#000000"|0 | |||
| align=center style="background-color:transparent;border-color:#000000;color:#000000"|100.0 | |||
| align=center style="background-color:transparent;border-color:#000000;color:#000000"|100.0 | |||
|- | |||
| align=left style="background-color:trasnparentborder-color:#000000"| dbus-test | |||
| align=center style="background-color:transparent;border-color:#000000"|15 | |||
| align=center style="background-color:transparent;border-color:#000000"|4 | |||
| align=center style="background-color:#ff8080;border-color:#000000"|11 | |||
| align=center style="background-color:transparent;border-color:#000000;color:#000000"|26.67 | |||
| align=center style="background-color:transparent;border-color:#000000;color:#000000"|26.67 | |||
|- | |||
| align=left style="background-color:trasnparentborder-color:#000000"| diffstat | |||
| align=center style="background-color:transparent;border-color:#000000"|440 | |||
| align=center style="background-color:transparent;border-color:#000000"|440 | |||
| align=center style="background-color:transparent;border-color:#000000"|0 | |||
| align=center style="background-color:transparent;border-color:#000000;color:#000000"|100.0 | |||
| align=center style="background-color:transparent;border-color:#000000;color:#000000"|100.0 | |||
|- | |||
| align=left style="background-color:trasnparentborder-color:#000000"| diffutils | |||
| align=center style="background-color:transparent;border-color:#000000"|16 | |||
| align=center style="background-color:transparent;border-color:#000000"|16 | |||
| align=center style="background-color:transparent;border-color:#000000"|0 | |||
| align=center style="background-color:transparent;border-color:#000000;color:#000000"|100.0 | |||
| align=center style="background-color:transparent;border-color:#000000;color:#000000"|100.0 | |||
|- | |||
| align=left style="background-color:trasnparentborder-color:#000000"| e2fsprogs | |||
| align=center style="background-color:transparent;border-color:#000000"|1 | |||
| align=center style="background-color:transparent;border-color:#000000"|0 | |||
| align=center style="background-color:#ff8080;border-color:#000000"|1 | |||
| align=center style="background-color:transparent;border-color:#000000;color:#000000"|0.0 | |||
| align=center style="background-color:transparent;border-color:#000000;color:#000000"|0.0 | |||
|- | |||
| align=left style="background-color:trasnparentborder-color:#000000"| gawk | |||
| align=center style="background-color:transparent;border-color:#000000"|284 | |||
| align=center style="background-color:transparent;border-color:#000000"|283 | |||
| align=center style="background-color:#ff8080;border-color:#000000"|1 | |||
| align=center style="background-color:transparent;border-color:#000000;color:#000000"|99.65 | |||
| align=center style="background-color:transparent;border-color:#000000;color:#000000"|99.65 | |||
|- | |||
| align=left style="background-color:trasnparentborder-color:#000000"| gdbm | |||
| align=center style="background-color:transparent;border-color:#000000"|22 | |||
| align=center style="background-color:transparent;border-color:#000000"|22 | |||
| align=center style="background-color:transparent;border-color:#000000"|0 | |||
| align=center style="background-color:transparent;border-color:#000000;color:#000000"|100.0 | |||
| align=center style="background-color:transparent;border-color:#000000;color:#000000"|100.0 | |||
|- | |||
| align=left style="background-color:trasnparentborder-color:#000000"| gdk-pixbuf | |||
| align=center style="background-color:transparent;border-color:#000000"|17 | |||
| align=center style="background-color:transparent;border-color:#000000"|17 | |||
| align=center style="background-color:transparent;border-color:#000000"|0 | |||
| align=center style="background-color:transparent;border-color:#000000;color:#000000"|100.0 | |||
| align=center style="background-color:transparent;border-color:#000000;color:#000000"|100.0 | |||
|- | |||
| align=left style="background-color:trasnparentborder-color:#000000"| glib-2.0 | |||
| align=center style="background-color:transparent;border-color:#000000"|249 | |||
| align=center style="background-color:transparent;border-color:#000000"|247 | |||
| align=center style="background-color:#ff8080;border-color:#000000"|2 | |||
| align=center style="background-color:transparent;border-color:#000000;color:#000000"|99.2 | |||
| align=center style="background-color:transparent;border-color:#000000;color:#000000"|99.2 | |||
|- | |||
| align=left style="background-color:trasnparentborder-color:#000000"| kbd | |||
| align=center style="background-color:transparent;border-color:#000000"|7 | |||
| align=center style="background-color:transparent;border-color:#000000"|7 | |||
| align=center style="background-color:transparent;border-color:#000000"|0 | |||
| align=center style="background-color:transparent;border-color:#000000;color:#000000"|100.0 | |||
| align=center style="background-color:transparent;border-color:#000000;color:#000000"|100.0 | |||
|- | |||
| align=left style="background-color:trasnparentborder-color:#000000"| libpcre | |||
| align=center style="background-color:transparent;border-color:#000000"|3 | |||
| align=center style="background-color:transparent;border-color:#000000"|3 | |||
| align=center style="background-color:transparent;border-color:#000000"|0 | |||
| align=center style="background-color:transparent;border-color:#000000;color:#000000"|100.0 | |||
| align=center style="background-color:transparent;border-color:#000000;color:#000000"|100.0 | |||
|- | |||
| align=left style="background-color:trasnparentborder-color:#000000"| libxml2 | |||
| align=center style="background-color:transparent;border-color:#000000"|85 | |||
| align=center style="background-color:transparent;border-color:#000000"|82 | |||
| align=center style="background-color:#ff8080;border-color:#000000"|3 | |||
| align=center style="background-color:transparent;border-color:#000000;color:#000000"|96.47 | |||
| align=center style="background-color:transparent;border-color:#000000;color:#000000"|96.47 | |||
|- | |||
| align=left style="background-color:trasnparentborder-color:#000000"| lttng-tools | |||
| align=center style="background-color:transparent;border-color:#000000"|2467 | |||
| align=center style="background-color:transparent;border-color:#000000"|2462 | |||
| align=center style="background-color:#ff8080;border-color:#000000"|5 | |||
| align=center style="background-color:transparent;border-color:#000000;color:#000000"|99.8 | |||
| align=center style="background-color:transparent;border-color:#000000;color:#000000"|99.8 | |||
|- | |||
| align=left style="background-color:trasnparentborder-color:#000000"| nettle | |||
| align=center style="background-color:transparent;border-color:#000000"|87 | |||
| align=center style="background-color:transparent;border-color:#000000"|86 | |||
| align=center style="background-color:#ff8080;border-color:#000000"|1 | |||
| align=center style="background-color:transparent;border-color:#000000;color:#000000"|98.85 | |||
| align=center style="background-color:transparent;border-color:#000000;color:#000000"|98.85 | |||
|- | |||
| align=left style="background-color:trasnparentborder-color:#000000"| openssh | |||
| align=center style="background-color:transparent;border-color:#000000"|56 | |||
| align=center style="background-color:transparent;border-color:#000000"|56 | |||
| align=center style="background-color:transparent;border-color:#000000"|0 | |||
| align=center style="background-color:transparent;border-color:#000000;color:#000000"|100.0 | |||
| align=center style="background-color:transparent;border-color:#000000;color:#000000"|100.0 | |||
|- | |||
| align=left style="background-color:trasnparentborder-color:#000000"| openssl | |||
| align=center style="background-color:transparent;border-color:#000000"|52 | |||
| align=center style="background-color:transparent;border-color:#000000"|52 | |||
| align=center style="background-color:transparent;border-color:#000000"|0 | |||
| align=center style="background-color:transparent;border-color:#000000;color:#000000"|100.0 | |||
| align=center style="background-color:transparent;border-color:#000000;color:#000000"|100.0 | |||
|- | |||
| align=left style="background-color:trasnparentborder-color:#000000"| pango | |||
| align=center style="background-color:transparent;border-color:#000000"|12 | |||
| align=center style="background-color:transparent;border-color:#000000"|12 | |||
| align=center style="background-color:transparent;border-color:#000000"|0 | |||
| align=center style="background-color:transparent;border-color:#000000;color:#000000"|100.0 | |||
| align=center style="background-color:transparent;border-color:#000000;color:#000000"|100.0 | |||
|- | |||
| align=left style="background-color:trasnparentborder-color:#000000"| perl | |||
| align=center style="background-color:transparent;border-color:#000000"|2263 | |||
| align=center style="background-color:transparent;border-color:#000000"|2262 | |||
| align=center style="background-color:#ff8080;border-color:#000000"|1 | |||
| align=center style="background-color:transparent;border-color:#000000;color:#000000"|99.96 | |||
| align=center style="background-color:transparent;border-color:#000000;color:#000000"|99.96 | |||
|- | |||
| align=left style="background-color:trasnparentborder-color:#000000"| perl5 | |||
| align=center style="background-color:transparent;border-color:#000000"|2263 | |||
| align=center style="background-color:transparent;border-color:#000000"|2262 | |||
| align=center style="background-color:#ff8080;border-color:#000000"|1 | |||
| align=center style="background-color:transparent;border-color:#000000;color:#000000"|99.96 | |||
| align=center style="background-color:transparent;border-color:#000000;color:#000000"|99.96 | |||
|- | |||
| align=left style="background-color:trasnparentborder-color:#000000"| python | |||
| align=center style="background-color:transparent;border-color:#000000"|32066 | |||
| align=center style="background-color:transparent;border-color:#000000"|31923 | |||
| align=center style="background-color:#ff8080;border-color:#000000"|143 | |||
| align=center style="background-color:#c5000b;border-color:#000000;color:#ffffff"|99.55 | |||
| align=center style="background-color:#c5000b;border-color:#000000;color:#ffffff"|99.61 | |||
| align=center style="background-color:transparent;border-color:transparent" |[[File:down_rate.png|15px]] | |||
|- | |||
| align=left style="background-color:trasnparentborder-color:#000000"| quilt | |||
| align=center style="background-color:transparent;border-color:#000000"|55 | |||
| align=center style="background-color:transparent;border-color:#000000"|54 | |||
| align=center style="background-color:#ff8080;border-color:#000000"|1 | |||
| align=center style="background-color:transparent;border-color:#000000;color:#000000"|98.18 | |||
| align=center style="background-color:transparent;border-color:#000000;color:#000000"|98.18 | |||
|- | |||
| align=left style="background-color:trasnparentborder-color:#000000"| strace | |||
| align=center style="background-color:transparent;border-color:#000000"|264 | |||
| align=center style="background-color:transparent;border-color:#000000"|263 | |||
| align=center style="background-color:#ff8080;border-color:#000000"|1 | |||
| align=center style="background-color:transparent;border-color:#000000;color:#000000"|99.62 | |||
| align=center style="background-color:transparent;border-color:#000000;color:#000000"|99.62 | |||
|- | |||
| align=left style="background-color:trasnparentborder-color:#000000"| tcl | |||
| align=center style="background-color:transparent;border-color:#000000"|148 | |||
| align=center style="background-color:transparent;border-color:#000000"|146 | |||
| align=center style="background-color:#ff8080;border-color:#000000"|2 | |||
| align=center style="background-color:#c5000b;border-color:#000000;color:#ffffff"|98.65 | |||
| align=center style="background-color:#c5000b;border-color:#000000;color:#ffffff"|100.0 | |||
| align=center style="background-color:transparent;border-color:transparent" |[[File:down_rate.png|15px]] | |||
|- | |||
| align=left style="background-color:trasnparentborder-color:#000000"| util-linux | |||
| align=center style="background-color:transparent;border-color:#000000"|131 | |||
| align=center style="background-color:transparent;border-color:#000000"|129 | |||
| align=center style="background-color:#ff8080;border-color:#000000"|2 | |||
| align=center style="background-color:#99ff66;border-color:#000000;color:#000000"|98.47 | |||
| align=center style="background-color:#99ff66;border-color:#000000;color:#000000"|98.46 | |||
| align=center style="background-color:transparent;border-color:transparent" |[[File:up_rate.png|15px]] | |||
|- | |||
| align=left style="background-color:trasnparentborder-color:#000000"| valgrind | |||
| align=center style="background-color:transparent;border-color:#000000"|553 | |||
| align=center style="background-color:transparent;border-color:#000000"|2 | |||
| align=center style="background-color:#ff8080;border-color:#000000"|551 | |||
| align=center style="background-color:transparent;border-color:#000000;color:#000000"|0.36 | |||
| align=center style="background-color:transparent;border-color:#000000;color:#000000"|0.36 | |||
|- | |||
| align=left style="background-color:trasnparentborder-color:#000000"| zlib | |||
| align=center style="background-color:transparent;border-color:#000000"|2 | |||
| align=center style="background-color:transparent;border-color:#000000"|2 | |||
| align=center style="background-color:transparent;border-color:#000000"|0 | |||
| align=center style="background-color:transparent;border-color:#000000;color:#000000"|100.0 | |||
| align=center style="background-color:transparent;border-color:#000000;color:#000000"|100.0 | |||
|- | |||
|} | |||
=Detailed Report= | =Detailed Report= | ||
Latest revision as of 01:57, 23 February 2018
Testing data
Test type: Full Pass Test Branch: morty Build name: 2.2.3 RC2 poky commit: e6626951501cdf8c5516ad42fd585894f19c2327 Autobuilder images repository: https://autobuilder.yocto.io/pub/releases/yocto-2.2.3.rc2 Distributions tested: CentOs 7 Debian 9 Fedora 26 Fedora 27 OpenSuse 42.3 Ubuntu 16.04 Ubuntu 17.10
{{#testopia: | run_id=0, {{#var:1_ADT_Fedora_25_i686}}, {{#var:2_ADT_Fedora_25_i686}}, {{#var:3_ADT_Ubuntu_16.04_x86_64}}, {{#var:4_ADT_Ubuntu_16.04_x86_64}}, {{#var:5_BSPs_genericx86_-_on_MMAX32bit}}, {{#var:6_BSPs_genericx86-64_on_NUC}}, {{#var:7_BSPs_Beaglebone_Black}}, {{#var:8_BSPs_EdgeRouter}}, {{#var:9_BSPs_MPC8315e-rdb}}, {{#var:10_BSPs_qemu-x86}}, {{#var:11_BSPs_qemuarm}}, {{#var:12_BSPs_qemuarm64}}, {{#var:13_BSPs_qemumips}}, {{#var:14_BSPs_qemumips64}}, {{#var:15_BSPs_qemuppc}}, {{#var:16_BSPs_qemux86-64}}, {{#var:17_BSPs_qemu-x86}}, {{#var:18_BSPs_qemuarm}}, {{#var:19_BSPs_qemuarm64}}, {{#var:20_BSPs_qemumips}}, {{#var:21_BSPs_qemumips64}}, {{#var:22_BSPs_qemuppc}}, {{#var:23_BSPs_qemux86-64,}}, {{#var:24_BSPs_qemux86-WIC}}, {{#var:25_BSPs_qemux86_64-WIC}}, {{#var:26_BSPs_genericx86_-_on_MMAX32bit}}, {{#var:27_BSPs_genericx86-64_on_MMAX64}}, {{#var:28_BSPs_genericx86-64_on_NUC}}, {{#var:29_BSPs_MPC8315e-rdb}}, {{#var:30_BSPs_Beaglebone_Black}}, {{#var:31_BSPs_EdgeRouter}}, {{#var:32_BSPs_genericx86-64-WIC_on_NUC}}, {{#var:33_BSPs_genericx86-64-lsb_on_MMAX64}}, {{#var:34_BSPs_genericx86-WIC_on_MMAX32}}, {{#var:35_Eclipse Plugin_Oxygen_on_Ubuntu_1604}}, {{#var:36_Eclipse Plugin_Neon_on_Ubuntu_1604}}, {{#var:37_eSDK_Ubuntu_16.04}}, {{#var:38_eSDK_Ubuntu_16.04}}, {{#var:39_General Runtime_user-preference_(any_supported_environment)}}, {{#var:40_Meta-yocto_Ubuntu_14.04_x86_64}}, {{#var:41_Runtime_VMWare_Player_on_Ubuntu}}, {{#var:42_Toaster_Ubuntu_16.04_x86_64}}, {{#var:43_BitBake_Ubuntu_16.04_x86_64}}, {{#var:44_OE-Core_CentOS_7_x86_64}}, {{#var:45_OE-Core_Debian_9_x86_64}}, {{#var:46_OE-Core_Fedora_26_x86_64}}, {{#var:47_OE-Core_Fedora_27_x86_64}}, {{#var:48_OE-Core_OpenSuse_42.3_x86_64}}, {{#var:49_OE-Core_Ubuntu_16.04_x86_64}}, {{#var:50_OE-Core_Ubuntu_17.10_x86_64}}, {{#var:51_Toaster_Ubuntu_16.04_x86_64}} | report_id=concatstats}}
|