Performance Test: Difference between revisions

From Yocto Project
Jump to navigationJump to search
No edit summary
 
(60 intermediate revisions by 4 users not shown)
Line 1: Line 1:
== Build Time Perfomance  ==
== Build time tracking ==
Script used for running the builds: (dylan/master branch): scripts/contrib/build-perf-test.sh


Two identical test systems are used for the results below, one with Ubuntu 12.04 (uname -nr: ubuntu12 3.8.0-31-generic) and one with Fedora 19 (uname -nr: fedora19 3.11.3-201.fc19.x86_64) with all updates available installed. Extra packages installed (besides what's by the default in the distro) are the ones in the Quick Start guide.
Script used for running the builds:  


All system/distro settings are distro's defaults, that includes:
*Before morty: poky/scripts/contrib/build-perf-test.sh
*After morty: poky/scripts/oe-build-perf-test
 
Two identical test systems are used for the results, with all updates available installed. Extra packages installed (besides what's by the default in the distro) are the ones in the Quick Start guide.
 
Configuration of the GDC Setup 2.1 - Current master
* Ubuntu 15.10 (Wily Werewolf)" -->  4.2.0-16-generic
* Fedora release 23 (Twenty Three) --> 4.7.10-100.fc23.x86_64
* HW configuration: Intel(R) Core(TM) i7-6700 CPU @ 3.40GHz, 16GB RAM,  1 x 600 GB SDD, 1 x 2 TB HDD
 
All system/distro settings are '''distro's defaults''', that includes:


- FS mount options: ext4 rw,relatime,data=ordered  
- FS mount options: ext4 rw,relatime,data=ordered  


- IO scheduler: cfq and CPU governer: ondemand
- IO scheduler: cfq and CPU governer: ondemand
HW config is: SandyBridge i7-2600 CPU @ 3.40GHz HT-enabled (4 cores/8 threads),  8GB RAM,  1 x 500 GB HDD


local.conf it's the default one (the script doesn't touch it except for changing DL_DIR if necessary).
local.conf it's the default one (the script doesn't touch it except for changing DL_DIR if necessary).
Line 20: Line 26:
MACHINE = "qemux86"</nowiki>
MACHINE = "qemux86"</nowiki>


In the tables below, unless otherwise noted, sizes are in kilobytes.
FAQ:
 
*Q: Where is 1.5_M2? It's there but it's called: 1.5_M1.final-412-geaa5df3 The M2 build wasn't released hence no git tag
'''Note:''' This page holds a re-run of milestone/RCs builds for the 1.5 cycle + 1.4.1/1.4.2 and for upcoming builds for 1.6. [https://wiki.yoctoproject.org/wiki/Performance_Test_Weekly_1.5 This page] has the old weekly results ran during the 1.5 cycle. The results differ because the test sytems used there are a bit different than those here (so don't cross compare results).
 
*Q: Where is M2 ? It's there but it's called: 1.5_M1.final-412-geaa5df3 The M2 build wasn't released hence no git tag
*Q: What's with the different number of RCs? See [[Yocto_Project_Release_Process#Milestone_Releases]]
*Q: What's with the different number of RCs? See [[Yocto_Project_Release_Process#Milestone_Releases]]
*Q: Why are there more results for some commits? The script is ran from a cron job multiple times a day, sometimes master changes daily sometimes it doesn't, hence more results for one commit (ie weekends)


== 1.4.1 & 1.4.2 Build Time results ==
<tab class=wikitable sep=comma head=top>
hostname,branch:commit,git describe,bitbake core-image-sato,bitbake virtual/kernel,bitbake core-image-sato (rm_work),bitbake core-image-sato -c rootfs,bitbake -p (rm -rf cache/ tmp/cache/), bitbake -p (rm -rf tmp/cache/),bitbake -p,size of tmp dir, size of tmp dir (rm_work)
ubuntu12,dylan:789b2b7e0cca404ff8ba8ed5ba823d215e6ba5b4,dylan-9.0.0,1:15:10,11:51.45,1:06:52,5:15.66,0:26.08,0:14.64,0:01.95,28104788,4693596
ubuntu12,dylan:73f103bf9b2cdf985464dc53bf4f1cfd71d4531f,yocto-1.4.1,1:16:41,11:32.41,1:06:51,5:14.78,0:26.75,0:14.81,0:01.96,28213628,4743912
ubuntu12,dylan:73f103bf9b2cdf985464dc53bf4f1cfd71d4531f,yocto-1.4.1,1:14:31,11:34.66,1:06:41,5:12.70,0:26.54,0:14.74,0:01.92,28213772,4743844
ubuntu12,dylan:73f103bf9b2cdf985464dc53bf4f1cfd71d4531f,yocto-1.4.1,1:12:39,11:35.99,1:06:59,5:16.48,0:26.77,0:14.71,0:01.96,28217808,4743800
ubuntu12,dylan:d734ab491a30078d43dee5440c03acce2d251425,yocto-1.4.2,1:12:58,12:56.22,1:07:45,4:54.36,0:26.78,0:14.73,0:01.95,25349116,4767324
ubuntu12,dylan:d734ab491a30078d43dee5440c03acce2d251425,yocto-1.4.2,1:13:33,12:44.85,1:08:10,5:01.71,0:26.87,0:14.80,0:01.99,25349708,4767348
ubuntu12,dylan:d734ab491a30078d43dee5440c03acce2d251425,yocto-1.4.2,1:13:00,12:59.19,1:08:13,5:07.04,0:26.83,0:14.73,0:01.97,25363408,4767380
</tab>
<tab class=wikitable sep=comma head=top>
hostname,branch:commit,git describe,bitbake core-image-sato,bitbake virtual/kernel,bitbake core-image-sato (rm_work),bitbake core-image-sato -c rootfs,bitbake -p (rm -rf cache/ tmp/cache/), bitbake -p (rm -rf tmp/cache/),bitbake -p,size of tmp dir, size of tmp dir (rm_work)
fedora19,dylan:73f103bf9b2cdf985464dc53bf4f1cfd71d4531f,yocto-1.4.1,1:14:55,10:11.74,1:10:35,4:06.95,0:34.06,0:17.79,0:01.99,28710092,4835464
fedora19,dylan:73f103bf9b2cdf985464dc53bf4f1cfd71d4531f,yocto-1.4.1,1:12:34,10:09.63,1:10:30,4:03.92,0:33.53,0:17.84,0:01.92,28713992,4835460
fedora19,dylan:73f103bf9b2cdf985464dc53bf4f1cfd71d4531f,yocto-1.4.1,1:17:15,10:11.67,1:11:47,4:45.40,0:33.90,0:17.31,0:01.97,28710884,4835484
fedora19,dylan:d734ab491a30078d43dee5440c03acce2d251425,yocto-1.4.2,1:14:50,10:53.97,1:15:47,4:19.85,0:34.18,0:17.92,0:01.92,25778860,4858892
fedora19,dylan:d734ab491a30078d43dee5440c03acce2d251425,yocto-1.4.2,1:14:58,10:43.16,1:12:00,4:07.15,0:33.93,0:17.88,0:01.98,25776164,4858920
fedora19,dylan:d734ab491a30078d43dee5440c03acce2d251425,yocto-1.4.2,1:16:11,11:23.14,1:12:31,4:08.25,0:33.64,0:18.03,0:01.95,25775976,4858876
</tab>
== 1.5 Milestones results ==
<tab class=wikitable sep=comma head=top>
hostname,branch:commit,git describe,bitbake core-image-sato,bitbake virtual/kernel,bitbake core-image-sato (rm_work),bitbake core-image-sato -c rootfs,bitbake -p (rm -rf cache/ tmp/cache/), bitbake -p (rm -rf tmp/cache/),bitbake -p,size of tmp dir, size of tmp dir (rm_work)
ubuntu12,master:a62aed41f2d8f874f7ae24d0e5be5dbc66ea2199,1.5_M1.final,1:14:16,12:53.53,1:08:28,5:11.74,0:27.39,0:15.28,0:02.18,25391176,4799632
ubuntu12,master:a62aed41f2d8f874f7ae24d0e5be5dbc66ea2199,1.5_M1.final,1:14:20,12:54.69,1:08:27,5:23.61,0:27.24,0:15.19,0:02.15,25390100,4799560
ubuntu12,master:a62aed41f2d8f874f7ae24d0e5be5dbc66ea2199,1.5_M1.final,1:13:14,12:42.64,1:08:49,5:05.26,0:27.49,0:15.16,0:02.14,25390404,4799588
-
ubuntu12,master:eaa5df34af42b6a37f6506847d0f3ef6ba0d298a,1.5_M1.final-412-geaa5df3,1:14:39,13:29.39,1:11:05,4:47.33,0:28.01,0:15.53,0:02.09,25316072,4827788
ubuntu12,master:eaa5df34af42b6a37f6506847d0f3ef6ba0d298a,1.5_M1.final-412-geaa5df3,1:15:39,13:39.82,1:11:14,5:00.98,0:28.02,0:15.74,0:02.11,25297644,4827824
ubuntu12,master:eaa5df34af42b6a37f6506847d0f3ef6ba0d298a,1.5_M1.final-412-geaa5df3,1:13:38,13:20.40,1:10:29,4:50.66,0:28.21,0:15.63,0:02.08,25296940,4823004
-
ubuntu12,master:9de0ad47462c13ac4a2d73e12e92be6c1b4e0415,1.5_M3,1:15:31,13:21.54,1:11:40,4:50.34,0:28.58,0:15.74,0:02.11,25440952,4838208
ubuntu12,master:9de0ad47462c13ac4a2d73e12e92be6c1b4e0415,1.5_M3,1:15:34,13:17.44,1:11:25,4:54.69,0:28.74,0:15.71,0:02.09,25421728,4833420
ubuntu12,master:9de0ad47462c13ac4a2d73e12e92be6c1b4e0415,1.5_M3,1:16:25,13:37.05,1:11:54,4:56.99,0:28.63,0:15.85,0:02.12,25378284,4833444
-
ubuntu12,master:5745e45b18e5099e94b4d5a73bc97dc6d4cdc91f,1.5_M4.rc2,1:17:58,13:44.95,1:12:30,5:03.64,0:28.95,0:15.97,0:02.32,26131176,4961052
ubuntu12,master:5745e45b18e5099e94b4d5a73bc97dc6d4cdc91f,1.5_M4.rc2,1:18:54,14:00.29,1:13:15,4:59.28,0:28.65,0:15.96,0:02.24,26192480,4966232
ubuntu12,master:5745e45b18e5099e94b4d5a73bc97dc6d4cdc91f,1.5_M4.rc2,1:18:56,14:17.80,1:14:11,4:59.10,0:28.81,0:16.12,0:02.25,26131008,4961248
-
ubuntu12,master:ed3ef0823fde89371a473d20c0127c0bd16d062b,1.5_M4.final,1:18:19,13:41.56,1:11:33,4:53.58,0:28.27,0:15.41,0:01.54,25688600,4898748
ubuntu12,master:ed3ef0823fde89371a473d20c0127c0bd16d062b,1.5_M4.final,1:15:25,13:59.10,1:11:41,4:51.22,0:28.39,0:15.36,0:01.56,25692980,4898576
ubuntu12,master:ed3ef0823fde89371a473d20c0127c0bd16d062b,1.5_M4.final,1:17:12,13:41.63,1:12:06,4:51.32,0:27.97,0:15.38,0:01.58,25688548,4903912
-
ubuntu12,master:566ca1e4769b4ec7e3f6c6a98bb817db2221cf82,1.5_M5.rc1,1:15:17,13:47.21,1:10:44,4:55.33,0:28.11,0:15.16,0:01.54,25726992,4897340
ubuntu12,master:566ca1e4769b4ec7e3f6c6a98bb817db2221cf82,1.5_M5.rc1,1:15:03,14:06.96,1:11:32,4:50.64,0:27.95,0:15.20,0:01.58,25704416,4902296
ubuntu12,master:566ca1e4769b4ec7e3f6c6a98bb817db2221cf82,1.5_M5.rc1,1:17:16,14:07.67,1:12:23,4:53.21,0:27.78,0:15.19,0:01.58,25686264,4902292
-
ubuntu12,master:853bc53cd58a621918f0e5ce662dba263d1befb4,1.5_M5.rc4,1:16:38,14:17.30,1:10:05,4:42.63,0:25.57,0:12.87,0:01.50,25770616,4925008
ubuntu12,master:853bc53cd58a621918f0e5ce662dba263d1befb4,1.5_M5.rc4,1:14:39,14:14.73,1:11:21,4:53.20,0:25.47,0:12.80,0:01.48,25770168,4924972
ubuntu12,master:853bc53cd58a621918f0e5ce662dba263d1befb4,1.5_M5.rc4,1:17:48,14:19.40,1:11:51,4:53.53,0:25.39,0:12.81,0:01.49,25773428,4924908
-
ubuntu12,master:4d8ab87d25ea6e59dc88375813ef88ee0ac813d0,1.5_M5.rc5,1:14:30,13:57.15,1:11:08,4:54.14,0:25.68,0:13.07,0:01.59,25763920,4920848
ubuntu12,master:4d8ab87d25ea6e59dc88375813ef88ee0ac813d0,1.5_M5.rc5,1:15:28,14:06.75,1:11:38,4:51.25,0:25.45,0:13.03,0:01.61,25761664,4921016
ubuntu12,master:4d8ab87d25ea6e59dc88375813ef88ee0ac813d0,1.5_M5.rc5,1:16:24,14:15.42,1:12:12,4:42.56,0:25.46,0:12.97,0:01.59,25780792,4915792
-
ubuntu12,master:a02280f99f05b1c21de2a78de9222a2f62a0dabc,1.5_M5.rc6,1:17:19,13:59.28,1:11:50,4:48.02,0:25.62,0:12.96,0:01.59,25759796,4919568
ubuntu12,master:a02280f99f05b1c21de2a78de9222a2f62a0dabc,1.5_M5.rc6,1:17:54,13:56.51,1:11:56,4:50.50,0:25.67,0:13.04,0:01.62,25763872,4919684
</tab>


<tab class=wikitable sep=comma head=top>
== Daily Test Results ==
hostname,branch:commit,git describe,bitbake core-image-sato,bitbake virtual/kernel,bitbake core-image-sato (rm_work),bitbake core-image-sato -c rootfs,bitbake -p (rm -rf cache/ tmp/cache/), bitbake -p (rm -rf tmp/cache/),bitbake -p,size of tmp dir, size of tmp dir (rm_work)
Build performance test suite is run multiple times per day. Latest test reports for master branch can found here:
fedora19,master:a62aed41f2d8f874f7ae24d0e5be5dbc66ea2199,1.5_M1.final,1:19:37,10:38.18,1:12:10,4:03.70,0:34.59,0:18.54,0:02.23,25801440,4887964
* [https://wiki.yoctoproject.org/charts/build_perf/latest/ypperf-ubuntu16_master_qemux86.html Ubuntu 16.04]
fedora19,master:a62aed41f2d8f874f7ae24d0e5be5dbc66ea2199,1.5_M1.final,1:19:03,11:24.42,1:12:52,4:01.95,0:34.92,0:18.56,0:02.13,25796764,4887836
* [https://wiki.yoctoproject.org/charts/build_perf/latest/ypperf-ubuntu16-xeon_master_qemux86.html Ubuntu 16.04 (Intel® Xeon® HW)]
* [https://wiki.yoctoproject.org/charts/build_perf/latest/ypperf-centos7_master_qemux86.html CentOS 7]
* [https://wiki.yoctoproject.org/charts/build_perf/latest/ypperf-opensuse422_master_qemux86.html OpenSUSE Leap 42.2] (retired)
* [https://wiki.yoctoproject.org/charts/build_perf/latest/ypperf-fedora25_master_qemux86.html Fedora 25] (retired)
* [https://wiki.yoctoproject.org/charts/build_perf/latest/ypperf01_master_qemux86.html Fedora 23] (retired)
* [https://wiki.yoctoproject.org/charts/build_perf/latest/ypperf02_master_qemux86.html Ubuntu 15.10] (retired)


fedora19,master:eaa5df34af42b6a37f6506847d0f3ef6ba0d298a,1.5_M1.final-412-geaa5df3,1:20:02,11:43.17,1:13:52,3:58.63,0:35.54,0:19.14,0:02.13,25704568,4911492
== Milestone Test results ==
fedora19,master:eaa5df34af42b6a37f6506847d0f3ef6ba0d298a,1.5_M1.final-412-geaa5df3,1:21:20,12:11.54,1:14:48,3:59.29,0:35.74,0:19.06,0:02.18,25710016,4911228
Guadalajara Setup from 2.1 M4 to actual master  


fedora19,master:9de0ad47462c13ac4a2d73e12e92be6c1b4e0415,1.5_M3,1:22:38,11:18.43,1:18:37,4:17.43,0:36.29,0:19.14,0:02.10,25783316,4921400
* [https://wiki.yoctoproject.org/charts/perf_milestone_GDC/performance_test.html GDC Charts]
fedora19,master:9de0ad47462c13ac4a2d73e12e92be6c1b4e0415,1.5_M3,1:22:51,11:29.19,1:17:53,4:09.21,0:35.81,0:19.49,0:02.13,25782900,4921324


fedora19,master:5745e45b18e5099e94b4d5a73bc97dc6d4cdc91f,1.5_M4.rc2,1:23:20,12:42.28,1:16:41,4:08.14,0:35.09,0:18.98,0:02.29,26535116,5049040
== Old Test results ==
fedora19,master:5745e45b18e5099e94b4d5a73bc97dc6d4cdc91f,1.5_M4.rc2,1:23:58,12:29.62,1:17:23,4:07.62,0:34.85,0:19.07,0:02.33,26540144,5048920


fedora19,master:ed3ef0823fde89371a473d20c0127c0bd16d062b,1.5_M4.final,1:21:45,12:37.91,1:15:53,4:05.91,0:34.19,0:18.32,0:01.60,26096196,4986476
Milestone test results from the retired Romania Setup (Yocto 1.6 – 2.1) can be found from here:
fedora19,master:ed3ef0823fde89371a473d20c0127c0bd16d062b,1.5_M4.final,1:22:51,12:50.33,1:17:02,4:06.61,0:34.13,0:18.61,0:01.62,26093328,4986408


fedora19,master:566ca1e4769b4ec7e3f6c6a98bb817db2221cf82,1.5_M5.rc1,1:22:27,13:14.06,1:16:17,3:57.51,0:34.03,0:18.30,0:01.61,26090072,4984856
* [https://wiki.yoctoproject.org/charts/perf_milestone/performance_test.html RO Charts]
fedora19,master:566ca1e4769b4ec7e3f6c6a98bb817db2221cf82,1.5_M5.rc1,1:22:51,13:12.33,1:16:06,3:58.44,0:34.06,0:18.15,0:01.62,26090640,4985060


fedora19,master:853bc53cd58a621918f0e5ce662dba263d1befb4,1.5_M5.rc4,1:20:58,12:53.74,1:14:58,3:53.32,0:31.15,0:15.29,0:01.61,26176264,5008052
Raw test data from the retired Romania test setup has been moved to
fedora19,master:853bc53cd58a621918f0e5ce662dba263d1befb4,1.5_M5.rc4,1:21:45,13:02.75,1:17:10,3:53.24,0:31.33,0:15.23,0:01.60,26177216,5007964
* [[Performance_Test_Old_Results]]
* [[Performance_Test_Weekly_1.5]] has the old weekly results ran during the 1.5 cycle.


fedora19,master:4d8ab87d25ea6e59dc88375813ef88ee0ac813d0,1.5_M5.rc5,1:23:16,13:40.63,1:16:49,3:47.56,0:31.17,0:15.36,0:01.67,26168976,5003560
Configuration for the Romania Setup 1.6 - 2.1:
fedora19,master:4d8ab87d25ea6e59dc88375813ef88ee0ac813d0,1.5_M5.rc5,1:22:23,12:48.88,1:16:17,3:51.08,0:31.24,0:15.36,0:01.62,26169684,5003712


fedora19,master:a02280f99f05b1c21de2a78de9222a2f62a0dabc,1.5_M5.rc6,1:23:49,12:43.59,1:16:15,3:49.05,0:31.30,0:15.55,0:01.63,26165572,5007660
* Ubuntu 12.04 (uname -nr: ubuntu12 3.8.0-31-generic) and
fedora19,master:a02280f99f05b1c21de2a78de9222a2f62a0dabc,1.5_M5.rc6,1:23:51,12:19.76,1:17:52,4:10.54,0:31.17,0:15.30,0:01.60,26164292,5007292
* Fedora 19 (uname -nr: fedora19 3.11.3-201.fc19.x86_64)
</tab>
* HW configuration: SandyBridge i7-2600 CPU @ 3.40GHz (4 cores/8 threads), 8GB RAM, 1 x 500 GB HDD

Latest revision as of 14:17, 28 January 2019

Build time tracking

Script used for running the builds:

  • Before morty: poky/scripts/contrib/build-perf-test.sh
  • After morty: poky/scripts/oe-build-perf-test

Two identical test systems are used for the results, with all updates available installed. Extra packages installed (besides what's by the default in the distro) are the ones in the Quick Start guide.

Configuration of the GDC Setup 2.1 - Current master

  • Ubuntu 15.10 (Wily Werewolf)" --> 4.2.0-16-generic
  • Fedora release 23 (Twenty Three) --> 4.7.10-100.fc23.x86_64
  • HW configuration: Intel(R) Core(TM) i7-6700 CPU @ 3.40GHz, 16GB RAM, 1 x 600 GB SDD, 1 x 2 TB HDD

All system/distro settings are distro's defaults, that includes:

- FS mount options: ext4 rw,relatime,data=ordered

- IO scheduler: cfq and CPU governer: ondemand

local.conf it's the default one (the script doesn't touch it except for changing DL_DIR if necessary).

It however exports these in the enviroment:

BB_NUMBER_THREADS = "8"
PARALLEL_MAKE = "-j 8"
MACHINE = "qemux86"

FAQ:

  • Q: Where is 1.5_M2? It's there but it's called: 1.5_M1.final-412-geaa5df3 The M2 build wasn't released hence no git tag
  • Q: What's with the different number of RCs? See Yocto_Project_Release_Process#Milestone_Releases
  • Q: Why are there more results for some commits? The script is ran from a cron job multiple times a day, sometimes master changes daily sometimes it doesn't, hence more results for one commit (ie weekends)


Daily Test Results

Build performance test suite is run multiple times per day. Latest test reports for master branch can found here:

Milestone Test results

Guadalajara Setup from 2.1 M4 to actual master

Old Test results

Milestone test results from the retired Romania Setup (Yocto 1.6 – 2.1) can be found from here:

Raw test data from the retired Romania test setup has been moved to

Configuration for the Romania Setup 1.6 - 2.1:

  • Ubuntu 12.04 (uname -nr: ubuntu12 3.8.0-31-generic) and
  • Fedora 19 (uname -nr: fedora19 3.11.3-201.fc19.x86_64)
  • HW configuration: SandyBridge i7-2600 CPU @ 3.40GHz (4 cores/8 threads), 8GB RAM, 1 x 500 GB HDD