Performance Test

From Yocto Project
Jump to navigationJump to search

Build time tracking

Script used for running the builds:

  • Before morty: poky/scripts/contrib/build-perf-test.sh
  • After morty: poky/scripts/oe-build-perf-test

Two identical test systems are used for the results, with all updates available installed. Extra packages installed (besides what's by the default in the distro) are the ones in the Quick Start guide.

Romania Setup 1.6 - 2.1

  • Ubuntu 12.04 (uname -nr: ubuntu12 3.8.0-31-generic) and
  • Fedora 19 (uname -nr: fedora19 3.11.3-201.fc19.x86_64)

GDC Setup 2.1 - Current master

  • Ubuntu 15.10 (Wily Werewolf)" --> 4.2.0-16-generic
  • Fedora release 23 (Twenty Three) --> 4.7.10-100.fc23.x86_64

All system/distro settings are distro's defaults, that includes:

- FS mount options: ext4 rw,relatime,data=ordered

- IO scheduler: cfq and CPU governer: ondemand

Romania Setup 1.6 - 2.1

  • HW config is: SandyBridge i7-2600 CPU @ 3.40GHz (4 cores/8 threads), 8GB RAM, 1 x 500 GB HDD

GDC Setup 2.1 - Current master

  • HW config is: Intel(R) Core(TM) i7-6700 CPU @ 3.40GHz, 16GB RAM, 1 x 600 GB SDD, 1 x 2 TB HDD

local.conf it's the default one (the script doesn't touch it except for changing DL_DIR if necessary).

It however exports these in the enviroment: 

BB_NUMBER_THREADS = "8"
PARALLEL_MAKE = "-j 8"
MACHINE = "qemux86"

In the tables below, unless otherwise noted, sizes are in kilobytes.

Note:

  • Master builds are run daily on whatever commit there is when the cron kicks in.
  • This page holds a re-run of milestone/RCs builds only for the 1.5 (dora) cycle + 1.4.1/1.4.2 (dylan).
  • This page has the old weekly results ran during the 1.5 cycle. The results differ because the test sytems used there are a bit different than those here (so don't cross compare results).

FAQ:

  • Q: Where is 1.5_M2? It's there but it's called: 1.5_M1.final-412-geaa5df3 The M2 build wasn't released hence no git tag
  • Q: What's with the different number of RCs? See Yocto_Project_Release_Process#Milestone_Releases
  • Q: Why are there more results for some commits? The script is ran from a cron job multiple times a day, sometimes master changes daily sometimes it doesn't, hence more results for one commit (ie weekends)


Ubuntu has the default system shell (/bin/sh -> /bin/dash) and Fedora has the default SELinux policy (enabled). (Some quick tests showed that switching to bash or disabling SELinux had an minimal impact on the total build time). 

# sh -> dash (default)
ubuntu12,master:ea92671d9823e3667d6ced7ac2af20f991da404d,poky-10.0.0.final-426-gea92671,1:14:02,12:03.51,1:09:57,4:49.58,0:26.57,0:13.32,0:01.72,25446936,4810304
ubuntu12,master:ea92671d9823e3667d6ced7ac2af20f991da404d,poky-10.0.0.final-426-gea92671,1:14:10,12:06.96,1:10:22,4:52.66,0:26.31,0:13.35,0:01.66,25425188,4815440
ubuntu12,master:ea92671d9823e3667d6ced7ac2af20f991da404d,poky-10.0.0.final-426-gea92671,1:14:16,12:09.09,1:10:34,4:47.50,0:26.10,0:13.51,0:01.57,25425852,4810240
# sh -> bash
ubuntu12,master:ea92671d9823e3667d6ced7ac2af20f991da404d,poky-10.0.0.final-426-gea92671,1:12:56,12:03.28,1:10:25,4:57.96,0:26.19,0:13.36,0:01.59,25426624,4810376
ubuntu12,master:ea92671d9823e3667d6ced7ac2af20f991da404d,poky-10.0.0.final-426-gea92671,1:13:36,12:04.68,1:10:30,5:04.22,0:26.15,0:13.34,0:01.64,25425332,4815348
ubuntu12,master:ea92671d9823e3667d6ced7ac2af20f991da404d,poky-10.0.0.final-426-gea92671,1:14:21,12:08.91,1:10:41,5:01.82,0:26.39,0:13.39,0:01.83,25426632,4815256
# selinux=1 (default)
fedora19,master:ea92671d9823e3667d6ced7ac2af20f991da404d,poky-10.0.0.final-426-gea92671,1:15:22,10:40.98,1:13:21,4:05.22,0:32.10,0:15.63,0:01.63,25831032,4898552
fedora19,master:ea92671d9823e3667d6ced7ac2af20f991da404d,poky-10.0.0.final-426-gea92671,1:16:28,10:40.84,1:14:15,4:04.72,0:31.64,0:15.79,0:01.63,25831752,4898616
fedora19,master:ea92671d9823e3667d6ced7ac2af20f991da404d,poky-10.0.0.final-426-gea92671,1:17:04,10:32.56,1:14:18,4:04.11,0:31.74,0:15.75,0:01.59,25831972,4898692
# selinux=0
fedora19,master:ea92671d9823e3667d6ced7ac2af20f991da404d,poky-10.0.0.final-426-gea92671,1:16:22,10:14.86,1:13:09,3:55.77,0:31.72,0:15.58,0:01.60,25832104,4898672
fedora19,master:ea92671d9823e3667d6ced7ac2af20f991da404d,poky-10.0.0.final-426-gea92671,1:16:45,10:24.95,1:13:25,4:04.07,0:31.64,0:15.64,0:01.60,25831380,4898616
fedora19,master:ea92671d9823e3667d6ced7ac2af20f991da404d,poky-10.0.0.final-426-gea92671,1:17:28,10:16.52,1:12:42,4:02.91,0:31.70,0:15.53,0:01.59,25834384,4898748

Milestone Test results

The results are split in tow different phases as below

Romania Setup from 1.6 to 2.1 releases

Guadalajara Setup from 2.1 M4 to actual master

1.4.1 & 1.4.2 & 1.4.3 Build Time results

hostname branch:commit git describe bitbake core-image-sato bitbake virtual/kernel bitbake core-image-sato (rm_work) bitbake core-image-sato -c rootfs bitbake -p (rm -rf cache/ tmp/cache/) bitbake -p (rm -rf tmp/cache/) bitbake -p size of tmp dir size of tmp dir (rm_work)
ubuntu12 dylan:789b2b7e0cca404ff8ba8ed5ba823d215e6ba5b4 dylan-9.0.0 1:15:10 11:51.45 1:06:52 5:15.66 0:26.08 0:14.64 0:01.95 28104788 4693596
ubuntu12 dylan:73f103bf9b2cdf985464dc53bf4f1cfd71d4531f yocto-1.4.1 1:16:41 11:32.41 1:06:51 5:14.78 0:26.75 0:14.81 0:01.96 28213628 4743912
ubuntu12 dylan:73f103bf9b2cdf985464dc53bf4f1cfd71d4531f yocto-1.4.1 1:14:31 11:34.66 1:06:41 5:12.70 0:26.54 0:14.74 0:01.92 28213772 4743844
ubuntu12 dylan:73f103bf9b2cdf985464dc53bf4f1cfd71d4531f yocto-1.4.1 1:12:39 11:35.99 1:06:59 5:16.48 0:26.77 0:14.71 0:01.96 28217808 4743800
ubuntu12 dylan:d734ab491a30078d43dee5440c03acce2d251425 yocto-1.4.2 1:12:58 12:56.22 1:07:45 4:54.36 0:26.78 0:14.73 0:01.95 25349116 4767324
ubuntu12 dylan:d734ab491a30078d43dee5440c03acce2d251425 yocto-1.4.2 1:13:33 12:44.85 1:08:10 5:01.71 0:26.87 0:14.80 0:01.99 25349708 4767348
ubuntu12 dylan:d734ab491a30078d43dee5440c03acce2d251425 yocto-1.4.2 1:13:00 12:59.19 1:08:13 5:07.04 0:26.83 0:14.73 0:01.97 25363408 4767380
ubuntu12 dylan:7ab82abcad9ede847c42c01d2a4f4c2c18877e6f yocto-1.4.2-64-g7ab82ab 1:09:23 12:58.26 1:07:54 4:57.79 0:26.79 0:14.88 0:01.96 25365532 4767572
ubuntu12 dylan:7ab82abcad9ede847c42c01d2a4f4c2c18877e6f yocto-1.4.2-64-g7ab82ab 1:09:33 13:00.44 1:07:48 5:01.62 0:27.07 0:14.91 0:02.11 25366496 4767568
ubuntu12 dylan:7ab82abcad9ede847c42c01d2a4f4c2c18877e6f yocto-1.4.2-64-g7ab82ab 1:12:25 13:05.14 1:09:17 4:58.21 0:26.99 0:14.87 0:01.95 25351352 4767496


hostname branch:commit git describe bitbake core-image-sato bitbake virtual/kernel bitbake core-image-sato (rm_work) bitbake core-image-sato -c rootfs bitbake -p (rm -rf cache/ tmp/cache/) bitbake -p (rm -rf tmp/cache/) bitbake -p size of tmp dir size of tmp dir (rm_work)
fedora19 dylan:73f103bf9b2cdf985464dc53bf4f1cfd71d4531f yocto-1.4.1 1:14:55 10:11.74 1:10:35 4:06.95 0:34.06 0:17.79 0:01.99 28710092 4835464
fedora19 dylan:73f103bf9b2cdf985464dc53bf4f1cfd71d4531f yocto-1.4.1 1:12:34 10:09.63 1:10:30 4:03.92 0:33.53 0:17.84 0:01.92 28713992 4835460
fedora19 dylan:73f103bf9b2cdf985464dc53bf4f1cfd71d4531f yocto-1.4.1 1:17:15 10:11.67 1:11:47 4:45.40 0:33.90 0:17.31 0:01.97 28710884 4835484
fedora19 dylan:d734ab491a30078d43dee5440c03acce2d251425 yocto-1.4.2 1:14:50 10:53.97 1:15:47 4:19.85 0:34.18 0:17.92 0:01.92 25778860 4858892
fedora19 dylan:d734ab491a30078d43dee5440c03acce2d251425 yocto-1.4.2 1:14:58 10:43.16 1:12:00 4:07.15 0:33.93 0:17.88 0:01.98 25776164 4858920
fedora19 dylan:d734ab491a30078d43dee5440c03acce2d251425 yocto-1.4.2 1:16:11 11:23.14 1:12:31 4:08.25 0:33.64 0:18.03 0:01.95 25775976 4858876
fedora19 dylan:7ab82abcad9ede847c42c01d2a4f4c2c18877e6f yocto-1.4.2-64-g7ab82ab 1:12:05 10:26.20 1:15:21 4:19.49 0:34.25 0:18.12 0:01.95 25774008 4858772
fedora19 dylan:7ab82abcad9ede847c42c01d2a4f4c2c18877e6f yocto-1.4.2-64-g7ab82ab 1:11:50 10:30.01 1:11:48 4:16.69 0:33.99 0:18.03 0:01.94 25775640 4858692
fedora19 dylan:7ab82abcad9ede847c42c01d2a4f4c2c18877e6f yocto-1.4.2-64-g7ab82ab 1:11:51 10:25.89 1:11:56 4:42.52 0:34.19 0:17.97 0:01.94 25776076 4858728
Retrieved from "https://wiki.yoctoproject.org/wiki/index.php?title=Performance_Test&oldid=27063"