Performance Test: Difference between revisions

From Yocto Project
Jump to navigationJump to search
 
(124 intermediate revisions by 4 users not shown)
Line 1: Line 1:
== Build Perfomance Data ==
== Build time tracking ==
Script used for running the builds: (dylan/master branch): scripts/contrib/build-perf-test.sh


Test systems:
Script used for running the builds:  


System1: SandyBridge i7-2600 CPU @ 3.40GHz HT-enabled (4 cores/8 threads),  8GB RAM,  1 x 500 GB HDD, Ubuntu 12.04 x64 (with all updates available installed)
*Before morty: poky/scripts/contrib/build-perf-test.sh
        System settings are distro's default:
*After morty: poky/scripts/oe-build-perf-test
              - FS mount options are Ubuntu's default: ext4 rw,relatime,user_xattr,barrier=1,data=ordered
              - IO scheduler: cfq (default) and CPU governer: ondemand (default)
System2: IvyBridge i7-3770K CPU @ 3.50GHz HT-enabled (4 cores/8 threads), 32GB RAM,  1 x 1 TB HDD, Fedora 18 x64 (with all updates available installed)
          System settings are distro's default:
              - FS mount options are Fedora's default: ext4 rw,seclabel,relatime,data=ordered
              - IO scheduler: cfq (default) and CPU governer: ondemand (default)


Two identical test systems are used for the results, with all updates available installed. Extra packages installed (besides what's by the default in the distro) are the ones in the Quick Start guide.


local.conf:
Configuration of the GDC Setup 2.1 - Current master
* Ubuntu 15.10 (Wily Werewolf)" -->  4.2.0-16-generic
* Fedora release 23 (Twenty Three) --> 4.7.10-100.fc23.x86_64
* HW configuration: Intel(R) Core(TM) i7-6700 CPU @ 3.40GHz, 16GB RAM,  1 x 600 GB SDD, 1 x 2 TB HDD


<nowiki>BB_NUMBER_THREADS = "8"
All system/distro settings are '''distro's defaults''', that includes:
PARALLEL_MAKE = "-j 8"
MACHINE = "qemux86"
DISTRO ?= "poky"
PACKAGE_CLASSES ?= "package_rpm"
EXTRA_IMAGE_FEATURES = "debug-tweaks"
USER_CLASSES ?= "buildstats image-mklibs image-prelink"
PATCHRESOLVE = "noop"
BB_DISKMON_DIRS = "\
    STOPTASKS,${TMPDIR},1G,100K \
    STOPTASKS,${DL_DIR},1G,100K \
    STOPTASKS,${SSTATE_DIR},1G,100K \
    ABORT,${TMPDIR},100M,1K \
    ABORT,${DL_DIR},100M,1K \
    ABORT,${SSTATE_DIR},100M,1K"
CONF_VERSION = "1"
DL_DIR = "/data/yocto/poky/build-perf-test/downloads"</nowiki>


- FS mount options: ext4 rw,relatime,data=ordered


== 1.5 nightly build time results ==
- IO scheduler: cfq and CPU governer: ondemand


(System1 = ubuntu12, System2=thunderbird from above description)
local.conf it's the default one (the script doesn't touch it except for changing DL_DIR if necessary).


It however exports these in the enviroment:
<nowiki>BB_NUMBER_THREADS = "8"
PARALLEL_MAKE = "-j 8"
MACHINE = "qemux86"</nowiki>


<tab class=wikitable sep=comma head=top>
FAQ:
hostname,branch:commit,git describe,bitbake core-image-sato,bitbake virtual/kernel,bitbake core-image-sato (rm_work),bitbake core-image-sato -c rootfs,bitbake -p (rm -rf cache/ tmp/cache/), bitbake -p (rm -rf tmp/cache/),bitbake -p,size of tmp dir, size of tmp dir (rm_work)
*Q: Where is 1.5_M2? It's there but it's called: 1.5_M1.final-412-geaa5df3 The M2 build wasn't released hence no git tag
ubuntu12,master:14010b4a60072641bab62c761edffc51148ae6e9,,1:22:59,12:44.92,1:11:48,4:58.42,0:27.81,0:15.54,0:02.20
*Q: What's with the different number of RCs? See [[Yocto_Project_Release_Process#Milestone_Releases]]
ubuntu12,master:a62aed41f2d8f874f7ae24d0e5be5dbc66ea2199,,1:21:07,12:40.53,1:12:38,4:58.00,0:27.65,0:15.35,0:02.18
*Q: Why are there more results for some commits? The script is ran from a cron job multiple times a day, sometimes master changes daily sometimes it doesn't, hence more results for one commit (ie weekends)
ubuntu12,master:a62aed41f2d8f874f7ae24d0e5be5dbc66ea2199,,1:20:10,12:28.66,1:12:24,5:00.78,0:27.98,0:15.64,0:02.20
ubuntu12,master:a62aed41f2d8f874f7ae24d0e5be5dbc66ea2199,,1:20:39,12:35.29,1:11:41,4:55.07,0:28.25,0:15.37,0:02.20
ubuntu12,master:a62aed41f2d8f874f7ae24d0e5be5dbc66ea2199,,1:20:13,12:52.21,1:13:21,5:01.17,0:27.83,0:15.47,0:02.25
ubuntu12,master:7bf5c38e0f8bed9295f46773ade5336ec41044f6,,1:21:38,13:18.64,1:12:37,4:54.43,0:28.14,0:15.19,0:02.17
ubuntu12,master:7bf5c38e0f8bed9295f46773ade5336ec41044f6,,1:21:17,12:58.65,1:12:12,4:57.34,0:27.63,0:15.36,0:02.19
ubuntu12,master:7bf5c38e0f8bed9295f46773ade5336ec41044f6,,1:21:46,13:09.45,1:13:13,4:57.01,0:27.92,0:15.96,0:02.18
ubuntu12,master:7bf5c38e0f8bed9295f46773ade5336ec41044f6,,1:22:06,13:06.07,1:12:44,4:54.78,0:27.80,0:15.49,0:02.20
ubuntu12,master:5128f0a8465f574f520282499410446a156fc54d,,1:24:33,13:31.94,1:13:06,4:51.96,0:28.59,0:15.22,0:02.19
ubuntu12,master:96b144a25598090b1e58eb5e33f216b5bfef7595,,1:23:11,12:58.86,1:14:20,4:57.85,0:28.32,0:15.29,0:02.21
</tab>
 
<tab class=wikitable sep=comma head=top>
hostname,branch:commit,git describe,bitbake core-image-sato,bitbake virtual/kernel,bitbake core-image-sato (rm_work),bitbake core-image-sato -c rootfs,bitbake -p (rm -rf cache/ tmp/cache/), bitbake -p (rm -rf tmp/cache/),bitbake -p,size of tmp dir, size of tmp dir (rm_work)
thunderbird,master:176a46b088135f2891983bb8540bc45389341d90,,1:07:09,11:39.62,1:05:35,4:57.37,0:31.22,0:16.63,0:02.00
thunderbird,master:21089515bd18ba18e86ec8f72ff8141b9fb1dea8,,1:06:27,11:26.78,1:05:48,4:53.59,0:31.22,0:17.96,0:02.20
thunderbird,master:e8197722de1f6006ceae0d4266de91c1db7278e3,,1:06:00,13:17.22,1:05:39,4:52.55,0:31.10,0:16.81,0:02.19
thunderbird,master:676fd3f394a0c0576b63c77ed2184bf5f42c29b6,,1:07:43,11:42.89,1:06:16,4:51.57,0:31.40,0:16.90,0:02.19
thunderbird,master:676fd3f394a0c0576b63c77ed2184bf5f42c29b6,,1:06:52,13:37.50,1:06:09,4:48.92,0:31.49,0:17.59,0:02.22
thunderbird,master:5cb59cc6910d8f3165528c4a71e29e4de897d242,,1:07:24,11:47.08,1:08:16,4:56.51,0:31.55,0:17.29,0:02.21
thunderbird,master:73dae11852c656adbf5a2b0e44ac4c7e0078cd8d,,1:06:25,11:47.57,1:17:12,4:53.33,0:31.27,0:16.94,0:02.22
thunderbird,master:73dae11852c656adbf5a2b0e44ac4c7e0078cd8d,,1:06:37,11:35.24,1:20:20,4:57.05,0:31.27,0:16.94,0:02.21
thunderbird,master:14010b4a60072641bab62c761edffc51148ae6e9,,1:06:40,17:00.83,1:17:01,5:13.92,0:31.54,0:16.87,0:02.21
thunderbird,master:a62aed41f2d8f874f7ae24d0e5be5dbc66ea2199,,1:06:19,16:02.83,1:15:05,4:54.64,0:31.35,0:18.24,0:02.19
thunderbird,master:a62aed41f2d8f874f7ae24d0e5be5dbc66ea2199,,1:06:36,11:50.66,1:18:42,4:56.03,0:31.22,0:16.85,0:02.19
thunderbird,master:a62aed41f2d8f874f7ae24d0e5be5dbc66ea2199,,1:06:21,11:23.33,1:06:18,4:56.64,0:31.32,0:16.87,0:02.23
thunderbird,master:7bf5c38e0f8bed9295f46773ade5336ec41044f6,,1:06:56,11:57.49,1:06:48,4:56.85,0:31.47,0:16.92,0:02.23
thunderbird,master:96b144a25598090b1e58eb5e33f216b5bfef7595,,1:06:48,11:37.51,1:07:05,6:46.44,0:32.94,0:18.59,0:02.23
thunderbird,master:96b144a25598090b1e58eb5e33f216b5bfef7595,,1:06:49,11:29.47,1:06:50,4:55.59,0:31.55,0:17.02,0:02.22
thunderbird,master:13bd8ad0214c645f67966c0a150644224522de18,1.5_M1.rc1-106-g13bd8ad,1:06:39,11:32.22,1:06:55,7:34.15,0:32.31,0:19.55,0:02.40,25G,4.7G
thunderbird,master:13bd8ad0214c645f67966c0a150644224522de18,1.5_M1.rc1-106-g13bd8ad,1:06:42,12:07.90,1:06:24,5:02.10,0:31.63,0:18.48,0:02.23,25G,4.7G
</tab>
 
== 1.4 Performance Data ==
 
 
Results:
 
[[File: Results1.png]]
 
 
[[File: Results2.png]]
 




Older results: [[Media: Results-feb.png]]
== Daily Test Results ==
Build performance test suite is run multiple times per day. Latest test reports for master branch can found here:
* [https://wiki.yoctoproject.org/charts/build_perf/latest/ypperf-ubuntu16_master_qemux86.html Ubuntu 16.04]
* [https://wiki.yoctoproject.org/charts/build_perf/latest/ypperf-ubuntu16-xeon_master_qemux86.html Ubuntu 16.04 (Intel® Xeon® HW)]
* [https://wiki.yoctoproject.org/charts/build_perf/latest/ypperf-centos7_master_qemux86.html CentOS 7]
* [https://wiki.yoctoproject.org/charts/build_perf/latest/ypperf-opensuse422_master_qemux86.html OpenSUSE Leap 42.2] (retired)
* [https://wiki.yoctoproject.org/charts/build_perf/latest/ypperf-fedora25_master_qemux86.html Fedora 25] (retired)
* [https://wiki.yoctoproject.org/charts/build_perf/latest/ypperf01_master_qemux86.html Fedora 23] (retired)
* [https://wiki.yoctoproject.org/charts/build_perf/latest/ypperf02_master_qemux86.html Ubuntu 15.10] (retired)


Compare two consecutive runs on System2: [[Media: Results_feb_2013.png]]
== Milestone Test results ==
Guadalajara Setup from 2.1 M4 to actual master


* [https://wiki.yoctoproject.org/charts/perf_milestone_GDC/performance_test.html GDC Charts]


== Old Test results ==


'''
Milestone test results from the retired Romania Setup (Yocto 1.6 – 2.1) can be found from here:


== Historical/obsolete data ==
* [https://wiki.yoctoproject.org/charts/perf_milestone/performance_test.html RO Charts]
'''


Raw test data from the retired Romania test setup has been moved to
* [[Performance_Test_Old_Results]]
* [[Performance_Test_Weekly_1.5]] has the old weekly results ran during the 1.5 cycle.


'''Poky/Bitbake Performance:'''<br>
Configuration for the Romania Setup 1.6 - 2.1:
Performance is an essential point for measuring the quality of a build system.<br>
This page is to track performance trend of poky build. Detailed development work is on https://wiki.yoctoproject.org/wiki/Performance.<br>
<br>
'''Hardware and software configuration:'''<br>
CPU: 4-core * 2-threads Intel(R) Core(TM) i7 CPU 860  @ 2.80GHz<br>
Memory: 4GB<br>
Harddisk: 1TB<br>
<br>
OS: Ubuntu 11.10 x86_64<br>
Kernel: 2.6.35-22<br>
<br>
local.conf:<br>
CONF_VERSION = "1"<br>
DL_DIR ?= "/sda1/sources/downloads" # Using a local download dir to avoid fetch.<br>
BB_NUMBER_THREADS = "8"<br>
PARALLEL_MAKE = "-j 8"<br>
MACHINE ?= "qemux86"<br>
DISTRO ?= "poky"<br>
BBMASK = ""<br>
EXTRA_IMAGE_FEATURES = "debug-tweaks"<br>
PACKAGE_CLASSES ?= "package_rpm"<br>
USER_CLASSES ?= "image-mklibs image-prelink"<br>
PACKAGE_DEBUG_SPLIT_STYLE = '.debug'<br>
BBINCLUDELOGS = "yes"<br>
ENABLE_BINARY_LOCALE_GENERATION = "1"<br>
NO32LIBS = "1"<br>
<br>


'''Build Performance Trend'''
* Ubuntu 12.04 (uname -nr: ubuntu12 3.8.0-31-generic) and
{|border="1"
* Fedora 19 (uname -nr: fedora19 3.11.3-201.fc19.x86_64)
|| '''Host''' ||'''[[Yocto1.3_M1-build]]'''|| '''[[Yocto1.2-M4-RC5-build]]'''|| '''[[Yocto1.2-M4-RC4-build]]'''|| '''[[Yocto1.2-M4-RC3-build]]'''|| '''[[Yocto1.2-M4-RC2-build]]'''|| '''[[Yocto1.2-M4-RC1-build]]'''|| '''[[Yocto1.2-M3-RC1-build]]''' || '''[[Yocto1.2-M2-RC1-build]]''' || '''[[Yocto1.2-M1-RC2-build]]'''|| '''[[Yocto1.1-M4-RC4-build]]'''|| '''[[Yocto1.1-M4-RC3-build]]''' || '''[[Yocto1.1-M4-RC2-build]]'''|| '''[[Yocto1.1-M2-RC3-build]] (both rpm and ipk built by default)'''
* HW configuration: SandyBridge i7-2600 CPU @ 3.40GHz (4 cores/8 threads), 8GB RAM, 1 x 500 GB HDD
|-
||Ubuntu11.10 x86_64 || real 94m57s <br> user 388m51.720s <br> sys 48m34.660s|| real 111m48.083s <br> user 381m29.410s <br> sys 46m52.080s || real 111m27.967s <br> user 383m32.690s <br> sys 46m41.830s || real 110m24.635s <br> user 381m29.740s <br> sys 46m32.430s || real 112m8.776s <br> user 379m10.670s <br> sys 47m19.270s || real 130m48.838s <br> user 370m56.000s <br> sys 45m36.060s || real 133m39.703s <br> user 371m37.450s <br> sys 46m57.510s || real 124m39s <br> user 373m16 <br> sys 52m17s || real 104m24s <br> user 374m16s <br> real 50m 37s || real 119m31s <br> user 355m3s <br> sys 47m42s  || real 127m33s <br> user 346m3s <br> sys 48m5s || real 124m54s <br> user 342m57s <br> sys 47m33s || real 109m53s <br> user 356m16s <br> sys 48m16s
|}

Latest revision as of 14:17, 28 January 2019

Build time tracking

Script used for running the builds:

  • Before morty: poky/scripts/contrib/build-perf-test.sh
  • After morty: poky/scripts/oe-build-perf-test

Two identical test systems are used for the results, with all updates available installed. Extra packages installed (besides what's by the default in the distro) are the ones in the Quick Start guide.

Configuration of the GDC Setup 2.1 - Current master

  • Ubuntu 15.10 (Wily Werewolf)" --> 4.2.0-16-generic
  • Fedora release 23 (Twenty Three) --> 4.7.10-100.fc23.x86_64
  • HW configuration: Intel(R) Core(TM) i7-6700 CPU @ 3.40GHz, 16GB RAM, 1 x 600 GB SDD, 1 x 2 TB HDD

All system/distro settings are distro's defaults, that includes:

- FS mount options: ext4 rw,relatime,data=ordered

- IO scheduler: cfq and CPU governer: ondemand

local.conf it's the default one (the script doesn't touch it except for changing DL_DIR if necessary).

It however exports these in the enviroment: 

BB_NUMBER_THREADS = "8"
PARALLEL_MAKE = "-j 8"
MACHINE = "qemux86"

FAQ:

  • Q: Where is 1.5_M2? It's there but it's called: 1.5_M1.final-412-geaa5df3 The M2 build wasn't released hence no git tag
  • Q: What's with the different number of RCs? See Yocto_Project_Release_Process#Milestone_Releases
  • Q: Why are there more results for some commits? The script is ran from a cron job multiple times a day, sometimes master changes daily sometimes it doesn't, hence more results for one commit (ie weekends)


Daily Test Results

Build performance test suite is run multiple times per day. Latest test reports for master branch can found here:

Milestone Test results

Guadalajara Setup from 2.1 M4 to actual master

Old Test results

Milestone test results from the retired Romania Setup (Yocto 1.6 – 2.1) can be found from here:

Raw test data from the retired Romania test setup has been moved to

Configuration for the Romania Setup 1.6 - 2.1:

  • Ubuntu 12.04 (uname -nr: ubuntu12 3.8.0-31-generic) and
  • Fedora 19 (uname -nr: fedora19 3.11.3-201.fc19.x86_64)
  • HW configuration: SandyBridge i7-2600 CPU @ 3.40GHz (4 cores/8 threads), 8GB RAM, 1 x 500 GB HDD
Retrieved from "https://wiki.yoctoproject.org/wiki/index.php?title=Performance_Test&oldid=46678"