Performance

From Yocto Project
Revision as of 03:15, 16 February 2011 by Dongxiao.xu (talk | contribs)
Jump to navigationJump to search

Poky/Bitbake Performance:
Performance is an essential point for measuring the quality of a build system.
We are continously working on improving poky/bitbake performance, including major parts of build time, disk footprint, and file parsing speed.
This page provides some performance optimization results. Part of the optimization patches have been accepted in master tree, while some others are pending for review.

The following optimization patches are mainly provided by:
Richard Purdie <richard.purdie@linuxfoundation.org>
Qing He <qing.he@intel.com>
Dongxiao Xu <dongxiao.xu@intel.com>

Hardware and software configuration:
CPU: 4-core * 2-threads Intel(R) Core(TM) i7 CPU 860 @ 2.80GHz
Memory: 4GB
Harddisk: 1TB

OS: Ubuntu 10.04 x86_64
Kernel: 2.6.32-21

local.conf:
CONF_VERSION = "1"
DL_DIR ?= "/sda1/sources/downloads" # Using a local download dir to avoid fetch.
BB_NUMBER_THREADS = "9"
PARALLEL_MAKE = "-j 9"
MACHINE ?= "qemux86"
DISTRO ?= "poky"
BBMASK = ""
EXTRA_IMAGE_FEATURES = "tools-debug tools-profile tools-testapps debug-tweaks"
EXTRA_IMAGE_FEATURES_c7x0 = "tools-testapps debug-tweaks"
EXTRA_IMAGE_FEATURES_mx31phy = "debug-tweaks"
EXTRA_IMAGE_FEATURES_mx31ads = "tools-testapps debug-tweaks"
PACKAGE_CLASSES ?= "package_rpm package_ipk"
USER_CLASSES ?= "image-prelink"
BBINCLUDELOGS = "yes"
ENABLE_BINARY_LOCALE_GENERATION = "1"
NO32LIBS = "1"

Source code and commits:
Green-3.3.1: http://autobuilder.yoctoproject.org/downloads/poky/poky-green-3.3.1.tar.bz2
Yocto-0.9: http://autobuilder.yoctoproject.org/downloads/poky/poky-laverne-4.0.tar.bz2
01/07/2011 build: commit b22e345e05efcc3f66278af8f09fb083afe32b68
02/14/2011 build: commit 6cb8fd6def4912e4aa76330649ba42a9ed2694fd

Other detail commit number for optimization will be annotated in the table note. See following.

Performance results:

Table 1: File Parsing Time Optmization Results
File parsing speed BB files Parsing time (s) Standardized BB files Standardized time (s) Performance compare
Green-3.3.1 925 16.5 770 13.7 100%
Yocto-0.9 844 44.8 770 40.9 298%
Optimization A (rm distro variables) 757 29.6 770 30.1 220%
01/07/2011 build 770 16.6 770 16.6 121%
Optimization B ("??=" re-implementation) 770 15.6 770 15.6 114%

Note:
Optimization A: commit: 53aff7d6775eb1c2c8f419f325b91c062d85eed5
01/07/2011 build: Latest master with parallel parsing mechanism
Optimization B: "??=" re-implementation. (Pending for review)


Table 2: Build Time Optmization Results
Build time (poky-image-minimal) Recipe number Build time (s) Standardized recipe number Standardized time (s) Performance compare
Green-3.3.1 88 32m57s 147 53m54s 100%
Yocto-0.9 147 88m50s 147 88m50s 165%
Exec (before pseudo wrapper) 147 114m6s 147 114m6s 212%
Fork (after pseudo wrapper) 147 99m52s 147 99m52s 185%
01/07/2011 build 147 134m39s 147 134m39s 250%
pipe buffer 1024 (before pipe buffer fix) 170 101m8s 147 87m27s 162%
pipe buffer 102400 (after pipe buffer fix) 170 97m32s 147 84m20s 156%
02/14/2011 build 227 75m10s 147 48m40s 90%

Note:
Exec (before pseudo wrapper): commit 84263dbf43eba5cd99ce59062cef807a966e7312
Fork (after pseudo wrapper): commit 995d4679d4bb6a28dd6fbdb1becc4231369311b7
pipe buffer 1024 (before pipe buffer fix): commit 754047b6ec01df5f3159cce93b17b8493d0af5e1
pipe buffer 102400 (after pipe buffer fix): commit 06c6db7929c75f576a395fb442abe447b833fc3b

Table 3: Disk Footprint Optmization Results (poky-image-minimal)
Disk space (poky-image-minimal) Recipe number Disk footprint (GB) Standardized recipe number Standardized disk footprint (GB) Performance compare
Green-3.3.1 88 7.6 147 12.7 100%
Yocto-0.9 147 29 147 29 228%
Optimization A (rm sstate-build-*) 147 24 147 24 189%
01/07/2011 build 147 24 147 24 189%
Optimization B (hard link files) 147 19 147 19 150%

Note:
Optimization A: commit: 7e7bb24b8512dd1452a11d8aeafc8dedcbc4d623).
Optimization B: Using hardlink to replace copy between "package-->packages-split", "image-->sysroot-destdir", and "sysroot-destdir-->/tmp/sysroots" (Pending for review).

There are some other data we may add later.
1) Using socket data accessing approach to avoid data re-parsing when fork/exec new bitbake processes.