Configure Cache Effects on Bash Build - Revision history

Bavery at 06:18, 10 November 2016

2016-11-10T06:18:46Z

← Older revision		Revision as of 06:18, 10 November 2016
Line 73:		Line 73:
	</code>		</code>

	This isn't how I'd recommend doing this by any means, but a couple of awk/sed scripts let me ~~extrace~~ and automate the creation of the above so it was pretty painless.		This isn't how I'd recommend doing this by any means, but a couple of awk/sed scripts let me extract and automate the creation of the above so it was pretty painless.

	== The Results ==		== The Results ==

Bavery at 06:15, 10 November 2016

2016-11-10T06:15:50Z

← Older revision		Revision as of 06:15, 10 November 2016
Line 1:		Line 1:
	== Building Bash - The experiment ==		== Building Bash - The experiment ==
	Why bash? Because it has enough pieces to make it interesting (~20-30 minutes on my 36 core workstation) and does not build an image, so I can ignore the rpm install/pseudo part of the build.		Why bash? Because it has enough pieces to make it interesting (~20-30 minutes on my 36 core workstation) and does not build an image, so I can ignore the rpm install/pseudo part of the build.
	I did:		== What I did ==
			Each build:
	<code>		<code>
	. ./poky/oe-init-build-env		. ./poky/oe-init-build-env
	bitbake bash		bitbake bash
	</code>		</code>
	The upstream code was in the downloads directory and I <code>rm -rf sstate-cache tmp cache </code> between each ~~run~~.		The upstream code was in the downloads directory and I <code>rm -rf sstate-cache tmp cache </code> between each build.
			<br>
	~~== What I did ==~~
	I have 4 builds, 2 with configure cache's for all the recipes and 2 without. I then used <code> buildstats-diff --multi </code> to compare the averages for cache vs non cache.		I have 4 builds, 2 with configure cache's for all the recipes and 2 without. I then used <code> buildstats-diff --multi </code> to compare the averages for cache vs non cache.
	To get the recipe specific caches I have the following in a conf/auto.conf file:		To get the recipe specific caches I have the following in a conf/auto.conf file:
Line 78:		Line 78:
	=== walltime ===		=== walltime ===
	<code>		<code>
			$ buildstats-diff --multi --diff-attr walltime tmp.bash.nocache.2runs/buildstats tmp.bash.cache.2runs/buildstats
	Ignoring tasks less than 00:05.0 (5.0s)		Ignoring tasks less than 00:05.0 (5.0s)
	Ignoring differences less than 00:02.0 (2.0s)		Ignoring differences less than 00:02.0 (2.0s)
Line 156:		Line 157:
	=== cputime ===		=== cputime ===
	<code>		<code>
			$ buildstats-diff --multi --diff-attr cputime tmp.bash.nocache.2runs/buildstats tmp.bash.cache.2runs/buildstats
	Ignoring tasks less than 00:03.0 (3.0s)		Ignoring tasks less than 00:03.0 (3.0s)
	Ignoring differences less than 00:01.0 (1.0s)		Ignoring differences less than 00:01.0 (1.0s)

Bavery: /* Building Bash - The experiment */

2016-11-10T06:13:04Z

Building Bash - The experiment

← Older revision		Revision as of 06:13, 10 November 2016
Line 3:		Line 3:
	I did:		I did:
	<code>		<code>
	. ./poky/oe-init-~~buildenv~~		. ./poky/oe-init-build-env
	bitbake bash		bitbake bash
	</code>		</code>

Bavery: /* Building Bash - The experiment */

2016-11-10T06:12:21Z

Building Bash - The experiment

← Older revision		Revision as of 06:12, 10 November 2016
Line 1:		Line 1:
	== Building Bash - The experiment ==		== Building Bash - The experiment ==
	Why bash? Because it has enough pieces to make it interesting (~20-30 minutes on my 36 core workstation) and does not build an image, so I can ignore the rpm install/pseudo part of the build.		Why bash? Because it has enough pieces to make it interesting (~20-30 minutes on my 36 core workstation) and does not build an image, so I can ignore the rpm install/pseudo part of the build.
			I did:
			<code>
			. ./poky/oe-init-buildenv
			bitbake bash
			</code>
			The upstream code was in the downloads directory and I <code>rm -rf sstate-cache tmp cache </code> between each run.

	== What I did ==		== What I did ==
	I have 4 builds, 2 with configure cache's for all the recipes and 2 without. I then used <code> buildstats-diff --multi </code> to compare the averages for cache vs non cache.		I have 4 builds, 2 with configure cache's for all the recipes and 2 without. I then used <code> buildstats-diff --multi </code> to compare the averages for cache vs non cache.

Bavery at 06:10, 10 November 2016

2016-11-10T06:10:42Z

@@ Line 3: / Line 3: @@
 == What I did ==
 I have 4 builds, 2 with configure cache's for all the recipes and 2 without. I then used <code> buildstats-diff --multi </code> to compare the averages for cache vs non cache.
 == The Results ==
 === walltime ===
@@ Line 157: / Line 222: @@
    -338.7s    -3.7%    2:32:59.4 (9179.4s) -> 2:27:20.7 (8840.7s)
 </code>

Bavery at 06:02, 10 November 2016

2016-11-10T06:02:45Z

← Older revision		Revision as of 06:02, 10 November 2016
Line 82:		Line 82:


	=== ~~walltime~~ ===		=== cputime ===
			<code>
			Ignoring tasks less than 00:03.0 (3.0s)
			Ignoring differences less than 00:01.0 (1.0s)

			PKG TASK ABSDIFF RELDIFF CPUTIME1 -> CPUTIME2
			gettext-native do_configure -47.1s -38.2% 123.2s -> 76.1s
			gettext do_configure -35.0s -29.2% 119.9s -> 84.9s
			rpm-native do_configure -29.8s -36.2% 82.4s -> 52.5s
			bison-native do_configure -24.8s -53.4% 46.3s -> 21.6s
			m4-native do_configure -17.6s -91.3% 19.2s -> 1.7s
			python-native do_configure -16.8s -50.8% 33.2s -> 16.3s
			diffutils do_configure -14.9s -44.6% 33.4s -> 18.5s
			db do_configure -12.3s -82.5% 14.9s -> 2.6s
			pkgconfig-native do_configure -11.1s -33.6% 33.1s -> 22.0s
			python do_configure -9.9s -38.3% 26.0s -> 16.0s
			ncurses-native do_configure -9.8s -67.3% 14.6s -> 4.8s
			sed do_configure -9.7s -51.6% 18.8s -> 9.1s
			db-native do_configure -8.3s -77.0% 10.8s -> 2.5s
			m4 do_configure -7.0s -35.1% 19.8s -> 12.9s
			gmp-native do_configure -6.9s -31.8% 21.6s -> 14.7s
			bash do_configure -6.6s -54.3% 12.1s -> 5.5s
			ncurses do_configure -5.5s -47.2% 11.7s -> 6.2s
			file-native do_configure -5.1s -37.3% 13.6s -> 8.5s
			xz-native do_configure -4.3s -25.7% 16.9s -> 12.5s
			flex-native do_configure -3.7s -26.2% 14.3s -> 10.6s
			gawk do_configure -3.7s -20.9% 17.7s -> 14.0s
			readline-native do_configure -3.6s -55.1% 6.5s -> 2.9s
			mpfr-native do_configure -3.5s -26.9% 13.1s -> 9.6s
			flex do_configure -3.2s -22.5% 14.4s -> 11.1s
			libtool-native do_configure -3.2s -27.1% 11.8s -> 8.6s
			gdbm do_configure -3.1s -23.9% 12.9s -> 9.8s
			cross-localedef-native do_configure -3.0s -50.0% 6.0s -> 3.0s
			libpcre-native do_configure -2.8s -22.2% 12.6s -> 9.8s
			ossp-uuid-native do_configure -2.8s -31.8% 8.7s -> 5.9s
			xproto-native do_configure -2.7s -39.3% 6.9s -> 4.2s
			makedepend-native do_configure -2.5s -36.9% 6.7s -> 4.2s
			make do_configure -2.4s -29.7% 8.2s -> 5.8s
			popt-native do_configure -2.2s -20.1% 11.2s -> 8.9s
			perl-native do_configure -2.2s -5.6% 39.9s -> 37.7s
			elfutils-native do_configure -2.2s -26.6% 8.2s -> 6.0s
			sqlite3-native do_configure -2.1s -28.5% 7.3s -> 5.2s
			readline do_configure -1.9s -37.4% 5.2s -> 3.2s
			db do_package_write_rpm -1.9s -5.9% 32.8s -> 30.8s
			libtool-cross do_configure -1.9s -17.9% 10.7s -> 8.8s
			prelink-native do_configure -1.9s -17.4% 10.8s -> 8.9s
			libffi do_configure -1.8s -15.1% 12.2s -> 10.3s
			expat do_configure -1.7s -16.9% 10.0s -> 8.3s
			bc do_configure -1.6s -29.1% 5.6s -> 3.9s
			sqlite3 do_configure -1.6s -22.4% 7.2s -> 5.5s
			gmp-native do_compile -1.6s -3.3% 48.0s -> 46.5s
			expat-native do_configure -1.6s -17.0% 9.2s -> 7.6s
			perl-native do_install -1.5s -3.8% 40.6s -> 39.1s
			libmpc-native do_configure -1.5s -14.3% 10.6s -> 9.1s
			glibc do_install -1.3s -0.9% 141.5s -> 140.2s
			attr-native do_configure -1.2s -15.7% 7.9s -> 6.7s
			bzip2-native do_configure -1.2s -19.6% 6.1s -> 4.9s
			gcc-cross-initial-x86_64 do_compile -1.0s -0.1% 824.5s -> 823.4s
			db-native do_compile -1.0s -1.6% 61.9s -> 60.9s
			bzip2 do_configure -1.0s -15.8% 6.3s -> 5.3s
			perl-native do_populate_sysroot 1.0s +12.2% 8.5s -> 9.5s
			perl do_compile 1.4s +0.5% 307.1s -> 308.5s
			binutils-native do_compile 1.8s +0.4% 442.8s -> 444.6s
			openssl do_install 1.9s +4.0% 46.6s -> 48.5s
			gcc-runtime do_compile 1.9s +0.9% 213.0s -> 215.0s
			ncurses-native do_compile 2.0s +2.0% 102.7s -> 104.8s
			ncurses do_compile 2.2s +1.9% 116.4s -> 118.6s
			gcc-cross-x86_64 do_compile 2.2s +0.2% 913.4s -> 915.6s
			binutils-cross-x86_64 do_compile 2.8s +0.7% 388.3s -> 391.1s
			glibc-locale do_package_write_rpm 3.0s +1.0% 294.4s -> 297.5s
			glibc-locale do_package 12.9s +3.0% 427.0s -> 439.9s

			Cumulative cputime:
			-338.7s -3.7% 2:32:59.4 (9179.4s) -> 2:27:20.7 (8840.7s)
			</code>

Bavery: Created page with "== Building Bash - The experiment == Why bash? Because it has enough pieces to make it interesting (~20-30 minutes on my 36 core workstation) and does not build an image, so I..."

2016-11-10T06:00:49Z

Created page with "== Building Bash - The experiment == Why bash? Because it has enough pieces to make it interesting (~20-30 minutes on my 36 core workstation) and does not build an image, so I..."

New page

== Building Bash - The experiment ==
Why bash? Because it has enough pieces to make it interesting (~20-30 minutes on my 36 core workstation) and does not build an image, so I can ignore the rpm install/pseudo part of the build.
== What I did ==
I have 4 builds, 2 with configure cache's for all the recipes and 2 without. I then used <code> buildstats-diff --multi </code> to compare the averages for cache vs non cache.
== The Results ==
=== walltime ===
<code>
Ignoring tasks less than 00:05.0 (5.0s)
Ignoring differences less than 00:02.0 (2.0s)

PKG TASK ABSDIFF RELDIFF WALLTIME1 -> WALLTIME2
gettext-native do_configure -63.7s -33.6% 189.8s -> 126.1s
gettext do_configure -45.5s -29.5% 154.0s -> 108.5s
bison-native do_configure -41.2s -61.4% 67.2s -> 26.0s
m4-native do_configure -40.3s -79.7% 50.6s -> 10.3s
rpm-native do_configure -36.2s -34.1% 106.2s -> 70.0s
ncurses-native do_configure -26.4s -70.1% 37.6s -> 11.2s
python-native do_configure -21.5s -49.3% 43.7s -> 22.2s
pkgconfig-native do_configure -17.1s -32.7% 52.2s -> 35.1s
db do_configure -16.7s -78.9% 21.2s -> 4.5s
diffutils do_configure -16.6s -39.0% 42.5s -> 25.9s
mpfr-native do_configure -15.5s -54.3% 28.6s -> 13.1s
sed do_configure -13.5s -49.5% 27.2s -> 13.8s
python do_configure -12.6s -36.0% 35.0s -> 22.4s
db-native do_configure -11.9s -73.4% 16.2s -> 4.3s
sqlite3-native do_configure -11.3s -57.9% 19.5s -> 8.2s
m4 do_configure -10.6s -36.8% 28.8s -> 18.2s
bash do_configure -10.0s -53.2% 18.9s -> 8.8s
xproto-native do_configure -9.7s -58.0% 16.7s -> 7.0s
file-native do_configure -9.1s -39.6% 23.0s -> 13.9s
gmp-native do_configure -8.2s -24.1% 34.1s -> 25.8s
readline-native do_configure -7.1s -55.9% 12.7s -> 5.6s
linux-libc-headers do_install -7.0s -35.1% 20.0s -> 13.0s
libtool-native do_configure -6.3s -24.3% 26.0s -> 19.7s
gawk do_configure -6.2s -24.1% 25.8s -> 19.6s
flex do_configure -5.9s -28.0% 20.9s -> 15.1s
libpcre-native do_configure -5.7s -27.1% 21.1s -> 15.4s
elfutils-native do_configure -5.7s -36.1% 15.7s -> 10.0s
ncurses do_configure -5.6s -29.5% 19.0s -> 13.4s
expat-native do_configure -5.5s -31.1% 17.6s -> 12.1s
gettext-native do_compile -4.8s -12.1% 39.3s -> 34.6s
prelink-native do_configure -4.6s -26.8% 17.0s -> 12.5s
gdbm do_configure -4.1s -24.1% 17.2s -> 13.0s
flex-native do_configure -4.1s -20.5% 19.9s -> 15.9s
xz-native do_configure -3.9s -17.5% 22.6s -> 18.7s
gmp-native do_compile -3.7s -30.4% 12.1s -> 8.4s
cross-localedef-native do_configure -3.5s -37.3% 9.5s -> 6.0s
make do_configure -3.2s -27.4% 11.5s -> 8.4s
ossp-uuid-native do_configure -2.9s -23.0% 12.8s -> 9.9s
perl-native do_configure -2.8s -4.5% 62.7s -> 59.9s
libtool-cross do_configure -2.7s -17.6% 15.1s -> 12.5s
ncurses-native do_populate_sysroot -2.5s -30.5% 8.1s -> 5.6s
readline do_configure -2.4s -32.0% 7.6s -> 5.1s
openssl-native do_install -2.3s -4.7% 50.0s -> 47.6s
bash do_package 2.1s +44.6% 4.7s -> 6.8s
gcc-source-6.2.0 do_patch 2.3s +38.9% 6.0s -> 8.4s
bash do_patch 2.5s +50.7% 4.9s -> 7.3s
openssl do_patch 2.5s +49.0% 5.0s -> 7.5s
glibc-locale do_package_write_rpm 2.5s +0.8% 298.4s -> 301.0s
glibc do_unpack 2.8s +33.7% 8.2s -> 11.0s
binutils-native do_unpack 3.0s +38.4% 7.8s -> 10.8s
ncurses-native do_install 3.3s +8.7% 37.8s -> 41.1s
rpm-native do_patch 3.4s +22.8% 15.0s -> 18.4s
elfutils-native do_patch 4.0s +85.7% 4.6s -> 8.6s
db do_unpack 4.1s +271.1% 1.5s -> 5.7s
libtool-native do_compile 4.2s +58.0% 7.2s -> 11.4s
automake-native do_populate_sysroot 4.4s +695.3% 0.6s -> 5.0s
glibc-initial do_patch 4.5s +91.0% 5.0s -> 9.5s
openssl-native do_patch 4.6s +88.2% 5.2s -> 9.7s
binutils-native do_fetch 5.0s +143.5% 3.5s -> 8.5s
cross-localedef-native do_unpack 5.0s +78.2% 6.4s -> 11.4s
perl-native do_compile 5.2s +16.4% 31.7s -> 36.9s
binutils-cross-x86_64 do_compile 6.7s +11.0% 60.7s -> 67.4s
gcc-source-6.2.0 do_unpack 7.8s +18.6% 42.2s -> 50.0s
binutils-native do_compile 8.3s +14.6% 57.2s -> 65.5s
ncurses-native do_compile 9.1s +80.1% 11.4s -> 20.5s
gettext-native do_install 13.1s +68.6% 19.0s -> 32.1s

Cumulative walltime:
-434.5s -7.9% 1:31:26.8 (5486.8s) -> 1:24:12.2 (5052.2s)
</code>

=== walltime ===