Build Performance - Revision history

RossBurton at 15:33, 11 October 2022

2022-10-11T15:33:07Z

← Older revision		Revision as of 15:33, 11 October 2022
Line 1:		Line 1:
			'''Note that this is fairly old and may be out of date'''.

	This page is intended to serve as a guide to tweaking your build system and bitbake configuration for optimal build time as well as to provide some guidance on how to collect build metrics and identify bottlenecks.		This page is intended to serve as a guide to tweaking your build system and bitbake configuration for optimal build time as well as to provide some guidance on how to collect build metrics and identify bottlenecks.

Bavery at 05:55, 10 November 2016

2016-11-10T05:55:14Z

← Older revision		Revision as of 05:55, 10 November 2016
Line 44:		Line 44:

	== discussion ==		== discussion ==
	Build system performance is a recurring mailing list topic. [http://lists.linuxtogo.org/pipermail/openembedded-core/2011-September/010179.html Here] is Richard Purdie discussing the challenges of improving many-core build performance and package dependency issues.		Build system performance is a recurring mailing list topic. [http://lists.linuxtogo.org/pipermail/openembedded-core/2011-September/010179.html Here] [http://lists.openembedded.org/pipermail/openembedded-core/2011-September/049160.html maybe there] is Richard Purdie discussing the challenges of improving many-core build performance and package dependency issues. The first link fails, possibly because of database renumbering?.
			<br>
			As it is recurring, here is Richard Purdie discussing [http://lists.openembedded.org/pipermail/openembedded-core/2015-November/113519.html a build performance analysis chronicle] in 2016.
			<br>
			Since the "performance analysis chronicle" focused on configure to a large extent, I was curious as to how/if the use of configure cache's would affect a build. The results from a limited experiment on building bash, are viewable on [[Configure Cache Effects on Bash Build]].

BjornStenberg at 07:26, 19 April 2012

2012-04-19T07:26:47Z

← Older revision		Revision as of 07:26, 19 April 2012
Line 34:		Line 34:
	If you enable buildstats (by adding <tt>INHERIT += "buildstats"</tt> to local.conf) you will get detailed build performance log data in tmp/buildstats.		If you enable buildstats (by adding <tt>INHERIT += "buildstats"</tt> to local.conf) you will get detailed build performance log data in tmp/buildstats.

	Richard Purdie has [https://lists.yoctoproject.org/pipermail/poky/2011-February/003740.html modified pybootbootchartgui] to parse this data and produce pretty charts ~~like above~~.		Richard Purdie has [https://lists.yoctoproject.org/pipermail/poky/2011-February/003740.html modified pybootbootchartgui] to parse this data and produce [http://tim.rpsys.net/bootchart.png pretty charts].

	== parallelism ==		== parallelism ==

BjornStenberg at 13:09, 16 April 2012

2012-04-16T13:09:37Z

← Older revision		Revision as of 13:09, 16 April 2012
Line 37:		Line 37:

	== parallelism ==		== parallelism ==
	Small builds do not scale to a large number of cores, due to package inter-dependencies. Going far beyond 8 is not giving much benefit. ~~See this~~ graph of ~~different build parameters on~~ a 64-core machine ~~when~~ building core-image-minimal:		Small builds do not scale to a large number of cores, due to package inter-dependencies. Going far beyond 8 is not giving much benefit. Here is a graph of a 64-core machine building core-image-minimal with different build parameters:

	[[File:Bb-matrix-f3-64.png]]		[[File:Bb-matrix-f3-64.png]]

BjornStenberg at 13:03, 16 April 2012

2012-04-16T13:03:22Z

← Older revision		Revision as of 13:03, 16 April 2012
Line 40:		Line 40:

	[[File:Bb-matrix-f3-64.png]]		[[File:Bb-matrix-f3-64.png]]

			Bigger builds are less hampered by linear dependency chains, and fill the cores better.

	== discussion ==		== discussion ==
	Build system performance is a recurring mailing list topic. [http://lists.linuxtogo.org/pipermail/openembedded-core/2011-September/010179.html Here] is Richard Purdie discussing the challenges of improving many-core build performance and package dependency issues.		Build system performance is a recurring mailing list topic. [http://lists.linuxtogo.org/pipermail/openembedded-core/2011-September/010179.html Here] is Richard Purdie discussing the challenges of improving many-core build performance and package dependency issues.

BjornStenberg at 13:01, 16 April 2012

2012-04-16T13:01:45Z

← Older revision		Revision as of 13:01, 16 April 2012
Line 37:		Line 37:

	== parallelism ==		== parallelism ==
	~~The small~~ builds ~~does~~ not scale to large number of cores, due to package inter-dependencies. Going far beyond 8 is not giving much benefit. See this graph ~~showing the speed difference in using~~ different build parameters on a 64-core machine when building core-image-minimal:		Small builds do not scale to a large number of cores, due to package inter-dependencies. Going far beyond 8 is not giving much benefit. See this graph of different build parameters on a 64-core machine when building core-image-minimal:

	[[File:Bb-matrix-f3-64.png]]		[[File:Bb-matrix-f3-64.png]]

BjornStenberg at 12:59, 16 April 2012

2012-04-16T12:59:46Z

← Older revision		Revision as of 12:59, 16 April 2012
Line 10:		Line 10:
	* Mount using the options "noatime,barrier=0,commit=6000"		* Mount using the options "noatime,barrier=0,commit=6000"
	* Use a tmpfs for /tmp		* Use a tmpfs for /tmp
	* The build does not scale well to large number of cores, due to package inter-dependencies. Going much beyond 10-12 is not giving much benefit.

	== bb-matrix ==		== bb-matrix ==
Line 35:		Line 34:
	If you enable buildstats (by adding <tt>INHERIT += "buildstats"</tt> to local.conf) you will get detailed build performance log data in tmp/buildstats.		If you enable buildstats (by adding <tt>INHERIT += "buildstats"</tt> to local.conf) you will get detailed build performance log data in tmp/buildstats.

	Richard Purdie has [https://lists.yoctoproject.org/pipermail/poky/2011-February/003740.html modified pybootbootchartgui] to parse this data and produce a pretty ~~chart~~. ([~~http~~:~~//tim.rpsys.net/bootchart~~.png ~~example~~])		Richard Purdie has [https://lists.yoctoproject.org/pipermail/poky/2011-February/003740.html modified pybootbootchartgui] to parse this data and produce pretty charts like above.

			== parallelism ==
			The small builds does not scale to large number of cores, due to package inter-dependencies. Going far beyond 8 is not giving much benefit. See this graph showing the speed difference in using different build parameters on a 64-core machine when building core-image-minimal:

			[[File:Bb-matrix-f3-64.png]]

	== discussion ==		== discussion ==
	Build system performance is a recurring mailing list topic. [http://lists.linuxtogo.org/pipermail/openembedded-core/2011-September/010179.html Here] is Richard Purdie discussing the challenges of improving many-core build performance and package dependency issues.		Build system performance is a recurring mailing list topic. [http://lists.linuxtogo.org/pipermail/openembedded-core/2011-September/010179.html Here] is Richard Purdie discussing the challenges of improving many-core build performance and package dependency issues.

BjornStenberg at 11:41, 13 April 2012

2012-04-13T11:41:25Z

← Older revision		Revision as of 11:41, 13 April 2012
Line 35:		Line 35:
	If you enable buildstats (by adding <tt>INHERIT += "buildstats"</tt> to local.conf) you will get detailed build performance log data in tmp/buildstats.		If you enable buildstats (by adding <tt>INHERIT += "buildstats"</tt> to local.conf) you will get detailed build performance log data in tmp/buildstats.

	Richard Purdie has [https://lists.yoctoproject.org/pipermail/poky/2011-February/003740.html modified ~~bootchart~~] to produce pretty ~~charts showing this data graphically~~. ([http://tim.rpsys.net/bootchart.png example])		Richard Purdie has [https://lists.yoctoproject.org/pipermail/poky/2011-February/003740.html modified pybootbootchartgui] to parse this data and produce a pretty chart. ([http://tim.rpsys.net/bootchart.png example])

	== discussion ==		== discussion ==
	Build system performance is a recurring mailing list topic. [http://lists.linuxtogo.org/pipermail/openembedded-core/2011-September/010179.html Here] is Richard Purdie discussing the challenges of improving many-core build performance and package dependency issues.		Build system performance is a recurring mailing list topic. [http://lists.linuxtogo.org/pipermail/openembedded-core/2011-September/010179.html Here] is Richard Purdie discussing the challenges of improving many-core build performance and package dependency issues.

BjornStenberg at 10:34, 13 April 2012

2012-04-13T10:34:02Z

← Older revision		Revision as of 10:34, 13 April 2012
Line 8:		Line 8:
	* Use the ext4 filesystem for the build disk		* Use the ext4 filesystem for the build disk
	* Turn off journaling for ext4 ("tune2fs -O ^has_journal <disk>")		* Turn off journaling for ext4 ("tune2fs -O ^has_journal <disk>")
	* Mount using the options "noatime~~,nodiratime~~,barrier=0,commit=6000"		* Mount using the options "noatime,barrier=0,commit=6000"
	* Use a tmpfs for /tmp		* Use a tmpfs for /tmp
			* The build does not scale well to large number of cores, due to package inter-dependencies. Going much beyond 10-12 is not giving much benefit.

	== bb-matrix ==		== bb-matrix ==
Line 30:		Line 31:

	[[File:Bb-matrix-f3.png]]		[[File:Bb-matrix-f3.png]]

			== buildstats ==
			If you enable buildstats (by adding <tt>INHERIT += "buildstats"</tt> to local.conf) you will get detailed build performance log data in tmp/buildstats.

			Richard Purdie has [https://lists.yoctoproject.org/pipermail/poky/2011-February/003740.html modified bootchart] to produce pretty charts showing this data graphically. ([http://tim.rpsys.net/bootchart.png example])

			== discussion ==
			Build system performance is a recurring mailing list topic. [http://lists.linuxtogo.org/pipermail/openembedded-core/2011-September/010179.html Here] is Richard Purdie discussing the challenges of improving many-core build performance and package dependency issues.

Rpurdie: Add general build tips

2011-09-27T10:19:12Z

Add general build tips

← Older revision		Revision as of 10:19, 27 September 2011
Line 1:		Line 1:
	This page is intended to serve as a guide to tweaking your build system and bitbake configuration for optimal build time as well as to provide some guidance on how to collect build metrics and identify bottlenecks.		This page is intended to serve as a guide to tweaking your build system and bitbake configuration for optimal build time as well as to provide some guidance on how to collect build metrics and identify bottlenecks.

			== General Tips ==

			Some general tips on how to ensure good performance of builds:

			* Put the build directory on its own disk. This is good practice in its own right since the build system has a tendency to wear disks heavily.
			* Use the ext4 filesystem for the build disk
			* Turn off journaling for ext4 ("tune2fs -O ^has_journal <disk>")
			* Mount using the options "noatime,nodiratime,barrier=0,commit=6000"
			* Use a tmpfs for /tmp

	== bb-matrix ==		== bb-matrix ==