TipsAndTricks/LockSharedState - Revision history

Stephano: reorg headers

2017-04-12T01:51:35Z

reorg headers

← Older revision		Revision as of 01:51, 12 April 2017
Line 1:		Line 1:
	==Overview==		=Overview=
	By design, the OpenEmbedded build system builds everything from scratch unless BitBake can determine that parts do not need to be rebuilt. The system makes this decision based on a checksum of the input to any given task and uses this as a digital signature. When using sstate cache, these signatures are compared and any difference will cause the task to be rerun. If we know that we want cached tasks to be used we can generate the needed signatures and configure bitbake to use those specific caches, regardless of any differences found.		By design, the OpenEmbedded build system builds everything from scratch unless BitBake can determine that parts do not need to be rebuilt. The system makes this decision based on a checksum of the input to any given task and uses this as a digital signature. When using sstate cache, these signatures are compared and any difference will cause the task to be rerun. If we know that we want cached tasks to be used we can generate the needed signatures and configure bitbake to use those specific caches, regardless of any differences found.

			=Locking Down SSTATE=

			These steps should guide you through locking the SSTATE cache to specific checksums. It also includes an optional "export" step. This can be helpful if you wish to distribute the SSTATE to others using the Yocto Project.

	==Generating Signatures==		==Generating Signatures==
Line 18:		Line 22:
	In this example we generate the hashes for all tasks required to build core-image-minimal and core-image-sato, however it can be any recipe or image.		In this example we generate the hashes for all tasks required to build core-image-minimal and core-image-sato, however it can be any recipe or image.

	==~~Copy~~ Required SSTATE Artifacts (optional)==		==Export Required SSTATE Artifacts (optional)==
	You may want to distribute the sstate, for example if you are generating a BSP and wish to supply pre-built artifacts. The gen-lockedsig-cache can be used to accomplish this:		You may want to distribute the sstate, for example if you are generating a BSP and wish to supply pre-built artifacts. The gen-lockedsig-cache can be used to accomplish this:

Line 32:		Line 36:
	NATIVELSBSTRING="universal"		NATIVELSBSTRING="universal"

	==~~Locking the Shared State Cache~~==		==Choose Tasks to Lock==
	As the locked-sigs.inc file implies, you will need to include this file in your configuration if you wish to lock the SSTATE. You can do so by using the '''include''' keyword in one of the included configuration files such as your distro's configuration file, or simply in the local.conf file (e.g. include locked-sigs.inc). Once you have done this, you will want to set the '''SIGGEN_LOCKEDSIGS_TASKSIG_CHECK''' variable to '''warn''' in order to ignore any errors. Settings include "error" and "warn", where error will stop the build and warn will not.		As the locked-sigs.inc file implies, you will need to include this file in your configuration if you wish to lock the SSTATE. You can do so by using the '''include''' keyword in one of the included configuration files such as your distro's configuration file, or simply in the local.conf file (e.g. include locked-sigs.inc). Once you have done this, you will want to set the '''SIGGEN_LOCKEDSIGS_TASKSIG_CHECK''' variable to '''warn''' in order to ignore any errors. Settings include "error" and "warn", where error will stop the build and warn will not.

Line 46:		Line 50:
	Now even if a task signatures have changed for some reason, Bitbake will use the shared state cache which matches the signatures you have provided. This allows you to force tasks to be cached, regardless of what has changed in your configuration.		Now even if a task signatures have changed for some reason, Bitbake will use the shared state cache which matches the signatures you have provided. This allows you to force tasks to be cached, regardless of what has changed in your configuration.

	==Pitfalls==		=Pitfalls=
	The downsides to this process should be obvious. By forcing sstate cache to be used regardless of input changes, the recipe metadata will no longer directly reflect the output of a bitbake command. This feature should be used with caution. If you find that signatures have changed and you cannot understand why, see the section on [[TipsAndTricks/Understanding_what_changed_(diffsigs_etc) \| understanding what changed]].		The downsides to this process should be obvious. By forcing sstate cache to be used regardless of input changes, the recipe metadata will no longer directly reflect the output of a bitbake command. This feature should be used with caution. If you find that signatures have changed and you cannot understand why, see the section on [[TipsAndTricks/Understanding_what_changed_(diffsigs_etc) \| understanding what changed]].

	==TODO==		=TODO=
	Update the manual with SIGGEN_LOCKEDSIGS & SIGGEN_LOCKEDSIGS_TASKSIG_CHECK definitions.''		Update the manual with SIGGEN_LOCKEDSIGS & SIGGEN_LOCKEDSIGS_TASKSIG_CHECK definitions.''

Stephano: update with reference to diffsigs

2017-04-11T21:04:04Z

update with reference to diffsigs

← Older revision		Revision as of 21:04, 11 April 2017
Line 47:		Line 47:

	==Pitfalls==		==Pitfalls==
	The downsides to this process should be obvious. By forcing sstate cache to be used regardless of input changes, the recipe metadata will no longer directly reflect the output of a bitbake command. This feature should be used with caution.		The downsides to this process should be obvious. By forcing sstate cache to be used regardless of input changes, the recipe metadata will no longer directly reflect the output of a bitbake command. This feature should be used with caution. If you find that signatures have changed and you cannot understand why, see the section on [[TipsAndTricks/Understanding_what_changed_(diffsigs_etc) \| understanding what changed]].

	==TODO==		==TODO==
	Update the manual with SIGGEN_LOCKEDSIGS & SIGGEN_LOCKEDSIGS_TASKSIG_CHECK definitions.''		Update the manual with SIGGEN_LOCKEDSIGS & SIGGEN_LOCKEDSIGS_TASKSIG_CHECK definitions.''

Stephano: Add overview and pitfall sections

2017-04-11T21:01:39Z

Add overview and pitfall sections

← Older revision		Revision as of 21:01, 11 April 2017
Line 1:		Line 1:
	==Generating ~~Hashes~~==		==Overview==
			By design, the OpenEmbedded build system builds everything from scratch unless BitBake can determine that parts do not need to be rebuilt. The system makes this decision based on a checksum of the input to any given task and uses this as a digital signature. When using sstate cache, these signatures are compared and any difference will cause the task to be rerun. If we know that we want cached tasks to be used we can generate the needed signatures and configure bitbake to use those specific caches, regardless of any differences found.

			==Generating Signatures==
	From the bitbake help docs:		From the bitbake help docs:
	-S SIGNATURE_HANDLER, --dump-signatures=SIGNATURE_HANDLER		-S SIGNATURE_HANDLER, --dump-signatures=SIGNATURE_HANDLER
Line 9:		Line 12:
	the dumped signature with the cached one.		the dumped signature with the cached one.

	In simpler terms, this function can generate the list of signatures that ~~bitbake~~ uses to determine if a task needs to be rerun. If we set the "handler" to '''none''', it will conveniently produce these ~~hashes~~ in a file called "locked-sigs.inc".		In simpler terms, this function can generate the list of signatures that Bitbake uses to determine if a task needs to be rerun. If we set the "handler" to '''none''', it will conveniently produce these signatures in a file called "locked-sigs.inc".

	$bitbake -S none core-image-minimal core-image-sato		$bitbake -S none core-image-minimal core-image-sato
Line 41:		Line 44:
	"		"

	Now even if a task ~~hash has~~ changed for some reason, ~~bitbake~~ will use the shared state cache of the ~~hash~~ you have provided. This allows you to force tasks to be cached, regardless of what has changed in your configuration.		Now even if a task signatures have changed for some reason, Bitbake will use the shared state cache which matches the signatures you have provided. This allows you to force tasks to be cached, regardless of what has changed in your configuration.

			==Pitfalls==
			The downsides to this process should be obvious. By forcing sstate cache to be used regardless of input changes, the recipe metadata will no longer directly reflect the output of a bitbake command. This feature should be used with caution.

	''TODO~~: note why this is dangerous and update~~ the manual with SIGGEN_LOCKEDSIGS_TASKSIG_CHECK ~~definition~~.''		==TODO==
			Update the manual with SIGGEN_LOCKEDSIGS & SIGGEN_LOCKEDSIGS_TASKSIG_CHECK definitions.''

Joshua Lock: /* Locking the Shared State Cache */ minor edit, the locked-sigs.inc should be included from a conf file, not a recipe

2017-04-05T09:23:50Z

Locking the Shared State Cache: minor edit, the locked-sigs.inc should be included from a conf file, not a recipe

← Older revision		Revision as of 09:23, 5 April 2017
Line 30:		Line 30:

	==Locking the Shared State Cache==		==Locking the Shared State Cache==
	As the locked-sigs.inc file implies, you will need to include this file in your configuration if you wish to lock the SSTATE. You can do so by using the '''include''' keyword in ~~any recipe~~, or simply in the local.conf file (e.g. include locked-sigs.inc). Once you have done this, you will want to set the '''SIGGEN_LOCKEDSIGS_TASKSIG_CHECK''' variable to '''warn''' in order to ignore any errors. Settings include "error" and "warn", where error will stop the build and warn will not.		As the locked-sigs.inc file implies, you will need to include this file in your configuration if you wish to lock the SSTATE. You can do so by using the '''include''' keyword in one of the included configuration files such as your distro's configuration file, or simply in the local.conf file (e.g. include locked-sigs.inc). Once you have done this, you will want to set the '''SIGGEN_LOCKEDSIGS_TASKSIG_CHECK''' variable to '''warn''' in order to ignore any errors. Settings include "error" and "warn", where error will stop the build and warn will not.

	Rather than using the entire file, you can pick and choose which tasks you wish to lock. For example, you may only need to copy a few tasks into the local.conf file:		Rather than using the entire file, you can pick and choose which tasks you wish to lock. For example, you may only need to copy a few tasks into the local.conf file:

Joshua Lock: /* Copy Required SSTATE Artifacts (optional) */ add a comment about nativelsbstring and unenative

2017-04-05T09:20:34Z

Copy Required SSTATE Artifacts (optional): add a comment about nativelsbstring and unenative

← Older revision		Revision as of 09:20, 5 April 2017
Line 24:		Line 24:
	gen-lockedsig-cache ./locked-sigs.inc /current/sstate-cache/ ~/new/sstate-cache universal		gen-lockedsig-cache ./locked-sigs.inc /current/sstate-cache/ ~/new/sstate-cache universal

	Your '''nativelsbstring''' variable can be found by running bitbake -e:		Your '''nativelsbstring''' variable can be found by running bitbake -e and will always be universal when using [http://www.yoctoproject.org/docs/2.2.1/ref-manual/ref-manual.html#ref-classes-uninative uninative] (the default in the Poky distribution since the Krogoth 2.1):

	$ bitbake -e \| grep NATIVELSBSTRING=		$ bitbake -e \| grep NATIVELSBSTRING=

Stephano: /* Locking the Shared State Cache - clarification */

2017-04-04T16:07:45Z

Locking the Shared State Cache - clarification

← Older revision		Revision as of 16:07, 4 April 2017
Line 30:		Line 30:

	==Locking the Shared State Cache==		==Locking the Shared State Cache==
	As the locked-sigs.inc file implies, you will need to include this file in your configuration if you wish to lock the SSTATE. You can do so by using the '''include''' keyword in any recipe, or simply in the local.conf file (e.g. include locked-sigs.inc). Once you have done this, you will want to set the '''SIGGEN_LOCKEDSIGS_TASKSIG_CHECK''' variable to '''warn''' in order to ignore any errors.		As the locked-sigs.inc file implies, you will need to include this file in your configuration if you wish to lock the SSTATE. You can do so by using the '''include''' keyword in any recipe, or simply in the local.conf file (e.g. include locked-sigs.inc). Once you have done this, you will want to set the '''SIGGEN_LOCKEDSIGS_TASKSIG_CHECK''' variable to '''warn''' in order to ignore any errors. Settings include "error" and "warn", where error will stop the build and warn will not.

	Rather than using the entire file, you can pick and choose which tasks you wish to lock. For example, you may only need to copy a few tasks into the local.conf file:		Rather than using the entire file, you can pick and choose which tasks you wish to lock. For example, you may only need to copy a few tasks into the local.conf file:

Stephano: /* Cleanup */

2017-04-03T22:49:33Z

Cleanup

← Older revision		Revision as of 22:49, 3 April 2017
Line 26:		Line 26:
	Your '''nativelsbstring''' variable can be found by running bitbake -e:		Your '''nativelsbstring''' variable can be found by running bitbake -e:

	bitbake -e \| grep NATIVELSBSTRING=		$ bitbake -e \| grep NATIVELSBSTRING=
	NATIVELSBSTRING="universal"		NATIVELSBSTRING="universal"

Stephano: /* cleanup */

2017-04-03T22:49:08Z

cleanup

← Older revision		Revision as of 22:49, 3 April 2017
Line 24:		Line 24:
	gen-lockedsig-cache ./locked-sigs.inc /current/sstate-cache/ ~/new/sstate-cache universal		gen-lockedsig-cache ./locked-sigs.inc /current/sstate-cache/ ~/new/sstate-cache universal

	Your '''nativelsbstring''' variable can be found by~~...~~		Your '''nativelsbstring''' variable can be found by running bitbake -e:

			bitbake -e \| grep NATIVELSBSTRING=
			NATIVELSBSTRING="universal"

	==Locking the Shared State Cache==		==Locking the Shared State Cache==

Stephano: /* Locking the Shared State Cache */

2017-04-03T22:28:11Z

Locking the Shared State Cache

← Older revision		Revision as of 22:28, 3 April 2017
Line 28:		Line 28:
	==Locking the Shared State Cache==		==Locking the Shared State Cache==
	As the locked-sigs.inc file implies, you will need to include this file in your configuration if you wish to lock the SSTATE. You can do so by using the '''include''' keyword in any recipe, or simply in the local.conf file (e.g. include locked-sigs.inc). Once you have done this, you will want to set the '''SIGGEN_LOCKEDSIGS_TASKSIG_CHECK''' variable to '''warn''' in order to ignore any errors.		As the locked-sigs.inc file implies, you will need to include this file in your configuration if you wish to lock the SSTATE. You can do so by using the '''include''' keyword in any recipe, or simply in the local.conf file (e.g. include locked-sigs.inc). Once you have done this, you will want to set the '''SIGGEN_LOCKEDSIGS_TASKSIG_CHECK''' variable to '''warn''' in order to ignore any errors.

			Rather than using the entire file, you can pick and choose which tasks you wish to lock. For example, you may only need to copy a few tasks into the local.conf file:

			SIGGEN_LOCKEDSIGS = "\
			gcc-cross:do_populate_sysroot:a8d91b35b98e1494957a2ddaf4598956 \
			eglibc:do_populate_sysroot:13e8c68553dc61f9d67564f13b9b2d67 \
			eglibc:do_packagedata:bfca0db1782c719d373f8636282596ee \
			gcc-cross:do_packagedata:4b601ff4f67601395ee49c46701122f6 \
			"

	Now even if a task hash has changed for some reason, bitbake will use the shared state cache of the hash you have provided. This allows you to force tasks to be cached, regardless of what has changed in your configuration.		Now even if a task hash has changed for some reason, bitbake will use the shared state cache of the hash you have provided. This allows you to force tasks to be cached, regardless of what has changed in your configuration.

	''TODO: note why this is dangerous and update the manual with SIGGEN_LOCKEDSIGS_TASKSIG_CHECK definition.''		''TODO: note why this is dangerous and update the manual with SIGGEN_LOCKEDSIGS_TASKSIG_CHECK definition.''

Stephano: /* cleanup */

2017-04-03T22:01:19Z

cleanup

← Older revision		Revision as of 22:01, 3 April 2017
Line 19:		Line 19:

	$ gen-lockedsig-cache		$ gen-lockedsig-cache
	syntax: gen-lockedsig-cache <locked-sigs.inc> <input-cachedir> <output-cachedir> <nativelsbstring> [filterfile]		syntax: gen-lockedsig-cache <locked-sigs.inc> <input-cachedir> <output-cachedir> <nativelsbstring> [filterfile]
	e.g.
	gen-lockedsig-cache ./locked-sigs.inc /current/sstate-cache/ ~/new/sstate-cache universal		e.g.
			gen-lockedsig-cache ./locked-sigs.inc /current/sstate-cache/ ~/new/sstate-cache universal

	Your '''nativelsbstring''' variable can be found by...		Your '''nativelsbstring''' variable can be found by...