TipsAndTricks/LockSharedState
Overview
By design, the OpenEmbedded build system builds everything from scratch unless BitBake can determine that parts do not need to be rebuilt. The system makes this decision based on a checksum of the input to any given task and uses this as a digital signature. When using sstate cache, these signatures are compared and any difference will cause the task to be rerun. If we know that we want cached tasks to be used we can generate the needed signatures and configure bitbake to use those specific caches, regardless of any differences found.
Locking Down SSTATE
These steps should guide you through locking the SSTATE cache to specific checksums. It also includes an optional "export" step. This can be helpful if you wish to distribute the SSTATE to others using the Yocto Project.
Generating Signatures
From the bitbake help docs:
-S SIGNATURE_HANDLER, --dump-signatures=SIGNATURE_HANDLER Dump out the signature construction information, with no task execution. The SIGNATURE_HANDLER parameter is passed to the handler. Two common values are none and printdiff but the handler may define more/less. none means only dump the signature, printdiff means compare the dumped signature with the cached one.
In simpler terms, this function can generate the list of signatures that Bitbake uses to determine if a task needs to be rerun. If we set the "handler" to none, it will conveniently produce these signatures in a file called "locked-sigs.inc".
$bitbake -S none core-image-minimal core-image-sato
In this example we generate the hashes for all tasks required to build core-image-minimal and core-image-sato, however it can be any recipe or image.
Export Required SSTATE Artifacts (optional)
You may want to distribute the sstate, for example if you are generating a BSP and wish to supply pre-built artifacts. The gen-lockedsig-cache can be used to accomplish this:
$ gen-lockedsig-cache syntax: gen-lockedsig-cache <locked-sigs.inc> <input-cachedir> <output-cachedir> <nativelsbstring> [filterfile] e.g. gen-lockedsig-cache ./locked-sigs.inc /current/sstate-cache/ ~/new/sstate-cache universal
Your nativelsbstring variable can be found by running bitbake -e and will always be universal when using uninative (the default in the Poky distribution since the Krogoth 2.1):
$ bitbake -e | grep NATIVELSBSTRING= NATIVELSBSTRING="universal"
Choose Tasks to Lock
As the locked-sigs.inc file implies, you will need to include this file in your configuration if you wish to lock the SSTATE. You can do so by using the include keyword in one of the included configuration files such as your distro's configuration file, or simply in the local.conf file (e.g. include locked-sigs.inc). Once you have done this, you will want to set the SIGGEN_LOCKEDSIGS_TASKSIG_CHECK variable to warn in order to ignore any errors. Settings include "error" and "warn", where error will stop the build and warn will not.
Rather than using the entire file, you can pick and choose which tasks you wish to lock. For example, you may only need to copy a few tasks into the local.conf file:
SIGGEN_LOCKEDSIGS = "\ gcc-cross:do_populate_sysroot:a8d91b35b98e1494957a2ddaf4598956 \ eglibc:do_populate_sysroot:13e8c68553dc61f9d67564f13b9b2d67 \ eglibc:do_packagedata:bfca0db1782c719d373f8636282596ee \ gcc-cross:do_packagedata:4b601ff4f67601395ee49c46701122f6 \ "
Now even if a task signatures have changed for some reason, Bitbake will use the shared state cache which matches the signatures you have provided. This allows you to force tasks to be cached, regardless of what has changed in your configuration.
Pitfalls
The downsides to this process should be obvious. By forcing sstate cache to be used regardless of input changes, the recipe metadata will no longer directly reflect the output of a bitbake command. This feature should be used with caution. If you find that signatures have changed and you cannot understand why, see the section on understanding what changed.
TODO
Update the manual with SIGGEN_LOCKEDSIGS & SIGGEN_LOCKEDSIGS_TASKSIG_CHECK definitions.