TipsAndTricks/LockSharedState: Difference between revisions

From Yocto Project
Jump to navigationJump to search
(reorg headers)
 
(6 intermediate revisions by 2 users not shown)
Line 1: Line 1:
==Generating Hashes==
=Overview=
By design, the OpenEmbedded build system builds everything from scratch unless BitBake can determine that parts do not need to be rebuilt. The system makes this decision based on a checksum of the input to any given task and uses this as a digital signature. When using sstate cache, these signatures are compared and any difference will cause the task to be rerun. If we know that we want cached tasks to be used we can generate the needed signatures and configure bitbake to use those specific caches, regardless of any differences found.
 
=Locking Down SSTATE=
 
These steps should guide you through locking the SSTATE cache to specific checksums. It also includes an optional "export" step. This can be helpful if you wish to distribute the SSTATE to others using the Yocto Project.
 
==Generating Signatures==
From the bitbake help docs:
From the bitbake help docs:
               -S SIGNATURE_HANDLER, --dump-signatures=SIGNATURE_HANDLER
               -S SIGNATURE_HANDLER, --dump-signatures=SIGNATURE_HANDLER
Line 9: Line 16:
                         the dumped signature with the cached one.
                         the dumped signature with the cached one.


In simpler terms, this function can generate the list of signatures that bitbake uses to determine if a task needs to be rerun. If we set the "handler" to '''none''', it will conveniently produce these hashes in a file called "locked-sigs.inc".
In simpler terms, this function can generate the list of signatures that Bitbake uses to determine if a task needs to be rerun. If we set the "handler" to '''none''', it will conveniently produce these signatures in a file called "locked-sigs.inc".


     $bitbake -S none core-image-minimal core-image-sato
     $bitbake -S none core-image-minimal core-image-sato
Line 15: Line 22:
In this example we generate the hashes for all tasks required to build core-image-minimal and core-image-sato, however it can be any recipe or image.
In this example we generate the hashes for all tasks required to build core-image-minimal and core-image-sato, however it can be any recipe or image.


==Copy Required SSTATE Artifacts (optional)==
==Export Required SSTATE Artifacts (optional)==
You may want to distribute the sstate, for example if you are generating a BSP and wish to supply pre-built artifacts. The gen-lockedsig-cache can be used to accomplish this:
You may want to distribute the sstate, for example if you are generating a BSP and wish to supply pre-built artifacts. The gen-lockedsig-cache can be used to accomplish this:


Line 24: Line 31:
   gen-lockedsig-cache ./locked-sigs.inc /current/sstate-cache/ ~/new/sstate-cache universal
   gen-lockedsig-cache ./locked-sigs.inc /current/sstate-cache/ ~/new/sstate-cache universal


Your '''nativelsbstring''' variable can be found by running bitbake -e:
Your '''nativelsbstring''' variable can be found by running bitbake -e and will always be universal when using [http://www.yoctoproject.org/docs/2.2.1/ref-manual/ref-manual.html#ref-classes-uninative uninative] (the default in the Poky distribution since the Krogoth 2.1):


   bitbake -e | grep NATIVELSBSTRING=
   $ bitbake -e | grep NATIVELSBSTRING=
   NATIVELSBSTRING="universal"
   NATIVELSBSTRING="universal"


==Locking the Shared State Cache==
==Choose Tasks to Lock==
As the locked-sigs.inc file implies, you will need to include this file in your configuration if you wish to lock the SSTATE. You can do so by using the '''include''' keyword in any recipe, or simply in the local.conf file (e.g. include locked-sigs.inc). Once you have done this, you will want to set the '''SIGGEN_LOCKEDSIGS_TASKSIG_CHECK''' variable to '''warn''' in order to ignore any errors.
As the locked-sigs.inc file implies, you will need to include this file in your configuration if you wish to lock the SSTATE. You can do so by using the '''include''' keyword in one of the included configuration files such as your distro's configuration file, or simply in the local.conf file (e.g. include locked-sigs.inc). Once you have done this, you will want to set the '''SIGGEN_LOCKEDSIGS_TASKSIG_CHECK''' variable to '''warn''' in order to ignore any errors. Settings include "error" and "warn", where error will stop the build and warn will not.


Rather than using the entire file, you can pick and choose which tasks you wish to lock. For example, you may only need to copy a few tasks into the local.conf file:
Rather than using the entire file, you can pick and choose which tasks you wish to lock. For example, you may only need to copy a few tasks into the local.conf file:
Line 41: Line 48:
  "
  "


Now even if a task hash has changed for some reason, bitbake will use the shared state cache of the hash you have provided. This allows you to force tasks to be cached, regardless of what has changed in your configuration.  
Now even if a task signatures have changed for some reason, Bitbake will use the shared state cache which matches the signatures you have provided. This allows you to force tasks to be cached, regardless of what has changed in your configuration.
 
=Pitfalls=
The downsides to this process should be obvious. By forcing sstate cache to be used regardless of input changes, the recipe metadata will no longer directly reflect the output of a bitbake command. This feature should be used with caution. If you find that signatures have changed and you cannot understand why, see the section on [[TipsAndTricks/Understanding_what_changed_(diffsigs_etc) | understanding what changed]].


''TODO: note why this is dangerous and update the manual with SIGGEN_LOCKEDSIGS_TASKSIG_CHECK definition.''
=TODO=
Update the manual with SIGGEN_LOCKEDSIGS & SIGGEN_LOCKEDSIGS_TASKSIG_CHECK definitions.''

Latest revision as of 01:51, 12 April 2017

Overview

By design, the OpenEmbedded build system builds everything from scratch unless BitBake can determine that parts do not need to be rebuilt. The system makes this decision based on a checksum of the input to any given task and uses this as a digital signature. When using sstate cache, these signatures are compared and any difference will cause the task to be rerun. If we know that we want cached tasks to be used we can generate the needed signatures and configure bitbake to use those specific caches, regardless of any differences found.

Locking Down SSTATE

These steps should guide you through locking the SSTATE cache to specific checksums. It also includes an optional "export" step. This can be helpful if you wish to distribute the SSTATE to others using the Yocto Project.

Generating Signatures

From the bitbake help docs:

             -S SIGNATURE_HANDLER, --dump-signatures=SIGNATURE_HANDLER
                       Dump out the signature construction information, with
                       no task execution. The SIGNATURE_HANDLER parameter is
                       passed to the handler. Two common values are none and
                       printdiff but the handler may define more/less. none
                       means only dump the signature, printdiff means compare
                       the dumped signature with the cached one.

In simpler terms, this function can generate the list of signatures that Bitbake uses to determine if a task needs to be rerun. If we set the "handler" to none, it will conveniently produce these signatures in a file called "locked-sigs.inc".

   $bitbake -S none core-image-minimal core-image-sato

In this example we generate the hashes for all tasks required to build core-image-minimal and core-image-sato, however it can be any recipe or image.

Export Required SSTATE Artifacts (optional)

You may want to distribute the sstate, for example if you are generating a BSP and wish to supply pre-built artifacts. The gen-lockedsig-cache can be used to accomplish this:

 $ gen-lockedsig-cache
 syntax: gen-lockedsig-cache <locked-sigs.inc> <input-cachedir> <output-cachedir> <nativelsbstring> [filterfile]
 
 e.g.
 gen-lockedsig-cache ./locked-sigs.inc /current/sstate-cache/ ~/new/sstate-cache universal

Your nativelsbstring variable can be found by running bitbake -e and will always be universal when using uninative (the default in the Poky distribution since the Krogoth 2.1):

 $ bitbake -e | grep NATIVELSBSTRING=
 NATIVELSBSTRING="universal"

Choose Tasks to Lock

As the locked-sigs.inc file implies, you will need to include this file in your configuration if you wish to lock the SSTATE. You can do so by using the include keyword in one of the included configuration files such as your distro's configuration file, or simply in the local.conf file (e.g. include locked-sigs.inc). Once you have done this, you will want to set the SIGGEN_LOCKEDSIGS_TASKSIG_CHECK variable to warn in order to ignore any errors. Settings include "error" and "warn", where error will stop the build and warn will not.

Rather than using the entire file, you can pick and choose which tasks you wish to lock. For example, you may only need to copy a few tasks into the local.conf file:

SIGGEN_LOCKEDSIGS = "\
gcc-cross:do_populate_sysroot:a8d91b35b98e1494957a2ddaf4598956 \
eglibc:do_populate_sysroot:13e8c68553dc61f9d67564f13b9b2d67 \
eglibc:do_packagedata:bfca0db1782c719d373f8636282596ee \
gcc-cross:do_packagedata:4b601ff4f67601395ee49c46701122f6 \
"

Now even if a task signatures have changed for some reason, Bitbake will use the shared state cache which matches the signatures you have provided. This allows you to force tasks to be cached, regardless of what has changed in your configuration.

Pitfalls

The downsides to this process should be obvious. By forcing sstate cache to be used regardless of input changes, the recipe metadata will no longer directly reflect the output of a bitbake command. This feature should be used with caution. If you find that signatures have changed and you cannot understand why, see the section on understanding what changed.

TODO

Update the manual with SIGGEN_LOCKEDSIGS & SIGGEN_LOCKEDSIGS_TASKSIG_CHECK definitions.