Enable sstate cache: Difference between revisions

From Yocto Project
Jump to navigationJump to search
(→‎Use local sstate cache: Removing warning about race condition as it is no longer accurate.)
(Fix for changes in 1.3+)
 
Line 1: Line 1:
== sstate background ==
== sstate background ==
To speed up the build process, sstate provides a cache mechanism, where sstate files from server can be reused to avoid build from scratch if the producer and consumer of the sstate has same environment. With perfect case, we can achieve more than 80% time decreasing.
To speed up the build process, sstate provides a cache mechanism, where sstate files from server can be reused to avoid build from scratch if the producer and consumer of the sstate have the same environment. With perfect case, we can achieve more than 80% time decreasing.


== use sstate cache server ==
== use sstate cache server ==
Another build machine is needed to produce the sstate file periodically. The default sstate files directory is '''sstate-cache''' under build dir, which is needed to be exported via NFS, http or ftp server.
Another build machine is needed to produce the sstate file periodically. The default sstate files directory is '''sstate-cache''' under build dir, which is needed to be exported via NFS, http or ftp server.


On your local machine, change the conf/local.conf as following:
On your local machine, change the conf/local.conf as follows:


SSTATE_MIRRORS ?= "\
<pre><nowiki>
file://.* http://someserver.tld/share/sstate/"
SSTATE_MIRRORS ?= "\
file://.* http://someserver.tld/share/sstate/PATH"
</nowiki></pre>


for NFS server
for an NFS server:
SSTATE_MIRRORS ?= "\
file://.* file:///local/mounted/dir/sstate/"


<pre><nowiki>
SSTATE_MIRRORS ?= "\
file://.* file:///local/mounted/dir/sstate/PATH"
</nowiki></pre>
The special "PATH" suffix is important as it is automatically expanded to the correct value for native packages (which are split by the host distribution being used).




Line 34: Line 40:
If there is no setscene tasks running, sstate cache can't be used because sstate function got broken or some environment value changes invalidate sstate cache. You can use following tools to check any changes between sstate cache and new produced one.
If there is no setscene tasks running, sstate cache can't be used because sstate function got broken or some environment value changes invalidate sstate cache. You can use following tools to check any changes between sstate cache and new produced one.


  #bitbake-diffsigs /mnt/sstate/sstate-packagename-checksum1-tgz.siginfo builddir/sstate-cache/sstate-packagename-checksum2-tgz.siginfo
If you wish to compare signatures to see any changes that caused a rebuild since the last built version, you can use <code>bitbake-diffsigs</code> with the <code>-t</code> option:
 
$ bitbake-diffsigs -t quilt-native do_configure
 
Alternatively you can compare siginfo files directly:


$ bitbake-diffsigs /mnt/sstate/sstate-packagename-checksum1-tgz.siginfo builddir/sstate-cache/sstate-packagename-checksum2-tgz.siginfo


== know issues ==
== Known issues ==
* Various build distributions may lead to checksum mismatch
* New changeset may invalidate part or all of the sstate cache occasionally - this is by design
* New changeset may invalidate whole sstate cache occasionally
* Native package’s sstate cache can’t be reused among 32bit and 64bit host
* Native package’s sstate cache can’t be reused among 32bit and 64bit host
* Missing sstate file on http/ftp server cause wget hang for a long time due to the retries and timeout
* Missing sstate file on http/ftp server cause wget hang for a long time due to the retries and timeout

Latest revision as of 11:23, 8 November 2013

sstate background

To speed up the build process, sstate provides a cache mechanism, where sstate files from server can be reused to avoid build from scratch if the producer and consumer of the sstate have the same environment. With perfect case, we can achieve more than 80% time decreasing.

use sstate cache server

Another build machine is needed to produce the sstate file periodically. The default sstate files directory is sstate-cache under build dir, which is needed to be exported via NFS, http or ftp server.

On your local machine, change the conf/local.conf as follows:

SSTATE_MIRRORS ?= "\
file://.* http://someserver.tld/share/sstate/PATH"

for an NFS server:

SSTATE_MIRRORS ?= "\
file://.* file:///local/mounted/dir/sstate/PATH"

The special "PATH" suffix is important as it is automatically expanded to the correct value for native packages (which are split by the host distribution being used).


Use local sstate cache

You can make all the builds pointing to the same SSTATE_DIR to share sstate files between them, like this:

SSTATE_DIR ?= "/share/sstate-cache"

So that each build would consume the sstate file if match, and produce new sstate file if not match.

Verify the sstate is working

If sstate works, you can see something like:

NOTE: Preparing runqueue
NOTE: Executing SetScene Tasks
NOTE: Running setscene task 118 of 155 (virtual:native:/home/lulianhao/poky-build/edwin/poky/meta/recipes-devtools/pseudo/pseudo_git.bb:do_populate_sysroot_setscene)
NOTE: Running setscene task 119 of 155 (/home/lulianhao/poky-build/edwin/poky/meta/recipes-devtools/quilt/quilt-native_0.48.bb:do_populate_sysroot_setscene)

If there is no setscene tasks running, sstate cache can't be used because sstate function got broken or some environment value changes invalidate sstate cache. You can use following tools to check any changes between sstate cache and new produced one.

If you wish to compare signatures to see any changes that caused a rebuild since the last built version, you can use bitbake-diffsigs with the -t option:

$ bitbake-diffsigs -t quilt-native do_configure

Alternatively you can compare siginfo files directly:

$ bitbake-diffsigs /mnt/sstate/sstate-packagename-checksum1-tgz.siginfo builddir/sstate-cache/sstate-packagename-checksum2-tgz.siginfo

Known issues

  • New changeset may invalidate part or all of the sstate cache occasionally - this is by design
  • Native package’s sstate cache can’t be reused among 32bit and 64bit host
  • Missing sstate file on http/ftp server cause wget hang for a long time due to the retries and timeout