Configure Cache Effects on Bash Build

From Yocto Project
Revision as of 06:15, 10 November 2016 by Bavery (talk | contribs)
Jump to navigationJump to search

Building Bash - The experiment

Why bash? Because it has enough pieces to make it interesting (~20-30 minutes on my 36 core workstation) and does not build an image, so I can ignore the rpm install/pseudo part of the build.

What I did

Each build:

 
. ./poky/oe-init-build-env
bitbake bash

The upstream code was in the downloads directory and I rm -rf sstate-cache tmp cache between each build.
I have 4 builds, 2 with configure cache's for all the recipes and 2 without. I then used buildstats-diff --multi to compare the averages for cache vs non cache. To get the recipe specific caches I have the following in a conf/auto.conf file:


 EXTRA_OECONF_append_pn-gettext :=" --cache-file=/YP/config-cache/gettext-0.19.8.1.cache"
 EXTRA_OECONF_append_pn-libffi :=" --cache-file=/YP/config-cache/libffi-3.2.1.cache"
 EXTRA_OECONF_append_pn-libtool-cross :=" --cache-file=/YP/config-cache/libtool-cross-2.4.6.cache"
 EXTRA_OECONF_append_pn-gdbm :=" --cache-file=/YP/config-cache/gdbm-1.12.cache"
 EXTRA_OECONF_append_pn-flex :=" --cache-file=/YP/config-cache/flex-2.6.0.cache"
 EXTRA_OECONF_append_pn-sed :=" --cache-file=/YP/config-cache/sed-4.2.2.cache"
 EXTRA_OECONF_append_pn-sqlite3 :=" --cache-file=/YP/config-cache/sqlite3-3.14.1.cache"
 EXTRA_OECONF_append_pn-ncurses :=" --cache-file=/YP/config-cache/ncurses-6.0+20160625.cache"
 EXTRA_OECONF_append_pn-python :=" --cache-file=/YP/config-cache/python-2.7.12.cache"
 EXTRA_OECONF_append_pn-bash :=" --cache-file=/YP/config-cache/bash-4.3.30.cache"
 EXTRA_OECONF_append_pn-diffutils :=" --cache-file=/YP/config-cache/diffutils-3.4.cache"
 EXTRA_OECONF_append_pn-glibc :=" --cache-file=/YP/config-cache/glibc-2.24.cache"
 EXTRA_OECONF_append_pn-expat :=" --cache-file=/YP/config-cache/expat-2.2.0.cache"
 EXTRA_OECONF_append_pn-make :=" --cache-file=/YP/config-cache/make-4.2.1.cache"
 EXTRA_OECONF_append_pn-m4 :=" --cache-file=/YP/config-cache/m4-1.4.17.cache"
 EXTRA_OECONF_append_pn-bc :=" --cache-file=/YP/config-cache/bc-1.06.cache"
 EXTRA_OECONF_append_pn-bzip2 :=" --cache-file=/YP/config-cache/bzip2-1.0.6.cache"
 EXTRA_OECONF_append_pn-readline :=" --cache-file=/YP/config-cache/readline-6.3.cache"
 EXTRA_OECONF_append_pn-gawk :=" --cache-file=/YP/config-cache/gawk-4.1.3.cache"
 EXTRA_OECONF_append_pn-db :=" --cache-file=/YP/config-cache/db-6.0.35.cache"
 EXTRA_OECONF_append_pn-rpm-native :=" --cache-file=/YP/config-cache/rpm-native-5.4.16.cache"
 EXTRA_OECONF_append_pn-cross-localedef-native :=" --cache-file=/YP/config-cache/cross-localedef-native-2.24.cache"
 EXTRA_OECONF_append_pn-readline-native :=" --cache-file=/YP/config-cache/readline-native-6.3.cache"
 EXTRA_OECONF_append_pn-attr-native :=" --cache-file=/YP/config-cache/attr-native-2.4.47.cache"
 EXTRA_OECONF_append_pn-libmpc-native :=" --cache-file=/YP/config-cache/libmpc-native-1.0.3.cache"
 EXTRA_OECONF_append_pn-db-native :=" --cache-file=/YP/config-cache/db-native-6.0.35.cache"
 EXTRA_OECONF_append_pn-m4-native :=" --cache-file=/YP/config-cache/m4-native-1.4.17.cache"
 EXTRA_OECONF_append_pn-binutils-native :=" --cache-file=/YP/config-cache/binutils-native-2.27.cache"
 EXTRA_OECONF_append_pn-gettext-native :=" --cache-file=/YP/config-cache/gettext-native-0.19.8.1.cache"
 EXTRA_OECONF_append_pn-mpfr-native :=" --cache-file=/YP/config-cache/mpfr-native-3.1.4.cache"
 EXTRA_OECONF_append_pn-byacc-native :=" --cache-file=/YP/config-cache/byacc-native-20160606.cache"
 EXTRA_OECONF_append_pn-gperf-native :=" --cache-file=/YP/config-cache/gperf-native-3.0.4.cache"
 EXTRA_OECONF_append_pn-bison-native :=" --cache-file=/YP/config-cache/bison-native-3.0.4.cache"
 EXTRA_OECONF_append_pn-prelink-native :=" --cache-file=/YP/config-cache/prelink-native-1.0+gitAUTOINC+ef20628dd7.cache"
 EXTRA_OECONF_append_pn-sqlite3-native :=" --cache-file=/YP/config-cache/sqlite3-native-3.14.1.cache"
 EXTRA_OECONF_append_pn-python-native :=" --cache-file=/YP/config-cache/python-native-2.7.12.cache"
 EXTRA_OECONF_append_pn-libpcre-native :=" --cache-file=/YP/config-cache/libpcre-native-8.39.cache"
 EXTRA_OECONF_append_pn-gcc-cross-initial-x86_64 :=" --cache-file=/YP/config-cache/gcc-cross-initial-x86_64-6.2.0.cache"
 EXTRA_OECONF_append_pn-flex-native :=" --cache-file=/YP/config-cache/flex-native-2.6.0.cache"
 EXTRA_OECONF_append_pn-util-macros-native :=" --cache-file=/YP/config-cache/util-macros-native-1.19.0.cache"
 EXTRA_OECONF_append_pn-gmp-native :=" --cache-file=/YP/config-cache/gmp-native-6.1.1.cache"
 EXTRA_OECONF_append_pn-elfutils-native :=" --cache-file=/YP/config-cache/elfutils-native-0.166.cache"
 EXTRA_OECONF_append_pn-pkgconfig-native :=" --cache-file=/YP/config-cache/pkgconfig-native-0.29.1+gitAUTOINC+87152c05be.cache"
 EXTRA_OECONF_append_pn-ossp-uuid-native :=" --cache-file=/YP/config-cache/ossp-uuid-native-1.6.2.cache"
 EXTRA_OECONF_append_pn-file-native :=" --cache-file=/YP/config-cache/file-native-5.28.cache"
 EXTRA_OECONF_append_pn-bzip2-native :=" --cache-file=/YP/config-cache/bzip2-native-1.0.6.cache"
 EXTRA_OECONF_append_pn-binutils-cross-x86_64 :=" --cache-file=/YP/config-cache/binutils-cross-x86_64-2.27.cache"
 EXTRA_OECONF_append_pn-quilt-native :=" --cache-file=/YP/config-cache/quilt-native-0.64.cache"
 EXTRA_OECONF_append_pn-xproto-native :=" --cache-file=/YP/config-cache/xproto-native-7.0.31.cache"
 EXTRA_OECONF_append_pn-acl-native :=" --cache-file=/YP/config-cache/acl-native-2.2.52.cache"
 EXTRA_OECONF_append_pn-xz-native :=" --cache-file=/YP/config-cache/xz-native-5.2.2.cache"
 EXTRA_OECONF_append_pn-popt-native :=" --cache-file=/YP/config-cache/popt-native-1.16.cache"
 EXTRA_OECONF_append_pn-makedepend-native :=" --cache-file=/YP/config-cache/makedepend-native-1.0.5.cache"
 EXTRA_OECONF_append_pn-expat-native :=" --cache-file=/YP/config-cache/expat-native-2.2.0.cache"
 EXTRA_OECONF_append_pn-automake-native :=" --cache-file=/YP/config-cache/automake-native-1.15.cache"
 EXTRA_OECONF_append_pn-autoconf-native :=" --cache-file=/YP/config-cache/autoconf-native-2.69.cache"
 EXTRA_OECONF_append_pn-gcc-cross-x86_64 :=" --cache-file=/YP/config-cache/gcc-cross-x86_64-6.2.0.cache"
 EXTRA_OECONF_append_pn-libtool-native :=" --cache-file=/YP/config-cache/libtool-native-2.4.6.cache"
 EXTRA_OECONF_append_pn-ncurses-native :=" --cache-file=/YP/config-cache/ncurses-native-6.0+20160625.cache

This isn't how I'd recommend doing this by any means, but a couple of awk/sed scripts let me extrace and automate the creation of the above so it was pretty painless.

The Results

walltime

$ buildstats-diff --multi --diff-attr walltime  tmp.bash.nocache.2runs/buildstats tmp.bash.cache.2runs/buildstats
Ignoring tasks less than 00:05.0 (5.0s)
Ignoring differences less than 00:02.0 (2.0s)
 PKG                       TASK                  ABSDIFF  RELDIFF  WALLTIME1 -> WALLTIME2
 gettext-native            do_configure           -63.7s   -33.6%     189.8s -> 126.1s
 gettext                   do_configure           -45.5s   -29.5%     154.0s -> 108.5s
 bison-native              do_configure           -41.2s   -61.4%      67.2s -> 26.0s
 m4-native                 do_configure           -40.3s   -79.7%      50.6s -> 10.3s
 rpm-native                do_configure           -36.2s   -34.1%     106.2s -> 70.0s
 ncurses-native            do_configure           -26.4s   -70.1%      37.6s -> 11.2s
 python-native             do_configure           -21.5s   -49.3%      43.7s -> 22.2s
 pkgconfig-native          do_configure           -17.1s   -32.7%      52.2s -> 35.1s
 db                        do_configure           -16.7s   -78.9%      21.2s -> 4.5s
 diffutils                 do_configure           -16.6s   -39.0%      42.5s -> 25.9s
 mpfr-native               do_configure           -15.5s   -54.3%      28.6s -> 13.1s
 sed                       do_configure           -13.5s   -49.5%      27.2s -> 13.8s
 python                    do_configure           -12.6s   -36.0%      35.0s -> 22.4s
 db-native                 do_configure           -11.9s   -73.4%      16.2s -> 4.3s
 sqlite3-native            do_configure           -11.3s   -57.9%      19.5s -> 8.2s
 m4                        do_configure           -10.6s   -36.8%      28.8s -> 18.2s
 bash                      do_configure           -10.0s   -53.2%      18.9s -> 8.8s
 xproto-native             do_configure            -9.7s   -58.0%      16.7s -> 7.0s
 file-native               do_configure            -9.1s   -39.6%      23.0s -> 13.9s
 gmp-native                do_configure            -8.2s   -24.1%      34.1s -> 25.8s
 readline-native           do_configure            -7.1s   -55.9%      12.7s -> 5.6s
 linux-libc-headers        do_install              -7.0s   -35.1%      20.0s -> 13.0s
 libtool-native            do_configure            -6.3s   -24.3%      26.0s -> 19.7s
 gawk                      do_configure            -6.2s   -24.1%      25.8s -> 19.6s
 flex                      do_configure            -5.9s   -28.0%      20.9s -> 15.1s
 libpcre-native            do_configure            -5.7s   -27.1%      21.1s -> 15.4s
 elfutils-native           do_configure            -5.7s   -36.1%      15.7s -> 10.0s
 ncurses                   do_configure            -5.6s   -29.5%      19.0s -> 13.4s
 expat-native              do_configure            -5.5s   -31.1%      17.6s -> 12.1s
 gettext-native            do_compile              -4.8s   -12.1%      39.3s -> 34.6s
 prelink-native            do_configure            -4.6s   -26.8%      17.0s -> 12.5s
 gdbm                      do_configure            -4.1s   -24.1%      17.2s -> 13.0s
 flex-native               do_configure            -4.1s   -20.5%      19.9s -> 15.9s
 xz-native                 do_configure            -3.9s   -17.5%      22.6s -> 18.7s
 gmp-native                do_compile              -3.7s   -30.4%      12.1s -> 8.4s
 cross-localedef-native    do_configure            -3.5s   -37.3%       9.5s -> 6.0s
 make                      do_configure            -3.2s   -27.4%      11.5s -> 8.4s
 ossp-uuid-native          do_configure            -2.9s   -23.0%      12.8s -> 9.9s
 perl-native               do_configure            -2.8s    -4.5%      62.7s -> 59.9s
 libtool-cross             do_configure            -2.7s   -17.6%      15.1s -> 12.5s
 ncurses-native            do_populate_sysroot     -2.5s   -30.5%       8.1s -> 5.6s
 readline                  do_configure            -2.4s   -32.0%       7.6s -> 5.1s 
 openssl-native            do_install              -2.3s    -4.7%      50.0s -> 47.6s
 bash                      do_package               2.1s   +44.6%       4.7s -> 6.8s 
 gcc-source-6.2.0          do_patch                 2.3s   +38.9%       6.0s -> 8.4s 
 bash                      do_patch                 2.5s   +50.7%       4.9s -> 7.3s
 openssl                   do_patch                 2.5s   +49.0%       5.0s -> 7.5s
 glibc-locale              do_package_write_rpm     2.5s    +0.8%     298.4s -> 301.0s
 glibc                     do_unpack                2.8s   +33.7%       8.2s -> 11.0s 
 binutils-native           do_unpack                3.0s   +38.4%       7.8s -> 10.8s 
 ncurses-native            do_install               3.3s    +8.7%      37.8s -> 41.1s
 rpm-native                do_patch                 3.4s   +22.8%      15.0s -> 18.4s
 elfutils-native           do_patch                 4.0s   +85.7%       4.6s -> 8.6s
 db                        do_unpack                4.1s  +271.1%       1.5s -> 5.7s 
 libtool-native            do_compile               4.2s   +58.0%       7.2s -> 11.4s
 automake-native           do_populate_sysroot      4.4s  +695.3%       0.6s -> 5.0s 
 glibc-initial             do_patch                 4.5s   +91.0%       5.0s -> 9.5s 
 openssl-native            do_patch                 4.6s   +88.2%       5.2s -> 9.7s
 binutils-native           do_fetch                 5.0s  +143.5%       3.5s -> 8.5s
 cross-localedef-native    do_unpack                5.0s   +78.2%       6.4s -> 11.4s 
 perl-native               do_compile               5.2s   +16.4%      31.7s -> 36.9s 
 binutils-cross-x86_64     do_compile               6.7s   +11.0%      60.7s -> 67.4s 
 gcc-source-6.2.0          do_unpack                7.8s   +18.6%      42.2s -> 50.0s 
 binutils-native           do_compile               8.3s   +14.6%      57.2s -> 65.5s 
 ncurses-native            do_compile               9.1s   +80.1%      11.4s -> 20.5s
 gettext-native            do_install              13.1s   +68.6%      19.0s -> 32.1s
Cumulative walltime:
 -434.5s    -7.9%    1:31:26.8 (5486.8s) -> 1:24:12.2 (5052.2s)


cputime


$ buildstats-diff --multi --diff-attr cputime  tmp.bash.nocache.2runs/buildstats tmp.bash.cache.2runs/buildstats
Ignoring tasks less than 00:03.0 (3.0s)
Ignoring differences less than 00:01.0 (1.0s)
 PKG                         TASK                  ABSDIFF  RELDIFF  CPUTIME1 -> CPUTIME2
 gettext-native              do_configure           -47.1s   -38.2%    123.2s -> 76.1s
 gettext                     do_configure           -35.0s   -29.2%    119.9s -> 84.9s
 rpm-native                  do_configure           -29.8s   -36.2%     82.4s -> 52.5s
 bison-native                do_configure           -24.8s   -53.4%     46.3s -> 21.6s
 m4-native                   do_configure           -17.6s   -91.3%     19.2s -> 1.7s
 python-native               do_configure           -16.8s   -50.8%     33.2s -> 16.3s
 diffutils                   do_configure           -14.9s   -44.6%     33.4s -> 18.5s
 db                          do_configure           -12.3s   -82.5%     14.9s -> 2.6s
 pkgconfig-native            do_configure           -11.1s   -33.6%     33.1s -> 22.0s
 python                      do_configure            -9.9s   -38.3%     26.0s -> 16.0s
 ncurses-native              do_configure            -9.8s   -67.3%     14.6s -> 4.8s
 sed                         do_configure            -9.7s   -51.6%     18.8s -> 9.1s
 db-native                   do_configure            -8.3s   -77.0%     10.8s -> 2.5s
 m4                          do_configure            -7.0s   -35.1%     19.8s -> 12.9s
 gmp-native                  do_configure            -6.9s   -31.8%     21.6s -> 14.7s
 bash                        do_configure            -6.6s   -54.3%     12.1s -> 5.5s
 ncurses                     do_configure            -5.5s   -47.2%     11.7s -> 6.2s
 file-native                 do_configure            -5.1s   -37.3%     13.6s -> 8.5s
 xz-native                   do_configure            -4.3s   -25.7%     16.9s -> 12.5s
 flex-native                 do_configure            -3.7s   -26.2%     14.3s -> 10.6s
 gawk                        do_configure            -3.7s   -20.9%     17.7s -> 14.0s
 readline-native             do_configure            -3.6s   -55.1%      6.5s -> 2.9s
 mpfr-native                 do_configure            -3.5s   -26.9%     13.1s -> 9.6s
 flex                        do_configure            -3.2s   -22.5%     14.4s -> 11.1s
 libtool-native              do_configure            -3.2s   -27.1%     11.8s -> 8.6s
 gdbm                        do_configure            -3.1s   -23.9%     12.9s -> 9.8s
 cross-localedef-native      do_configure            -3.0s   -50.0%      6.0s -> 3.0s
 libpcre-native              do_configure            -2.8s   -22.2%     12.6s -> 9.8s
 ossp-uuid-native            do_configure            -2.8s   -31.8%      8.7s -> 5.9s
 xproto-native               do_configure            -2.7s   -39.3%      6.9s -> 4.2s
 makedepend-native           do_configure            -2.5s   -36.9%      6.7s -> 4.2s
 make                        do_configure            -2.4s   -29.7%      8.2s -> 5.8s
 popt-native                 do_configure            -2.2s   -20.1%     11.2s -> 8.9s
 perl-native                 do_configure            -2.2s    -5.6%     39.9s -> 37.7s
 elfutils-native             do_configure            -2.2s   -26.6%      8.2s -> 6.0s
 sqlite3-native              do_configure            -2.1s   -28.5%      7.3s -> 5.2s
 readline                    do_configure            -1.9s   -37.4%      5.2s -> 3.2s
 db                          do_package_write_rpm    -1.9s    -5.9%     32.8s -> 30.8s
 libtool-cross               do_configure            -1.9s   -17.9%     10.7s -> 8.8s
 prelink-native              do_configure            -1.9s   -17.4%     10.8s -> 8.9s
 libffi                      do_configure            -1.8s   -15.1%     12.2s -> 10.3s
 expat                       do_configure            -1.7s   -16.9%     10.0s -> 8.3s 
 bc                          do_configure            -1.6s   -29.1%      5.6s -> 3.9s 
 sqlite3                     do_configure            -1.6s   -22.4%      7.2s -> 5.5s
 gmp-native                  do_compile              -1.6s    -3.3%     48.0s -> 46.5s
 expat-native                do_configure            -1.6s   -17.0%      9.2s -> 7.6s 
 perl-native                 do_install              -1.5s    -3.8%     40.6s -> 39.1s
 libmpc-native               do_configure            -1.5s   -14.3%     10.6s -> 9.1s
 glibc                       do_install              -1.3s    -0.9%    141.5s -> 140.2s
 attr-native                 do_configure            -1.2s   -15.7%      7.9s -> 6.7s
 bzip2-native                do_configure            -1.2s   -19.6%      6.1s -> 4.9s 
 gcc-cross-initial-x86_64    do_compile              -1.0s    -0.1%    824.5s -> 823.4s 
 db-native                   do_compile              -1.0s    -1.6%     61.9s -> 60.9s
 bzip2                       do_configure            -1.0s   -15.8%      6.3s -> 5.3s 
 perl-native                 do_populate_sysroot      1.0s   +12.2%      8.5s -> 9.5s
 perl                        do_compile               1.4s    +0.5%    307.1s -> 308.5s
 binutils-native             do_compile               1.8s    +0.4%    442.8s -> 444.6s
 openssl                     do_install               1.9s    +4.0%     46.6s -> 48.5s
 gcc-runtime                 do_compile               1.9s    +0.9%    213.0s -> 215.0s 
 ncurses-native              do_compile               2.0s    +2.0%    102.7s -> 104.8s 
 ncurses                     do_compile               2.2s    +1.9%    116.4s -> 118.6s
 gcc-cross-x86_64            do_compile               2.2s    +0.2%    913.4s -> 915.6s 
 binutils-cross-x86_64       do_compile               2.8s    +0.7%    388.3s -> 391.1s 
 glibc-locale                do_package_write_rpm     3.0s    +1.0%    294.4s -> 297.5s 
 glibc-locale                do_package              12.9s    +3.0%    427.0s -> 439.9s 
Cumulative cputime:
 -338.7s    -3.7%    2:32:59.4 (9179.4s) -> 2:27:20.7 (8840.7s)

As you can see, it was a 4% gain in cputime or an almost 8% gain in walltime. The configure caches will be nowhere near as effective as sstate cache, but they *should* be far less brittle in the face of minor changes. This approach needs more investigation, but I wanted to get some #'s down to quantify what we might see/expect.