Buildhistory

When upgrading a recipe to a new version of an upstream software package, or trying out new configuration options, subtle changes can occur in the output that may not be noticed until later. For example, a new version of a piece of software might introduce an optional dependency on another library which is auto-detected; if that library has already been built when the software is building, then the software will link to that library and it will now be pulled into your image with the software even if you didn't want it.

The buildhistory class was created to highlight unexpected and possibly unwanted changes such as these in the build system output. It will record information about the contents of each package (e.g. size, dependencies, list of files) and image (e.g. packages and files in the image), and commit it to a git repository so you can see when something changes.

Enabling buildhistory

Buildhistory is off by default. To enable it, add the following to the end of your conf/local.conf:

INHERIT += "buildhistory"
BUILDHISTORY_COMMIT = "1"

This will enable collection of the build output information as well as committing it to a local git repository.

Note that enabling buildhistory will increase your build times slightly (particularly for images) and will use up extra disk space.

Viewing the output

The default output directory is $TMPDIR/buildhistory.

An example (abbreviated) listing:

buildhistory/
|-- images
|   `-- atom_pc
|       `-- eglibc
|           `-- core-image-minimal
|               |-- build-id
|               |-- depends.dot
|               |-- depends-nokernel.dot
|               |-- depends-nokernel-nolibc.dot
|               |-- depends-nokernel-nolibc-noupdate.dot
|               |-- depends-nokernel-nolibc-noupdate-nomodules.dot
|               |-- files-in-image.txt
|               |-- image-info.txt
|               |-- installed-package-names.txt
|               |-- installed-package-sizes.txt
|               `-- installed-packages.txt
`-- packages
    `-- core2-poky-linux
        `-- busybox
            |-- busybox
            |   `-- latest
            |-- busybox-dbg
            |   `-- latest
            |-- busybox-dev
            |   `-- latest
            |-- busybox-doc
            |   `-- latest
            |-- busybox-httpd
            |   `-- latest
            |-- busybox-locale
            |   `-- latest
            |-- busybox-mdev
            |   `-- latest
            |-- busybox-staticdev
            |   `-- latest
            |-- busybox-syslog
            |   `-- latest
            |-- busybox-udhcpc
            |   `-- latest
            |-- busybox-udhcpd
            |   `-- latest
            `-- latest

Package information

For each package there is a text file containing name-value pairs with information about the package. For example, buildhistory/packages/core2-poky-linux/busybox/busybox/latest contains the following:

PV = 1.19.3
PR = r3
RDEPENDS = update-rc.d eglibc (>= 2.13)
RRECOMMENDS = busybox-syslog busybox-udhcpc
PKGSIZE = 564701
FILES = /usr/bin/* /usr/sbin/* /usr/libexec/* /usr/lib/lib*.so.* /etc /com /var /bin/* /sbin/* /lib/*.so.* /usr/share/busybox /usr/lib/busybox/* /usr/share/pixmaps /usr/share/applications /usr/share/idl /usr/share/omf /usr/share/sounds /usr/lib/bonobo/servers
FILELIST = /etc/busybox.links /etc/init.d/hwclock.sh /bin/busybox /bin/sh

Most of these name/values correspond to variables used to produce the package, except for FILELIST (the actual list of files in the package) and PKGSIZE (the total size of files in the package, in bytes).

There is also a file corresponding to the recipe from which the package came, for example buildhistory/packages/core2-poky-linux/busybox/latest:

PV = 1.19.3
PR = r3
DEPENDS = virtual/i586-poky-linux-gcc virtual/i586-poky-linux-compilerlibs virtual/libc update-rc.d-native
PACKAGES = busybox-httpd busybox-udhcpd busybox-udhcpc busybox-syslog busybox-mdev busybox-dbg busybox busybox-doc busybox-dev busybox-staticdev busybox-locale

Image information

The files produced for each image are as follows:

File	Purpose
build-id	Human-readable information about the build configuration and metadata source revisions
*.dot	graphviz-compatible dependency graphs for the image
files-in-image.txt	A list of files in the image with permissions, owner/group, size, and symlink information.
image-info.txt	A text file containing name-value pairs with information about the image (see below)
installed-package-names.txt	A list of installed packages (names only)
installed-package-sizes.txt	A list of installed packages ordered by size
installed-packages.txt	A list of installed packages (full package filename)

Note that installed package information is able to be gathered and produced even if packaging is disabled for the final image.

An example image-info.txt:

DISTRO = poky
DISTRO_VERSION = 1.1+snapshot-20120207
USER_CLASSES = image-mklibs image-prelink
IMAGE_CLASSES = image_types
IMAGE_FEATURES = debug-tweaks x11-base apps-x11-core package-management ssh-server-dropbear package-management
IMAGE_LINGUAS = en-us en-gb
IMAGE_INSTALL = task-core-boot task-base-extended
BAD_RECOMMENDATIONS = 
ROOTFS_POSTPROCESS_COMMAND = buildhistory_get_image_installed ;   rootfs_update_timestamp ;  
IMAGE_POSTPROCESS_COMMAND = buildhistory_get_imageinfo ; 
IMAGESIZE = 171816

Other than IMAGESIZE (the total size of the files in the image in KiB) the name/value pairs are variables that may have influenced the content of the image. These are often useful when you are trying to determine why a change in the package/file listings has occurred.

Analysing changes

Command line

To see any changes that have occurred (assuming you have BUILDHISTORY_COMMIT = "1") you can simply use git log -p or some other method of viewing the git history, however this will show changes that aren't significant e.g. a package changing in size by a few bytes. However, there is a command-line tool called buildhistory-diff which will query the git repository and print just the differences that might be significant in human-readable form. For example:

paul@helios:~/poky/poky/build/tmp/buildhistory$ ~/poky/poky/scripts/buildhistory-diff . HEAD^
Changes to images/qemux86_64/eglibc/core-image-minimal (files-in-image.txt):
  /etc/anotherpkg.conf was added
  /sbin/anotherpkg was added
  * (installed-package-names.txt):
  *   anotherpkg was added
Changes to images/qemux86_64/eglibc/core-image-minimal (installed-package-names.txt):
  anotherpkg was added
packages/qemux86_64-poky-linux/v86d: PACKAGES: added "v86d-extras"
  * PR changed from "r0" to "r1"
  * PV changed from "0.1.10" to "0.1.12"
packages/qemux86_64-poky-linux/v86d/v86d: PKGSIZE changed from 110579 to 144381 (+30%)
  * PR changed from "r0" to "r1"
  * PV changed from "0.1.10" to "0.1.12"

Buildhistory web UI

Web UI

A web-based frontend for processing and flagging these reported changes is also available:

http://git.yoctoproject.org/cgit/cgit.cgi/buildhistory-web/

For more details please see the README file in the repository.

Disabling buildhistory temporarily

Adding and removing the inherit of buildhistory from your configuration can change the do_package task checksums, which if you are using the OEBasicHash signature generator (the default for some distro configurations) will result in the packaging tasks being re-run each time you do this. To turn off buildhistory functionality without causing this to happen you can just add the following to your local.conf:

BUILDHISTORY_FEATURES = ""

Usage for image content analysis only

buildhistory outputs image information, including dependency graphs, so you can see easily why something was pulled into the image. If you are just interested in this information and not collecting history or any package information, you can do this by adding the following to your local.conf:

INHERIT += "buildhistory"
BUILDHISTORY_COMMIT = "0"
BUILDHISTORY_FEATURES = "image"

Example repositories

History data using buildhistory is already being collected from various autobuilders and published:

• Angstrom
• Poky
• SHR