Profiling

Tips and tricks for profiling the build system

Using bitbake -P

This provides a trace about where the bitbake server is spending its time. This used to be really great for profiling all aspects of bitbake. When the -P option is added to any usual commandline, bitbake will save out a profile.log file which contains the raw python profiling information and a "profile.log.processed" file which contains a human readable version of the trace information. The processed file contains four different views of the data, one after the other. One type of chart shows the time spent in functions, the second type of chart shows the same information but includes callgraph information so we you see where the functions were called from. These two charts are printed twice, firstly sorted by internal function time and secondly be cumulative function time.

For the actual tasks themselves there is a patch, "build.py: Dump out performance data of individual tasks" available in the poky-contrib tree which enables each task to dump out performance information. Its still pending to integrate this patch into the main tree in some form.

The other big area where profiling is useful is during the parsing itself. Unfortunately this is a harder area to profile due to the parallelisation of parsing. We don't have a good way to profile this section of code at the moment.

Using pybootchart

This tool allows visualisation of the build process and task execution. Firstly, you need to have enabled the buildstats class in your local build config (buildstats in USER_CLASSES in local.conf). You can then runa command like:

<poky checkout>/scripts/pybootchartgui/pybootchartgui.py build/tmp/buildstats/bash-qemux86-64/201211190818/

where the second argument is a path to a given buildstats file of the build you want to visualise. By default, tasks taking less than 8 seconds are not graphed to remove short "noise" from the graph. The -m commandline option can be used to change the minimum task length to filter with, "0" being no filtering.

Common tasks are colour coded to aid visualisation. The graph can also be panned and scaled. The tool is good at spotting bottlenecks in builds and allowing the user to appreciate where time is being spent during the build.

If a directory is specified as a third option, graphic files are saved to the directory in svg format. For large graphs, these can be split into a number of files using the "-n" option.

Disk Usage

A simple "du" call can tell how much space a given build directory is taking up. Equally, the size of the sstate data or source downloads can also be measured. One thing to be aware of is we hardlink files to save disk space and increase performance.

Timing bitbake commands

Using commands in the form "time bitbake <options>" gives a good indication of how long a command takes and provides a simplistic way of measuring some aspects of performance.

Python memory usage

This is hard to measure but we have made worthwhile improvements in this area in the past. Bitbake takes significance advantage of "copy-on-write" behaviour both in the way it executes tasks and also the way the data store works internally.