Bitbake Internals Primer

General

BitBake is a build tool - a little like make, except that execution is mostly considered at the task level rather than the target level - each target can have multiple tasks associated with it, these tasks can have dependencies on eachother. This makes it well suited to building software made up of multiple independent software packages, each of which require build steps such as downloading, unpacking, patching, compiling, packaging etc.

Metadata

BitBake doesn't do very much on its own - it requires a set of metadata (configuration files and recipes) to do anything useful. OpenEmbedded (OE), in particular OpenEmbedded Core, is the main example, but there is also a "hello world" example in the BitBake manual which may be useful to see how simple this can be.

One thing to keep in mind in particular when adding functionality to BitBake is that we try to avoid polluting its codebase with implementation details - where possible, operation should be defined by the metadata, and only the generic parts that are needed will be added to BitBake itself. Consequently you will find that a lot of functionality that you might consider "core" to BitBake operation is actually defined by the metadata and not within BitBake itself. Sometimes this unfortunately does complicate implementing certain functionality. It's worth mentioning that BitBake is used by one or two non-OpenEmbedded projects, one example being isar.

A signature is produced based on all of the inputs to a task (the variable values, and the contents of the task function itself). A stamp file for the task written out when the task completes successfully includes the signature, such that the presence of the stamp on subsequent bitbake runs can be used to determine if the task needs to be re-run or not. BitBake also provides for the existence of "setscene" tasks, which can take the place of a real task if it is defined and succeeds - this is is the generic "interface" that is used to implement the shared state cache in OE.

Major components

BitBake itself is broken up into several processes:

• The server ("cooker") which orchestrates most of the operation
• The client or UI (default TTY UI is "knotty" but other UI implementations also exist)
• The worker which actually runs tasks

The server and client use an RPC mechanism to communicate. The UI sends commands, some of which are synchronous (return a result within the call) and some of which are asynchronous (return a result in the form of an event). The event system is used both for this and also from the metadata for performing certain operations in response to events, although there are certain events that don't make it to the "class handlers" (i.e. those defined in the metadata).

A pool of worker processes gets started by the server, and task information is passed from the server through a pipe in order to set up the task environment and execute the task. The worker re-reads the configuration rather than having it completely passed over the pipe.

Additionally there is also an optional PR server that manages incrementing PR values in response to changes in the metadata. This may be remote or it can be started and managed locally by the server.

The Code

Listed below are certain modules and directories that are useful to know - there are more modules than this of course but these are some of the key ones.

bin/

This is where actual executable scripts provided by bitbake are located, such as bitbake itself, and utilities such as bitbake-layers. The bitbake script itself doesn't do very much except perform basic checks and then hand off to lib/bb/main.py.

lib/bb/main.py

main.py takes care of parsing command-line options, command-line help and starting the server / UI.

lib/bb/runqueue.py

The runqueue is concerned with executing tasks. There is a separate runqueue implemented here for setscene tasks and "real" tasks, since these are run separately (we need to know which setscene tasks succeed before we know which real tasks will need to run). There are several different scheduler implementations which can be selected between in configuration.

lib/bb/command.py

This is the implementation for the RPC interface provided by the server to the UI.

NOTE: be careful about what you pass over RPC - the xmlrpc server implementation that is used when bitbake is memory resident has a very limited number of types that it supports and thus data of unsupported types will trigger an error.

lib/bb/cooker.py

The cooker is the main part of the server which holds the configuration and executes some of the commands.

NOTE: There are some functions that were implemented in cooker for the purposes of supporting the now removed Hob UI (e.g. the code to edit configuration files), these were poorly placed and should be removed in future.

lib/bb/fetch2

A set of fetcher modules, providing the ability to fetch source code / binaries from remote or local URLs.

lib/bb/ui

UI implementations - knotty.py being the default TTY-based one.

lib/bb/server

Server implementations - there is more than one, though in practice the "process" one is used during the default mode of operation and the "xmlrpc" one is used if you run bitbake in memory-resident mode. This abstraction may be removed in future.

lib/bb/data_smart.py

The datastore - this is where variable values are stored (as represented by "d" within python functions). lib/bb/data.py also contains some code that manipulates this object.

lib/bb/build.py

Code for actually executing functions and tasks, and dealing with the results in the form of events.

lib/bb/parse.py

Parsing code for recipes and configuration files.

lib/bb/codeparser.py

Parsing code for python/shell functions and in-line expressions. Used mainly to extract dependencies on other functions and variables, which are used when calculating signatures and when preparing the runfiles for a task.

lib/bb/event.py

Event handling code.

lib/bb/siggen.py

The base code for task signature generation as well as the code used for dumping/comparing signatures (as used by bitbake-dumpsig, bitbake-diffsigs and bitbake -S printdiff). NOTE: OpenEmbedded has additional signature generation code which complements this in meta/lib/oe/sstatesig.py.

lib/bb/utils.py

A set of utility functions for various things. Some of these are frequently called from within the metadata e.g. bb.utils.contains(), others are used only within BitBake code.

lib/bb/tinfoil.py

A wrapper API that makes it easier to write small utilities that call into BitBake code. bitbake-layers uses this, as do devtool and recipetool in OpenEmbedded.

Tests

BitBake comes with a small test suite in lib/bb/tests/ and a runner script called bitbake-selftest. It doesn't yet cover all of the code, but does exercise some key parts including the fetcher code, datastore and parsing. If you're making changes to these parts of the code you should definitely run the tests. We should also be extending the test suite to cover new code where appropriate. Note that bitbake-selftest can take arguments to tell it which specific tests to run, so you don't need to run them all every time.