Technical FAQ

NOTE: This is currently a draft. Not sure where this should end up but I've been gathering these based on my interactions with people on IRC and email over the years. - PaulEggleton (talk) 21:13, 27 June 2016 (PDT)

Basics

How do I control what's in the final image?

Images specify a list of packages that they should contain. See: http://www.yoctoproject.org/docs/current/dev-manual/dev-manual.html#usingpoky-extend-customimage

Note: if you're doing anything more than basic experimentation / testing then you almost certainly should create your own image recipe rather than using one of the example images e.g. core-image-minimal - though you can certainly start by copying one of the example images. This way you have easier control over what goes into the image.

Where do I find build logs?

For the overall build, the output of bitbake gets logged to tmp/log/cooker/<machine>.

For each individual recipe, there is a "temp" directory under the work directory for the recipe that contains log.<taskname> and run.<taskname> files - the logs and the runfiles respectively. Within the build system this path is represented by the T variable, so if you need to you can find it by using bitbake -e:

bitbake -e recipename | grep ^T=

How do I add a patch to a recipe?

There are two concerns - how the recipe can fetch the patch and how it can be applied. For fetching, patch files are usually placed in a subdirectory next to the recipe; by default this directory should be named "files" or the the recipe name without any class prefix or suffix (for example for both "xyz" and "xyz-native" the subdirectory would be "xyz"). A pointer to it then needs to be added to SRC_URI within the recipe, which usually takes the form file://patchname.patch - i.e. just the filename, no path.

The devtool utility can help you modify the sources for a recipe and create a patch - basically devtool modify, edit the sources, commit the changes with git commit, then devtool update-recipe.

How do I enable package management in the final image?

Add "package-management" to IMAGE_FEATURES (or EXTRA_IMAGE_FEATURES). You should then be able to use smart/rpm, opkg, or apt-get/dpkg from the running system depending on the packaging format you have selected through PACKAGE_CLASSES. For more information see: http://www.yoctoproject.org/docs/current/dev-manual/dev-manual.html#using-runtime-package-management

Layers

See http://www.openembedded.org/Layers_FAQ

Dependencies

I disabled runtime package management and yet it still seems to be building rpm/opkg, why?

"package-management" being in IMAGE_FEATURES (possibly indirectly via EXTRA_IMAGE_FEATURES) controls whether the package manager is used at runtime i.e. whether it is present in the target image. However, the build system always uses a package manager on the host to assemble images since it is usually the best tool for this job. This is completely independent of whether the package manager is in the target image though.

Why is opkg-native / opkg-utils being built when I don't have ipk packaging enabled?

opkg-utils provides update-alternatives which is the default tool used to manage the alternatives system (for selecting between multiple providers of the same file, e.g. busybox and bash both provide /bin/sh).

Why is rpm-native being built when I don't have rpm packaging enabled?

rpm-native is needed for two things in the generic packaging code implemented in the package class:

1. Debug symbol splitting - rpm-native provides the debugedit tool which this code uses
2. Per-file dependencies - although this was originally just feeding into rpm when rpm was being used, it also now gets verified by QA checks regardless of which packaging backend is in use.

I see a recipe built, but building an image containing the corresponding package fails at do_rootfs because it can't find the package. How does this happen?

Usually this is because the recipe claimed to provide the specified package (via PACKAGES or PACKAGES_DYNAMIC) but it wasn't actually produced, possibly because it ended up empty (since by default empty packages aren't produced). If this is a recipe you are writing yourself the probable cause is your recipe isn't installing any files and thus the main package for the recipe is empty. Fix do_install (or what do_install is already running, e.g. make install) such that files are installed into the correct location such that they can then subsequently be packaged, and then all should be well.

X11 and various other items are being built but I'm only building core-image-minimal - why?

This is where it helps to understand the difference between build-time dependencies and runtime dependencies - often, a recipe will require things at build time (for example tools that help the build process, or to satisfy optional dependencies) that it doesn't necessarily need at runtime. The default configuration includes "x11" in DISTRO_FEATURES, and thus anything that can optionally support X11 will have its X11 support enabled; however when it comes to actually producing the image there won't be any X11 packages included. If you never intend to use X11 you can set your own DISTRO_FEATURES value that excludes x11 and then these items won't even be built.

How do I avoid the kernel itself being pulled into my image when installing kernel modules?

By default, the kernel class sets a dependency on the kernel-base package (which kernel modules always depend on) onto kernel-image, which contains the actual kernel binary. If you don't want this, set the following either in your kernel recipe or at the configuration level:

RDEPENDS_kernel-base = ""

How do I find out why something is being built?

bitbake -g <recipe> will produce some .dot files that allow you to see the dependency relationships - usually pn-depends.dot holds the answers although sometimes you may need to look at task-depends.dot if the dependency is only in the form of a task dependency. Note that these graphs are much too large for most graphviz visualisation tools to process, so you'll probably find it's easiest to view them with "less" or a text editor and search for the item you're looking for.

How do I find out why something is in my image?

Enable the buildhistory class and build the image again, and it will write out a depends.dot file containing the relationships between packages in the final image. If the package name isn't mentioned it is probably explicitly mentioned in IMAGE_INSTALL or being brought in via IMAGE_FEATURES. See http://www.yoctoproject.org/docs/current/ref-manual/ref-manual.html#maintaining-build-output-quality

Troubleshooting

I set a variable but it doesn't seem to be having an effect, how do I fix this?

First, double-check that you haven't misspelled the variable name.

The main tool to help troubleshoot any variable-related issue is bitbake -e - this lists all the variables and the complete history of how each one has been set. Usually it's best to pipe this through less so you can easily see the history - you can press / to search for the variable name. Often you will be dealing with the behaviour of a variable within the context of a specific recipe, so specify that recipe on the bitbake -e command line to get the variables as set within the context of the recipe rather than the global context.

If you're setting a variable in a bbappend, double-check that the bbappend is actually being applied - see the next question.

I've created a bbappend for a recipe but what I'm setting there isn't having any effect, how do I fix this?

Here are some things to check:

1. Check if the layer the bbappend is in is listed in bitbake-layers show-layers. If it isn't, you need to edit your bblayers.conf and ensure the path to the layer is included in the BBLAYERS value
2. Check that the bbappend is being picked up by running bitbake-layers show-appends - if your bbappend file isn't listed, it could be named incorrectly (such that it doesn't match the recipe name) or it may be that the BBFILES value in the conf/layer.conf for the layer containing the bbappend file doesn't include an expression that will match the bbappend files.
3. If there are multiple versions of the recipe you have bbappended, it could be that the actual recipe being built is a different version than the one you have bbappended. bitbake-layers show-recipes recipename will list all the versions, with the first one listed being the one that will be built. If this is the case there are several different solutions to this - (a) Rename your bbappend to match the version being built, (b) use a % wildcard in your bbappend so it will apply to any version, (c) set PREFERRED_VERSION_<recipename> in the configuration to select a the version you want to be built.

I'm getting warnings that a recipe is tainted - what does this mean?

Usually this happens because you have used I used bitbake's -f or -C option to force a task to re-execute. The assumption is that if you forced a task, it is possible that a rebuild from scratch would not include whatever changes you made that necessitated forcing (e.g. if you modified the source in the work directory for the recipe and then ran bitbake -c compile -f). Generally, forcing a task should be reserved for situations where the build system has failed to detect a change you made rather than for everyday usage - if you're finding yourself needing to do it regularly then either there's a bug or you're doing something wrong. Running bitbake -c clean on the recipe will get rid of the taint flag.

There is one other situation where we apply a taint, and that is bitbake -c menuconfig on the kernel. In this case, the configuration has been saved into the work directory for the kernel, but that is temporary - any rebuild from scratch will use the default configuration, so it is a reminder that you need to take the configuration and apply it back to the metadata.

I'm fetching from a git repository over ssh / http / https but it's not fetching properly, how do I fix this?

Bitbake expects the prefix of entries in SRC_URI to specify the fetcher to be used, not the actual protocol. Thus, instead of:

# This will not work
SRC_URI = "http://git.example.com/repository"

You should specify:

# This is better
SRC_URI = "git://git.example.com/repository;protocol=http"

The same applies for ssh and https.

I tried bitbake <some target package name> and it told me that nothing PROVIDES this...?

There are two namespaces that bitbake concerns itself with - recipe names (a.k.a. build time targets) and package names (a.k.a. runtime targets). You can specify a build time target on the bitbake command line, but not a runtime target; you need to find the recipe that provides the package you are trying to build and build that instead (or simply add that package to your image and build the image).

I'm required to set LIC_FILES_CHKSUM but the software I'm building doesn't have a license statement, what do I do?

Ideally, all software should come with some kind of license statement so that the terms of distribution are clearly stated (especially if its source code is made publicly available); if not a text file describing the license then at the very least a line or two in the accompanying documentation, README file or source header comments. If there really is absolutely no statement you can point to as part of the upstream source, you can point LIC_FILES_CHKSUM to one of the generic license files in ${COMMON_LICENSE_DIR} (meta/files/common-licenses/). However, it is worth noting that LIC_FILES_CHKSUM is intended to give you a warning if upstream changes its license terms when you do an upgrade of the recipe, and by pointing it to this common license file that is part of the metadata, that mechanism will not function. You may wish to consider encouraging the upstream provider of the software your recipe is building to follow best practices and include a proper license statement, so that you can point to it in a future version.

Misc

How do I remove a value from a list variable?

BitBake supports a _remove operator. See: http://www.yoctoproject.org/docs/current/bitbake-user-manual/bitbake-user-manual.html#removing-override-style-syntax

Note that _remove is final - you cannot undo it, thus you should really only make use of it in your distro / local configuration and not in layers that you expect others to re-use for different purposes (and therefore they may need to undo your changes). An alternative way to effectively remove an item is to set the list outright to include all the items minus the one you want to remove.

How do I change how my recipe is built depending on what image I'm building?

The short answer is you cannot - the reason is that OpenEmbedded builds packages based on the overall configuration, and then the image only selects which of these packages should go into the final image. However, there are some solutions when you do want to do this:

1. Have separate packages for the two different versions. This could take the form of different recipes or you could do it within the same recipe. The two packages do have to have different names however; this may create problems if you have other packages that depend on the package.
2. Use a postprocessing function within the image(s) - within the image recipe, define a shell or python function that makes the desired changes to the files in the image and add a call to it to ROOTFS_POSTPROCESS_COMMAND within the image recipe. Note that this may not be appropriate if you have runtime package management enabled since the postprocessing will only happen at image creation time and not if the package is installed later on at runtime - you may need to use a postinstall script instead in this case.
3. Use a postinstall script (pkg_postinst_<package> function) within the recipe. In order to work, the postinstall script will need to be able to determine what to do when it's run - this may not be practical depending on what you're trying to achieve.

Can I use a toolchain built by OE as the external toolchain?

In general, this is not recommended and not something that is tested or directly supported out of the box. If you are wanting to do this solely as a means of speeding up the build, it is strongly suggested that you use shared state instead.

There is a meta-sourcery layer available to enable support for the CodeSourcery toolchain, you may be able to use this as a template for bringing in an external toolchain however there are no guarantees.

When I run bitbake -c devshell it looks like it's running as root! How is that possible?

It's not running as the actual root user, it's just pretending for the benefit of programs that run under it (including bash) that it is, via pseudo. This is important, because you normally want any owner/group/permission values that you set on files to be reflected in files that the recipe installs and packages and thus reflected in the final image - without this mechanism the actual build would have to run as root which would be very risky. There are no actual elevated privileges through this mechanism however, so you need not be worried.

How do I find out what packages are produced by a recipe?

The Toaster web UI provides easy ways to query this.

In the 1.8 (fido) release and newer you can use the following command, assuming the recipe has already been built:

oe-pkgdata-util list-pkgs -p recipename

Alternatively you can look in the "packages-split" subdirectory under the work directory for the recipe - each package produced by the recipe will have a subdirectory under that. If you're not sure how to find the work directory you can run the following command:

bitbake -e recipename | grep ^WORKDIR=

Before a recipe gets built it is a bit trickier, since the system often doesn't know exactly which packages will be produced until do_package time; this is particularly true for recipes that package plugins or modules (e.g. kernel modules). You can get a reasonable idea though by looking at the value of PACKAGES (and PACKAGES_DYNAMIC for recipes that produce plugins).

I have a source tree I want to build instead of the upstream source a recipe normally fetches, how do I do that?

If it's for development purposes i.e. you have your own local source tree you want to work on and have built, then use:

devtool modify -n <recipename> path/to/sourcetree/

Once you are done you can use devtool reset to return to building the source specified in the recipe.

Alternatively if it's more permanent, use the externalsrc class - you can inherit this in the original recipe or a bbappend

inherit externalsrc
EXTERNALSRC = "/path/to/sources"

Alternatively you can inherit it globally at the configuration level:

INHERIT += "externalsrc"
EXTERNALSRC_pn-<recipename> = "/path/to/sources"

How do I specify the default shell? (e.g. bash instead of busybox)

It depends what you mean. As far as which provides /bin/sh, this is controlled through the alternatives system, and by default bash has a higher priority than busybox, so simply installing bash into your image will automatically have /bin/sh link to bash rather than busybox.

If you mean you want a user's login shell to be a specific shell, you'll need to modify /etc/passwd. One fairly easy way to achieve this is to use the extrausers class in your image recipe:

inherit extrausers
EXTRA_USERS_PARAMS = "usermod -s /bin/bash <username>; "

How do I allow a variable's value through from the external environment?

Add the variable's name to the BB_ENV_EXTRAWHITE in the external environment before running bitbake. Note that the oe-init-build-env script sets a default for this which you will want to preserve, so add to the default value rather than overwriting it.

Alternatively if you just want to get the external value of a variable from python code within the metadata, you can use the BB_ORIGENV variable which itself contains a datastore of the original environment. For example to get the value of the DISPLAY variable from the environment within a python function you would do this:

display = d.getVar("BB_ORIGENV", False).getVar("DISPLAY", True)

Note that you must specify "false" for the expand parameter when getting the BB_ORIGENV variable, because it's not a string and therefore cannot be expanded in the normal manner.

Why is bitbake showing "AUTOINC" in the version for some recipes?

Recipes where you see AUTOINC within the version will be those that set PV to include "${SRCPV}" to get the SCM revision in the package version. In order to have the version increment properly there needs to be an number in front of the revision which automatically increments each time the revision changes (assuming you have a PR server enabled), which is where AUTOINC comes in. During the build, AUTOINC is a stand-in for this auto-incrementing number, and during do_package it gets replaced with the real number.

Why are .so files in the -dev package instead of the main package for a recipe?

In standard Unix library packaging, non-versioned .so symlinks (e.g. libgd.so) are intended for development purposes only. At runtime, binaries should be linked to the versioned .so file/symlink e.g. libgd.so.3. This (theoretically) allows multiple major versions of the same library as well as binaries that depend upon each of them to coexist on the same system. If the library is versioned but you have a binary that links to the unversioned .so file, it has almost certainly been linked incorrectly.

Non-symlink .so files on the other hand are sometimes produced and are entirely legal - however these will be picked up in the -dev package simply by virtue of their name, which is almost always not what you want. In this case you can do one of two things:

1. Fix the build of the library so it gets versioned. This may not alwasy be appropriate, especially not for things like plugins.
2. Set FILES_${PN}-dev within the recipe so that it does not include ${FILES_SOLIBSDEV}. If the software the recipe is building also produces symlink .so files you'll need to set FILES_${PN}-dev such that those do still get packaged in the -dev package though, or you'll get a package QA warning.