Technical FAQ - Revision history

Simonew: Fix syntax

2023-12-23T20:31:25Z

Fix syntax

← Older revision		Revision as of 20:31, 23 December 2023
Line 200:		Line 200:
	=== How do I remove a value from a list variable? ===		=== How do I remove a value from a list variable? ===

	For variables that are expected to contain a space-separated list of items, BitBake supports a ~~_remove~~ operator to remove items from it. See [http://www.yoctoproject.org/docs/current/bitbake-user-manual/bitbake-user-manual.html#removing-override-style-syntax Removal (override style syntax)] in the BitBake user manual.		For variables that are expected to contain a space-separated list of items, BitBake supports a :remove operator to remove items from it. See [http://www.yoctoproject.org/docs/current/bitbake-user-manual/bitbake-user-manual.html#removing-override-style-syntax Removal (override style syntax)] in the BitBake user manual.

	'''NOTE:''' the ~~_remove~~ operation is final - you cannot "undo" it with other operations elsewhere, thus you should really only make use of it in your distro / local configuration and not in layers that you expect others to re-use for different purposes (and therefore they may need to undo your changes). An alternative way to effectively remove an item is to set the list outright to include all the items minus the one you want to remove.		'''NOTE:''' the :remove operation is final - you cannot "undo" it with other operations elsewhere, thus you should really only make use of it in your distro / local configuration and not in layers that you expect others to re-use for different purposes (and therefore they may need to undo your changes). An alternative way to effectively remove an item is to set the list outright to include all the items minus the one you want to remove.

	=== How do I change how my recipe is built depending on what image I'm building? ===		=== How do I change how my recipe is built depending on what image I'm building? ===

Simonew: Fix syntax

2023-12-23T20:30:44Z

Fix syntax

← Older revision		Revision as of 20:30, 23 December 2023
Line 344:		Line 344:
	Whilst it is possible to do this within a single recipe building for the target, it is tricky to do so because in that context everything is set up for cross-compiling for the target, and you would have to undo all of that to build host tools. The standard and much easier way of handling this is to create a native variant of the recipe using BBCLASSEXTEND and have your host tools built within that, and then have the target variant depend on the native variant. For example, assume your recipe were called xyz (xyz_1.1.bb), then you would include something like this in the recipe:		Whilst it is possible to do this within a single recipe building for the target, it is tricky to do so because in that context everything is set up for cross-compiling for the target, and you would have to undo all of that to build host tools. The standard and much easier way of handling this is to create a native variant of the recipe using BBCLASSEXTEND and have your host tools built within that, and then have the target variant depend on the native variant. For example, assume your recipe were called xyz (xyz_1.1.bb), then you would include something like this in the recipe:

	~~DEPENDS_append_class~~-target = " xyz-native"		DEPENDS:append_class-target = " xyz-native"

	...		...

Simonew: Fix syntax

2023-12-23T20:30:15Z

Fix syntax

← Older revision		Revision as of 20:30, 23 December 2023
Line 455:		Line 455:
	Ultimately, overriding bbclass files is not good practice long term - you are opening yourself up to maintenance issues when the original class changes, and the override is fragile as hinted above. The best solution is to try to get whatever changes you need into the original class; this does of course require additional work and time though.		Ultimately, overriding bbclass files is not good practice long term - you are opening yourself up to maintenance issues when the original class changes, and the override is fragile as hinted above. The best solution is to try to get whatever changes you need into the original class; this does of course require additional work and time though.

	=== There's a bbappend in a layer I'm using that defines a <code>~~do_something_append~~()</code> and I want to append to that function also, how do I do this? ===		=== There's a bbappend in a layer I'm using that defines a <code>do_something:append()</code> and I want to append to that function also, how do I do this? ===

	Simply create a bbappend in your layer and define your own <code>~~do_something_append~~()</code>, and your commands will be executed ''as well as'' those of the other bbappend.		Simply create a bbappend in your layer and define your own <code>do_something:append()</code>, and your commands will be executed ''as well as'' those of the other bbappend.

	You might assume that defining <code>~~do_something_append~~()</code> will overwrite any previously defined <code>~~do_something_append~~()</code>, as would be the case with <code>do_something()</code> in the same situation, but that is not the case - the key is that <code>~~_append~~</code> (and <code>~~_prepend~~</code>, <code>~~_remove~~</code>, etc.) are ''operators'' and they will be applied in sequence, where that sequence is the order in which they are parsed (which for bbappends will be in ascending layer priority order).		You might assume that defining <code>do_something:append()</code> will overwrite any previously defined <code>do_something:append()</code>, as would be the case with <code>do_something()</code> in the same situation, but that is not the case - the key is that <code>:append</code> (and <code>:prepend</code>, <code>:remove</code>, etc.) are ''operators'' and they will be applied in sequence, where that sequence is the order in which they are parsed (which for bbappends will be in ascending layer priority order).

PaulEggleton: Add why not root question

2020-07-09T21:01:45Z

Add why not root question

← Older revision		Revision as of 21:01, 9 July 2020
Line 217:		Line 217:

	There is a [http://layers.openembedded.org/layerindex/branch/master/layer/meta-sourcery/ meta-sourcery layer] available to enable support for the CodeSourcery toolchain, you may be able to use this as a template for bringing in an external toolchain however there are no guarantees.		There is a [http://layers.openembedded.org/layerindex/branch/master/layer/meta-sourcery/ meta-sourcery layer] available to enable support for the CodeSourcery toolchain, you may be able to use this as a template for bringing in an external toolchain however there are no guarantees.

			=== Why can't I run bitbake as root? ===

			If you try to run bitbake as root you may have noticed you get a sanity check failure that prevents the build from continuing. The reason we have this check is that it is very risky to run builds as root; if any build script mistakenly tries to install files to the root path (/) instead of where it is supposed to, you want it to fail immediately and not (in the worst case) overwrite files critical to your Linux system's operation e.g. under /bin or /etc. Thus, we do not support running the build as root. The only time root access is needed is (completely outside of a build) where the runqemu script uses sudo to set up TAP devices for networking.

	=== When I run bitbake -c devshell it looks like it's running as root! How is that possible? ===		=== When I run bitbake -c devshell it looks like it's running as root! How is that possible? ===

	It's not running as the actual root user, it's just pretending for the benefit of programs that run under it (including your shell) that it is, via pseudo. This is important, because you normally want any owner/group/permission values that you set on files to be reflected in files that the recipe installs and packages and thus reflected in the final image - without this mechanism the actual build would have to run as root which would be very risky. There are no actual elevated privileges through this mechanism however, so you need not be worried.		It's not running as the actual root user, it's just pretending for the benefit of programs that run under it (including your shell) that it is, via pseudo. This is important, because you normally want any owner/group/permission values that you set on files to be reflected in files that the recipe installs and packages and thus reflected in the final image - without this mechanism the actual build would have to run as root which would be very risky (see above). There are no actual elevated privileges through this mechanism however, so you need not be worried.

	=== Why does OE use pseudo? Why not use fakeroot / fakechroot instead? ===		=== Why does OE use pseudo? Why not use fakeroot / fakechroot instead? ===

PaulEggleton: Consistency: use instead of recipename

2020-07-01T01:30:56Z

Consistency: use <recipename> instead of recipename

PaulEggleton: /* I'm getting warnings that a recipe is tainted - what does this mean? */

2020-07-01T01:25:54Z

I'm getting warnings that a recipe is tainted - what does this mean?

← Older revision		Revision as of 01:25, 1 July 2020
Line 85:		Line 85:
	Usually this happens because you have used I used bitbake's <code>-f</code> or <code>-C</code> option to force a task to re-execute. The assumption is that if you forced a task, it is possible that a rebuild from scratch would not include whatever changes you made that necessitated forcing (e.g. if you modified the source in the work directory for the recipe and then ran <code>bitbake -c compile -f</code>). Generally, forcing a task should be reserved for situations where the build system has failed to detect a change you made rather than for everyday usage - if you're finding yourself needing to do it regularly then either there's a bug, you're doing something wrong, or perhaps you're using <code>-f</code> or <code>-C</code> when it's not really needed. Running <code>bitbake -c clean</code> on the recipe will get rid of the taint flag.		Usually this happens because you have used I used bitbake's <code>-f</code> or <code>-C</code> option to force a task to re-execute. The assumption is that if you forced a task, it is possible that a rebuild from scratch would not include whatever changes you made that necessitated forcing (e.g. if you modified the source in the work directory for the recipe and then ran <code>bitbake -c compile -f</code>). Generally, forcing a task should be reserved for situations where the build system has failed to detect a change you made rather than for everyday usage - if you're finding yourself needing to do it regularly then either there's a bug, you're doing something wrong, or perhaps you're using <code>-f</code> or <code>-C</code> when it's not really needed. Running <code>bitbake -c clean</code> on the recipe will get rid of the taint flag.

	There is one other situation where we apply a taint, and that is <code>bitbake -c menuconfig</code> on the kernel. In this case, the configuration has been saved into the work directory for the kernel, but that is temporary - any rebuild from scratch will use the default configuration, so it is a reminder that you need to take the configuration and apply it back to the metadata and then run <code>bitbake -c clean</code> on the kernel recipe.		There is one other situation where we apply a taint, and that is <code>bitbake -c menuconfig</code> on the kernel (or some other kconfig-using recipe). In this case, the configuration has been saved into the work directory for the kernel, but that is temporary - any rebuild from scratch will use the default configuration, so it is a reminder that you need to take the configuration and apply it back to the metadata and then run <code>bitbake -c clean</code> on the kernel recipe.

	=== I'm fetching from a git repository over ssh / http / https but it's not fetching properly, how do I fix this? ===		=== I'm fetching from a git repository over ssh / http / https but it's not fetching properly, how do I fix this? ===

PaulEggleton: /* I'm fetching from a git repository over ssh / http / https but it's not fetching properly, how do I fix this? */

2020-07-01T01:24:56Z

I'm fetching from a git repository over ssh / http / https but it's not fetching properly, how do I fix this?

← Older revision		Revision as of 01:24, 1 July 2020
Line 92:		Line 92:

	# This will NOT work		# This will NOT work
	SRC_URI = "<nowiki>~~http~~://git.example.com/repository</nowiki>"		SRC_URI = "<nowiki>https://git.example.com/repository</nowiki>"

	You should specify:		You should specify:

	# This is better		# This is better
	SRC_URI = "<nowiki>git://git.example.com/repository;protocol=~~http~~</nowiki>"		SRC_URI = "<nowiki>git://git.example.com/repository;protocol=https</nowiki>"

	The same applies for ssh and ~~https~~.		The same applies for ssh and of course http.

	=== I tried bitbake <some target package name> that I know exists and it told me that nothing PROVIDES this...? ===		=== I tried bitbake <some target package name> that I know exists and it told me that nothing PROVIDES this...? ===

PaulEggleton: /* X11 and various other items are being built but I'm only building core-image-minimal - why? */

2020-07-01T01:22:39Z

X11 and various other items are being built but I'm only building core-image-minimal - why?

← Older revision		Revision as of 01:22, 1 July 2020
Line 180:		Line 180:
	In other situations the reference to the package in question is spurious and either it should be removed entirely or there's another package that should be used instead. For example, the avahi and dhcp recipes both have an empty main package since the client and server are split out into their own packages, and those are the ones you should be using instead (avahi-daemon, avahi-utils, dhcp-server, dhcp-client - there are other packages as well, please see [[#How_do_I_find_out_what_packages_are_produced_by_a_recipe.3F\|How do I find out what packages are produced by a recipe?]].) You could argue that these recipes shouldn't claim to provide the main package, or they should have a main package that depends on all the other packages (as some other recipes do).		In other situations the reference to the package in question is spurious and either it should be removed entirely or there's another package that should be used instead. For example, the avahi and dhcp recipes both have an empty main package since the client and server are split out into their own packages, and those are the ones you should be using instead (avahi-daemon, avahi-utils, dhcp-server, dhcp-client - there are other packages as well, please see [[#How_do_I_find_out_what_packages_are_produced_by_a_recipe.3F\|How do I find out what packages are produced by a recipe?]].) You could argue that these recipes shouldn't claim to provide the main package, or they should have a main package that depends on all the other packages (as some other recipes do).

	=== X11 and various other items are being built but I'm only building core-image-minimal - why? ===		=== X11 and various other items are being built but I'm only building core-image-minimal that doesn't have X11 in it - why? ===

	This is where it helps to understand the difference between build-time dependencies and runtime dependencies - often, a recipe will require things at build time (for example tools that help the build process, or to satisfy optional dependencies) that it doesn't necessarily need at runtime. The default configuration includes "<code>x11</code>" in <code>DISTRO_FEATURES</code>, and thus anything that can optionally support X11 will have its X11 support enabled; however when it comes to actually producing the image there won't be any X11 packages included as long as there are no hard dependencies and there aren't any X11 packages explicitly requested.		This is where it helps to understand the difference between build-time dependencies and runtime dependencies - often, a recipe will require things at build time (for example tools that help the build process, or to satisfy optional dependencies) that it doesn't necessarily need at runtime. The default configuration includes "<code>x11</code>" in <code>DISTRO_FEATURES</code>, and thus anything that can optionally support X11 will have its X11 support enabled; however when it comes to actually producing the image there won't be any X11 packages included as long as there are no hard dependencies and there aren't any X11 packages explicitly requested.

PaulEggleton: /* Can I disable shared state? */

2020-07-01T01:16:40Z

Can I disable shared state?

← Older revision		Revision as of 01:16, 1 July 2020
Line 325:		Line 325:
	You cannot, no. Shared state (sstate) is an intrinsic part of staging files into the sysroot. It is possible to construct a recipe that bypasses sstate for some tasks (the kernel does this), however this is quite difficult and if not done properly will lead to many other problems.		You cannot, no. Shared state (sstate) is an intrinsic part of staging files into the sysroot. It is possible to construct a recipe that bypasses sstate for some tasks (the kernel does this), however this is quite difficult and if not done properly will lead to many other problems.

	Almost always when you are having a problem with shared state the issue is either (a) you're adding/changing files in the sysroot directly (i.e. outside sstate control), or (b) what is being placed into the sysroot isn't relocatable. The solution for (a) is do not do that - files should always be installed under <code>${D}</code> within <code>do_install</code> and then a subset of those are staged into the sysroot automatically. For (b) you need to fix or adapt the hardcoded path(s) - if the program reads (or can be made to read) each path from an environment variable, then you can use the <code>create_wrapper</code> utility function to create a wrapper script that will set the path appropriately. Run <code>git grep create_wrapper</code> in the <code>meta</code> subdirectory to see examples.		Almost always when you are having a problem with shared state the issue is either (a) you're adding/changing files in the sysroot directly (i.e. outside sstate control), or (b) what is being placed into the sysroot isn't relocatable due to hardcoded paths. The solution for (a) is do not do that - files should always be installed under <code>${D}</code> within <code>do_install</code> and then a subset of those are staged into the sysroot automatically. For (b) you need to fix or adapt the hardcoded path(s) - if the program reads (or can be made to read) each path from an environment variable, then you can use the <code>create_wrapper</code> utility function to create a wrapper script that will set the path appropriately. Run <code>git grep create_wrapper</code> in the <code>meta</code> subdirectory to see examples.

	=== Files I installed into /opt or some other path never make it into the sysroot but I need them - how do I fix this? ===		=== Files I installed into /opt or some other path never make it into the sysroot but I need them - how do I fix this? ===

PaulEggleton: /* Can I disable shared state? */

2020-07-01T01:15:59Z

Can I disable shared state?

← Older revision		Revision as of 01:15, 1 July 2020
Line 323:		Line 323:
	=== Can I disable shared state? ===		=== Can I disable shared state? ===

	You cannot, no. Shared state is an intrinsic part of staging files into the sysroot. It is possible to construct a recipe that bypasses sstate for some tasks (the kernel does this), however this is quite difficult and if not done properly will lead to many other problems.		You cannot, no. Shared state (sstate) is an intrinsic part of staging files into the sysroot. It is possible to construct a recipe that bypasses sstate for some tasks (the kernel does this), however this is quite difficult and if not done properly will lead to many other problems.

	Almost always when you are having a problem with shared state the issue is either (a) you're adding/changing files in the sysroot directly (i.e. outside sstate control), or (b) what is being placed into the sysroot isn't relocatable. The solution for (a) is do not do that - files should always be installed under ${D} within do_install and then a subset of those are staged into the sysroot automatically. For (b) you need to fix or adapt the hardcoded path(s) - if the program reads (or can be made to read) each path from an environment variable, then you can use the create_wrapper utility function to create a wrapper script that will set the path appropriately. Run "git grep create_wrapper" in the meta subdirectory to see examples.		Almost always when you are having a problem with shared state the issue is either (a) you're adding/changing files in the sysroot directly (i.e. outside sstate control), or (b) what is being placed into the sysroot isn't relocatable. The solution for (a) is do not do that - files should always be installed under <code>${D}</code> within <code>do_install</code> and then a subset of those are staged into the sysroot automatically. For (b) you need to fix or adapt the hardcoded path(s) - if the program reads (or can be made to read) each path from an environment variable, then you can use the <code>create_wrapper</code> utility function to create a wrapper script that will set the path appropriately. Run <code>git grep create_wrapper</code> in the <code>meta</code> subdirectory to see examples.

	=== Files I installed into /opt or some other path never make it into the sysroot but I need them - how do I fix this? ===		=== Files I installed into /opt or some other path never make it into the sysroot but I need them - how do I fix this? ===

← Older revision		Revision as of 01:30, 1 July 2020
Line 68:		Line 68:
	First, double-check that you haven't misspelled the variable name.		First, double-check that you haven't misspelled the variable name.

	The main tool to help troubleshoot any variable-related issue is <code>bitbake -e</code> - this lists all the variables and the complete history of how each one has been set (use <code>bitbake -e recipename</code> if you're dealing with issues in a variable value within a recipe as opposed to the global level). Usually it's best to pipe this through <code>less</code> so you can easily see the history - within less you can press / to search for the variable name. Often you will be dealing with the behaviour of a variable within the context of a specific recipe, so specify that recipe on the <code>bitbake -e</code> command line to get the variables as set within the context of the recipe rather than the global context.		The main tool to help troubleshoot any variable-related issue is <code>bitbake -e</code> - this lists all the variables and the complete history of how each one has been set (use <code>bitbake -e <recipename></code> if you're dealing with issues in a variable value within a recipe as opposed to the global level). Usually it's best to pipe this through <code>less</code> so you can easily see the history - within less you can press / to search for the variable name. Often you will be dealing with the behaviour of a variable within the context of a specific recipe, so specify that recipe on the <code>bitbake -e</code> command line to get the variables as set within the context of the recipe rather than the global context.

	If you're setting a variable in a bbappend, double-check that the bbappend is actually being applied - see the next question.		If you're setting a variable in a bbappend, double-check that the bbappend is actually being applied - see the next question.
Line 79:		Line 79:
	# Check that the bbappend is being picked up by running <code>bitbake-layers show-appends</code> - if your bbappend file isn't listed, it could be named incorrectly (such that it doesn't match the recipe name) or it may be that the BBFILES value in the conf/layer.conf for the layer containing the bbappend file doesn't include an expression that will match the bbappend files.		# Check that the bbappend is being picked up by running <code>bitbake-layers show-appends</code> - if your bbappend file isn't listed, it could be named incorrectly (such that it doesn't match the recipe name) or it may be that the BBFILES value in the conf/layer.conf for the layer containing the bbappend file doesn't include an expression that will match the bbappend files.
	# If there are multiple versions of the recipe you have bbappended, it could be that the actual recipe being built is a different version than the one you have bbappended. <code>bitbake-layers show-recipes <recipename></code> will list all the versions, with the first one listed being the one that will be built. If this is the case there are several different solutions to this - (a) Rename your bbappend to match the version being built, (b) use a % wildcard in your bbappend so it will apply to any version, (c) set <code>PREFERRED_VERSION_<recipename></code> in the configuration to select a the version you want to be built.		# If there are multiple versions of the recipe you have bbappended, it could be that the actual recipe being built is a different version than the one you have bbappended. <code>bitbake-layers show-recipes <recipename></code> will list all the versions, with the first one listed being the one that will be built. If this is the case there are several different solutions to this - (a) Rename your bbappend to match the version being built, (b) use a % wildcard in your bbappend so it will apply to any version, (c) set <code>PREFERRED_VERSION_<recipename></code> in the configuration to select a the version you want to be built.
	# Finally, as with any other issue with setting variables, use <code>bitbake -e recipename \| less</code> and search with <code>/</code> to see the history of how the variable has been set - you may find that the value you're trying set is being overridden.		# Finally, as with any other issue with setting variables, use <code>bitbake -e <recipename> \| less</code> and search with <code>/</code> to see the history of how the variable has been set - you may find that the value you're trying set is being overridden.

	=== I'm getting warnings that a recipe is tainted - what does this mean? ===		=== I'm getting warnings that a recipe is tainted - what does this mean? ===
Line 109:		Line 109:
	Check the simple stuff: verify that the package is really in the image - look at the manifest file next to the image to ensure the package is listed. Also if you're flashing the image, double-check that you did indeed flash the right image and if there are multiple partitions / storage devices on your board or device that you're booting the one that you think you are.		Check the simple stuff: verify that the package is really in the image - look at the manifest file next to the image to ensure the package is listed. Also if you're flashing the image, double-check that you did indeed flash the right image and if there are multiple partitions / storage devices on your board or device that you're booting the one that you think you are.

	Once you're sure of the above, it may be a matter of the package splitting - a lot of recipes split less commonly used components out to separate packages, so it's possible that the files you are looking for are in a different package. You can look at the recipe for this (look for PACKAGES and FILES statements) or assuming the recipe has been built, you can use <code>oe-pkgdata-util list-pkgs -p recipename</code> and <code>oe-pkgdata-util list-pkg-files</code> to inspect the packages provided by the recipe and the files they contain. Once you find the right package you can add it to your image.		Once you're sure of the above, it may be a matter of the package splitting - a lot of recipes split less commonly used components out to separate packages, so it's possible that the files you are looking for are in a different package. You can look at the recipe for this (look for PACKAGES and FILES statements) or assuming the recipe has been built, you can use <code>oe-pkgdata-util list-pkgs -p <recipename></code> and <code>oe-pkgdata-util list-pkg-files</code> to inspect the packages provided by the recipe and the files they contain. Once you find the right package you can add it to your image.

	=== I'm required to set LIC_FILES_CHKSUM but the software I'm building doesn't have a license statement, what do I do? ===		=== I'm required to set LIC_FILES_CHKSUM but the software I'm building doesn't have a license statement, what do I do? ===
Line 234:		Line 234:
	In the 1.8 (fido) release and newer you can use the following command, assuming the recipe has already been built:		In the 1.8 (fido) release and newer you can use the following command, assuming the recipe has already been built:

	oe-pkgdata-util list-pkgs -p recipename		oe-pkgdata-util list-pkgs -p <recipename>

	Alternatively you can look in the "packages-split" subdirectory under the work directory for the recipe - each package produced by the recipe will have a subdirectory under that. If you're not sure how to find the work directory you can run the following command:		Alternatively you can look in the "packages-split" subdirectory under the work directory for the recipe - each package produced by the recipe will have a subdirectory under that. If you're not sure how to find the work directory you can run the following command:

	bitbake -e recipename \| grep ^WORKDIR=		bitbake -e <recipename> \| grep ^WORKDIR=

	Before a recipe gets built it is a bit trickier, since the system often doesn't know exactly which packages will be produced until do_package time; this is particularly true for recipes that package plugins or modules (e.g. kernel modules). You can get a reasonable idea though by looking at the value of PACKAGES (and PACKAGES_DYNAMIC for recipes that produce plugins).		Before a recipe gets built it is a bit trickier, since the system often doesn't know exactly which packages will be produced until do_package time; this is particularly true for recipes that package plugins or modules (e.g. kernel modules). You can get a reasonable idea though by looking at the value of PACKAGES (and PACKAGES_DYNAMIC for recipes that produce plugins).