Combo-layer

Introduction

combo-layer is a command-line utility for creating and maintaining git repositories that are a combination of several other git repositories. It was created to help stitch together multiple layers on top of OE-Core, in particular to manage the Poky repository, although there is nothing specific in the code about OE layers.

It works by extracting the changes from each component repository as patches, modifying them in a customisable way, and then applying them to the combo repository. It can handle multiple branches.

Note: because of the patch-shipping method combo-layer uses, it does not work well with merge commits. Typically with OE-Core and most other layers we avoid merges (i.e. branches are rebased before applying) so this is not a significant limitation for us.

Why not use other similar tools (repo, submodules, subtree)?

combo-layer performs a similar function to other available tools; however each of these has their own limitations and none of them allows modification of changes as they pass through (either manually or automatically). Additionally combo-layer allows you to take just part of a repository, i.e. one or more subdirectories or files.

Setup

Note: the following assumes OE-Core/Poky's scripts subdirectory is in your PATH (as it would be after sourcing oe-init-build-env); however this is not mandatory - you can just run it directly e.g. ../../oe-core/scripts/combo-layer ....

1. Create an empty directory to hold your combo repository and cd into it
2. Create a configuration file that tells combo-layer about the component repositories. You can copy the example config file from the scripts subdirectory and modify it. By default, combo-layer expects to find this config file at conf/combo-layer.conf from the current directory, however it can be anywhere if you use the -c option. If the configuration file is within the combo repository and not set to be ignored, it will be committed if changed.
3. Run combo-layer init - this will fetch the component repos and create a combo repo based on the current revision on the specified branches for each component.
4. You can now make any desired changes to the initial state of the combo repository, and when you're ready use git commit to commit it.

Updating

Updating is as easy as combo-layer update - this will pull each component repository, prepare patches for the changes since the last update, and then apply them to the combo repository. If desired, you can specify -i to use interactive mode which allows you to edit the list of patches and the patches themselves before they are applied.

Advanced configuration

Hook script

A script can be specified for each component to be called automatically for each patch before it is applied using the hook option in the configuration file. The script is called with three arguments: the patch file, the revision from the component repo, and the component name in that order. There is an example hook script in scripts/combo-layer-hook-default.sh which will add the component name and upstream revision to the commit message and prefix the first line (shortlog) with the component name.

Python code in configuration

If a value in the configuration file starts with @ it will be assumed to be a python statement to be evaluated. For example, you could use it to read the value of an environment variable:

[bitbake]
...
local_repo_dir = @os.getenv("LOCAL_REPO_DIR") + "/bitbake"
...

Split local configuration

It is possible to split out the local parts of the configuration (e.g. local_repo_dir and last_revision) so that the generic parts within the config file can be committed to the combo repository and the local parts (and last_revision which changes often) remain local. To set this up:

1. Create a file with the same name and location as the main config file except with '.conf' replaced with '-local.conf'. Thus, if you're using the default path/name of conf/combo-layer.conf, the local counterpart would be conf/combo-layer-local.conf.
2. Move the local configuration options from the main config file to the local one. The last_revision option should be within a section named using the component name and combo layer branch name separated by a | (pipe) character. Other options need not use this style.
3. If the configuration files are within the combo repository, add the local conf file to .gitignore and commit it.

An example local configuration file (values may be artificial):

[bitbake]
local_repo_dir = ../repos/bitbake
   
[oe-core]
local_repo_dir = ../repos/oe-core
   
[bitbake|master]
last_revision = db689a99beffea1a285cdfc74a58fe73f1666987
   
[oe-core|master]
last_revision = 121a1499a81706366acc0081272a6bff634d4d62
   
[bitbake|denzil]
last_revision = 24b631acdaa143a4de39c6e1328849660c66f219
   
[oe-core|denzil]
last_revision = 741146fa90f28f7ce8d82ee7f7e254872d519724

(In the above example it is assumed that the main config file that is committed to the combo repository sets the branch option for each component as appropriate.)

Practical example - How to use combo-layer to create a mediaserver layer for Poky

In this example we're going to use the combo-layer tool to create a new layer that can easily be integrated into a yocto image.

Setting up

Let's start from an empty folder. Create a conf/ directory inside and cd into it. Now we have to create a configuration file for the combo-layer tool that will contain the package dependencies for our layer. For handiness we're going to name the file combo-layer.conf. An example of combo-layer.conf:

[meta-guacamayo]                                                                   
src_uri = https://github.com/Guacamayo/meta-guacamayo                              
local_repo_dir = /intel/poky/builds/combo/repos/meta-guacamayo                     
dest_dir = meta-dlna                                                               
last_revision =                   
file_filter = meta-guacamayo/recipes-core/images/guacamayo-image-mediaserver.bb meta-guacamayo/recipes-core/tasks/task-guacamayo.bb meta-guacamayo/recipes-connectivity/connman meta-guacamayo/recipes-gnome/gsettings-overrides meta-guacamayo/recipes-upnp meta-guacamayo/conf/layer.conf meta-guacamayo/classes/guacamayo-image.bbclass meta-guacamayo/classes/gsettings.bbclass meta-guacamayo/classes/vala.bbclass meta-guacamayo/recipes-multimedia/alsa meta-guacamayo/recipes-multimedia/gstreamer meta-guacamayo/recipes-core/guacamayo-session/guacamayo-session-mediaserver* meta-guacamayo/recipes-gnome/dconf meta-guacamayo/recipes-demos meta-guacamayo/recipes-gnome/libgee meta-guacamayo/recipes-devtools/vala meta-guacamayo/recipes-devtools/orc meta-guacamayo/recipes-devtools/dbus meta-guacamayo/recipes-support/tracker meta-guacamayo/recipes-support/gvfs meta-guacamayo/recipes-support/gmime meta-guacamayo/recipes-support/fuse meta-guacamayo/recipes-core/glib-2.0 meta-guacamayo/COPYING.MIT

Now, that's everything's set, you can run the combo-layer init command in the root folder. In this case the command will generate the meta-dlna folder which will include all the recipes that are needed for the new layer.

Integrate the new layer into Poky

Copy the meta-dlna folder into the root directory of a poky distribution. Cd into the poky folder. In order to use this new layer, you'll have to add it into the build/conf/bblayers.conf:

[...]
BBLAYERS="
         $POKY_HOME/meta-dlna/meta-guacamayo
[...]

For the mediaserver image, you're going to need to add some demo content. For this, edit the build/conf/local.conf:

[...]
IMAGE_FEATURES_append = " guacamayo-demos"
[...]

For this particular example, you'll also need to add some distro features. This is done by writing the following line in your build/local.conf:

[...]
DISTRO_FEATURES = "ext2 largefile usbhost xattr nfs zeroconf pci ${DISTRO_FEATURES_LIBC} x11" 
[...]

Notes on how Richard manages the Poky repository

Richard (RP) uses combo-layer to construct the poky repository from its component repositories (yocto-docs, oe-core, bitbake and meta-yocto). The setup is that there are a set of repositories in /home/x/Repos and the Repos/pokyconfig directory contains the actual combo-layer configuration data. Rather than share the contents of all the files, the pokyconfig diriectory is shared as a repository at http://git.yoctoproject.org/cgit.cgi/poky-config/. This would be easy to change to a different set of local paths. The buildit and buildit-next commands automatically construct the poky repository based on commits in the other trees. This includes handling of the various release branches. The pushit command syncs the local state to the repositories on the main server.