What's New In 2.1

What's new in the Yocto Project 2.1 release

As with most Yocto Project releases, the Yocto Project 2.1 "krogoth" release comes with a large number of changes in different areas. If one were to try to extract a general theme for 2.1 however it would probably be cleaning out the dusty corners of the system, as well as finishing off some functionality that has been a long time in the making. This guide doesn't cover all changes (in fact, it doesn't cover the hundreds of bugfixes and minor tweaks made since the last release) but attempts to give an overview of the most visible improvements.

General

There was the usual crop of recipe upgrades (~240 recipes upgraded), headed with an upgrade of the Linux kernel to 4.4 (plus 4.1 for the stable LTSI kernel), gcc 5.3 and glibc 2.3. While we are on the topic of C libraries, the musl size-optimised C library has been upgraded and fully integrated in this release - whilst there was some earlier support via the addition of an extra layer in previous releases, musl can now be built out of the box and in fact is now used in the reference "poky-tiny" distribution, replacing the earlier pared down glibc configuration.

Full GObject introspection support replaces the stub support available in previous releases. This enables better support for building applications that make use of GObject introspection (e.g. anything implemented using language bindings for GTK+3, which includes parts of the GNOME stack). The caveat is that QEMU system emulation must be supported for the target architecture in order to support this.

The uninative feature which allows shared state to be used across different host distributions more effectively has been extended and enabled by default in the Poky reference distribution. An inc file (meta/conf/distro/include/yocto-uninative.inc) has been created to make this easier to incorporate into your custom distro configuration if desired.

Static library generation is now disabled for most recipes by default in the Poky reference distribution. Disabling this generation saves some build time as well as the size used for build output artifacts. Static libraries are disabled through an inc file (meta/conf/distro/include/no-static-libs.inc) making this easier to incorporate into your custom distro configuration if desired.

.debug directories are now automatically packaged in the -dbg package - now it no longer matters where you install binaries, their split out debug symbols will always go into the -dbg package instead of triggering a QA error/warning that you needed to resolve. Additionally, build sysroot paths are now excluded from debug symbol files as part of a wider effort to remove build path dependent binary output.

Added optional support for signing ipk packages as well as shared state artifacts.

Added an npm module fetcher to BitBake and corresponding support class in OE-Core (requires separate nodejs recipe). Because of the way npm fetches dependencies as part of its operation, it was previously difficult to write reliable recipes for npm recipes - modules were fetched on the fly and outside of the do_fetch task, leading to problems with reproducibility as well as the inability to work behind a network proxy.

Switching PREFERRED_PROVIDER to another virtual/* providers now results in old files being removed from the sysroot; earlier this would likely result in "The recipe <recipe> is trying to install files into a shared area when those files already exist" errors, requiring either explicit cleanup or simply deleting TMPDIR and restarting the build.

The runqemu script for more easily starting the QEMU system emulator gained support for qcow2 and vdi images, as well as images produced by the wic tool.

Dependencies on busybox can now be replaced relatively easily with another provider (e.g. toybox).

bash-completion has been imported from the meta-oe layer, providing optional support for extensible auto-completion when the bash shell is installed. Bash completion support has been added to individual recipes; this is placed in a separate -bash-completion package so that it can be installed separately if desired.

swupd (The SoftWare UPDater) from Clear Linux provides a new way of adding functionality to and updating a Linux-based OS. swupd uses binary-delta technology to efficiently update only the files that have changed between OS updates. This means that updates are small, resulting in fast downloads, and fast to apply. swupd is made available as a separate meta-swupd layer alongside the Yocto Project 2.1 release. See https://wiki.yoctoproject.org/wiki/Meta-swupd for more information.

Optimisation and cleanup

• The Hob UI has now been removed. Hob, which uses GTK+2, has been largely unmaintained for several releases. The Toaster web-based UI now provides all of the functionality of Hob, plus much more. See below for further information on Toaster.

• The Application Developer Toolkit (ADT) has also been removed for 2.1, having been functionally replaced by the standard and extensible SDKs. (The Eclipse plugin is unaffected by this change however.) More details on the extensible SDK below.

• The troublesome "-e MAKEFLAGS=" has finally been removed from default EXTRA_OEMAKE. This was a historical accident and was something which recipe writers often needed to work around. Additionally, libexecdir has been reverted to ${prefix}/libexec to be in line with other distributions.

• The meta-yocto layer has been renamed to meta-poky to better match its purpose, which is to provide the Poky reference distribution. (The meta-yocto-bsp layer retains its original name since it provides reference machines for the Yocto Project and it is otherwise unrelated to Poky.)

• All support for Qt 4.x has been moved out to a separate meta-qt4 layer since Qt 4 is no longer supported upstream. Newer Qt-based applications can make use of the meta-qt5 layer, which is actively maintained by the OpenEmbedded community.

• libjpeg has been replaced by libjpeg-turbo - most traditional Linux distributions have made this move already.

• libav has been replaced by ffmpeg. Again, many traditional Linux distributions have made this move already.

• udev has been replaced by eudev - as part of the Linux kernel 4.4 upgrade, for those still using SysV init (which is still the default) an update of the standalone udev was required. The easiest course of action here turned out to be to migrate to the latest stable version of Gentoo's eudev fork.

• The first step in moving towards the changing the default of the expand parameter in d.getVar() from False to True has been made by removing the default and making the parameter mandatory. This will force specifying the value; in a future release once layer maintainers have had a chance to adapt, a default of True will be set up.

Toaster

The Toaster web-based UI saw significant work in this release, bringing it to the point where it is now able to fully replace the deprecated Hob. Improvements include:

• Image customization: Toaster now allows you to create you own custom images, by modifying the package content of existing ones. Toaster provides dependency information for easy package addition and removal, builds your custom images, and generates a recipe (.bb) file that you can download.
• Isolated project configurations: Toaster now creates a separate build directory per project, avoiding build failures caused by the reuse of intermediate build artifacts. This provides cleaner configurations that can be easily shared, and improves build reliability. Shared state artifacts and downloads are still shared by default but this can be configured per project if desired.
• Cancel builds from the Toaster web interface: you can now cancel the builds you start from the Toaster web interface with a single click, without having to switch to the command line interface.
• Improved interaction with BitBake: we have removed the need for the BitBake observer process, and we run the BitBake server only during the build, stopping it once the build is done. We have also returned to executing the BitBake server from the selected project release (Yocto Project 2.0 or 2.1), while the BitBake client runs from the same codebase used to start Toaster. This allows us to provide backward compatibility with the previous stable release of the Yocto Project (2.0), enabling the next feature in this list
• Support for building two stable releases: Yocto Project 2.1 "Krogoth" and Yocto Project 2.0 "Jethro" releases, and/or the OpenEmbedded jethro and krogoth branches. Toaster also builds the development branch (master).
• Upgraded to Django 1.8.x from 1.6.x: Django 1.8 has been designated as a Long Term Support release, with extended support until at least April 2018.
• Improved build statistics data: following changes to buildstats, Toaster now provides a more accurate measure of Disk I/O per task, and displays both system and user CPU time.
• Fix date filters for builds: ensure you can retrieve a subset of builds by specifying a date range.
• Show PROVIDES information in recipe dependencies: Toaster now identifies recipe build dependencies satisfied via the PROVIDES variable.

Extensible SDK

The Extensible SDK (bitbake -c populate_sdk_ext) was introduced in preview form in the previous release, and builds on top of the standard SDK functionality with additional tools allowing developers to more easily add new applications / libraries to the build, modify the source of existing components, test changes on the target and integrate them into a system image. In 2.1 it has been solidified with the following added features:

• Support for building a minimal (~35MB) SDK installer with the rest installed on-demand from shared state
• Ability to bring in pkgdata for world, which improves dependency mapping when adding new components through "devtool add" as well as the ability to find recipes by keyword using "devtool search"
• Added the ability to produce a derivative SDK installer containing items in the workspace in prebuilt form

Going together with this, a new SDK Manual has been written providing information on how to use (and build) the standard SDK, extensible SDK and also how to install and use the Eclipse plugin.

devtool

The devtool command is at the heart of the aforementioned extensible SDK, but is also available as a tool alongside a standard installation of the build system. It provides subcommands to add new software (auto-generating a recipe based on a source tree), modify the source of an existing recipe, easily deploy installed output to the target device, generate patches from source changes, and more. Improvements to devtool in 2.1 include:

• Proper support for modifying the kernel source - "devtool modify virtual/kernel" will not only check out the source tree for the kernel but also now correctly set up the initial configuration specified via the recipe.
• New "edit-recipe" subcommand to quickly edit created recipes
• New "configure-help" subcommand which presents the current value of EXTRA_OECONF and the output of the configure script's --help option (if applicable) to make setting configure arguments in a recipe a bit easier
• New "sync" subcommand to synchronise a previously extracted source tree with upstream
• Improved the presentation of commands in the help text, and simplified command-line usage for the add and modify subcommands
• Deployment to the target has been reworked - deploy-target now preserves any existing files and writes the deployed files list to the target device; undeploy-target restores previously existing files and can now be directed to undeploy all deployed recipes at once.

The "devtool add" command which auto-generates a recipe for a source tree internally uses "recipetool create" to do just the recipe creation process. recipetool create has been enhanced as follows:

• Support for extracting dependencies from cmake files
• Significantly improved autotools support
• Basic support for generating custom kernel recipes based on the linux-yocto-custom template
• Support for npm fetcher/class
• Support for out-of-tree kernel modules
• Extract recipe name/version from filename, github/bitbucket URLs, build scripts so that these don't always need to be specified
• Extract SRC_URI from the remote in a local git repository if specified
• Improved license detection - new "crunching" technique to ignore formatting differences between license texts that would otherwise prevent automated matching
• Support creating native variants or standalone native/nativesdk recipes
• Support specifying a subdirectory within the fetched source - useful when adding a recipe for a subcomponent of a source tree (where the source tree structure permits this)
• Handle https://....git URLs - BitBake expects git:// so any URLs in the former form will be converted
• Allow extension of recipe creation process through additional layers - e.g. adding meta-qt5 now enables support for creating recipes for Qt5-using software

Additionally, when externalsrc is in use shortcut symlinks for the work directory and logs directory are now created during do_configure. This will be mostly visible in the context of building a recipe when using devtool modify, devtool add or devtool upgrade (all of which make use of the externalsrc class).

You can find more information on devtool in the Yocto Project Development Manual.

Images

The following improvements have been made to building images:

• Image generation is now split out from filesystem generation - i.e. turning the filesystem into the various image file formats, e.g. tar.gz and ext3, is now done in separate tasks. This not only simplifies the code but also makes it more visible as to what part of the process is being run at a given time - instead of a long-running do_rootfs you'll see a slightly shorter do_rootfs followed by several do_image_* tasks, one per type specified in IMAGE_FSTYPES.
• Write out an image license manifest, which includes other image dependencies such as bootloaders. These may not part of the actual image file (depending on the type) but are associated with it and thus this new manifest gives a complete picture of the licenses of the software shipped with the image.
• Support chaining compression (aka conversion) commands and use to allow producing checksums for images
• Allow vm and live image types to be built at the same time

The "wic" standalone image creator has seen a couple of improvements:

• Allow specifying a custom config for the bootloader
• Implement includes in wks files

The mkefidisk.sh script has seen a couple of improvements as well:

• Display boot log on console
• Added a startup script for automated boot

BitBake

Unless you're familiar with the code it can sometimes be hard to determine where BitBake ends and OpenEmbedded starts, but BitBake the tool does remain independent of the OE metadata even though it is developed alongside it. Improvements to bitbake in this release include:

• Set process names to be meaningful - here's some output from pstree demonstrating the new naming:

└─bash───KnottyUI───Cooker─┬─Worker─┬─fixesproto:conf───run.do_configur───configure
                           │        ├─fontconfig:conf───run.do_configur───autoreconf─┬─autoheader───sh───autom4te───sh───m4
                           │        │                                                └─autom4te
                           │        ├─groff-native:co───run.do_configur───configure───configure───g++─┬─as
                           │        │                                                                 └─cc1plus
                           │        ├─icu-native:conf───run.do_configur───configure───sh
                           │        ├─libpng:configur───run.do_configur───autoreconf─┬─autom4te
                           │        │                                                └─automake───autom4te
                           │        ├─libusb1:configu───run.do_configur───autoreconf───autom4te
                           │        └─libxext:configu───run.do_configur───configure
                           ├─pseudo───Worker (Fakeroo
                           ├─2*[{Cooker}]
                           └─{ProcessEQueue}

• Messages produced from within tasks are now always prefixed with the recipe/task name, avoiding the problem of confusing warning/error messages with no context
• Added support for unpacking 7-Zip archives
• Added --setscene-only option to only run setscene tasks (i.e. in OE context this means only tasks whose output can be restored from shared state).
• Added ability to build up to but not including a task
• Provide more information with "multiple .bb files are due to be built" message
• List close matching runtime providers with close matches when invalid target specified
• Allow BBMASK to contain multiple regular expressions
• Adjust styling of graph lines for RRECOMMENDS and RDEPENDS in -g output so that the difference between these is clearer

Further information

As always, the reference manual provides a section on Moving to the Yocto Project 2.1 Release which you should definitely read if you are upgrading.

The 2.1 release notes cover the above information in slightly briefer form along with a list of upgraded recipes.