Binary Distro Prototype

Scope of a Yocto Binary Distribution Prototype

The Yocto Project has received funding for developing the processes and tooling for enabling binary distributions (see the Engineering Request for Quotation). As part of that, the project would need some form of prototype test binary distro to develop and test this. The main goal of this would be to develop the tools, and documentation needed to build a binary distribution and allow application and system developers to progress through a hierarchy of different uses of the resulting environment:

Beginner use case: begin by fetching a binary image, ready to boot on a target machine. Extend the system by installing extra applications through package feeds.
Intermediate use case: tweak and rebuild existing packages, compile new applications and modify images by using an eSDK and other binary artifacts.
Advanced use case: completely reorganize and optimize the system for a specific target, switching to building from source

The first step in this effort is to define the scope of the prototype we will develop. We are looking for feedback and comments about issues we may not have considered.

Bear in mind that a prototype has to be limited in scope. However, the developed tools, procedures and documentation will open a pathway for the Yocto Project and OpenEmbedded community to experiment with more architectures, features, recipes and images.

Also bear in mind that the effort is not planning to adopt, replace or repeat any existing distributions such as Ångström or Yoe.

Notes from discussions:

Alex Kanavin suggests as "Intermediate" step to use the build bundles that he submitted. However, this is closer to the "Advanced" use case as this already uses BitBake and OE/Yocto (discussion).

Target architectures and machines

The first prototype would only target the below architectures:

"genericx86-64" architecture through the genericx86-64 machine.
"arm64" architecture through the qemuarm64 machine (and later "genericarm64" when ready).

Testing would happen through QEMU system emulation. The emulated architectures will be sufficient to test the outputs of this effort, but the developed tooling and documentation should allow the community to experiment with other target architectures and machines.

Target tunes

The binary images and other artifacts should be generated for the below tunes:

See DEFAULTTUNE for details.

Distribution, Init Manager, C library

Images and package feeds will be generated for the poky-altcfg distribution, using the systemd init manager.

Choosing systemd makes the binary reference distribution as generic as possible, enabling the widest set of features, and being more familiar to more users. Of course, switching to other init managers and more restricted features will be possible by later compiling the system from source (advanced use case).

This also implies the choice of the GNU C library (glibc). Selecting glibc also makes sense to enable the widest possible selection of packages, and some packages have limited or no support for the Musl C library. That's typically the case of systemd, which upstream only supports glibc.

Last but not least, people who want to use the Musl C library probably do that to optimize the size of their system, which is a use case not well adapted to using ready-made binary images. Such an advanced use case will be addressed by re-compiling from source, after testing initial prototypes with the binary image based on glibc.

Binary images

The prototype will provide as a reference, for each targeted architecture:

The root filesystem binary image, generated by building the core-image-full-cmdline target.
An image containing on-target development tools (e.g compiler, debugger)
The corresponding SD card images to be booted through QEMU. Other boot methods will be possible, but will be outside of the scope of the reference distribution.

Package feeds

For each targeted architecture, several package feeds will be available to allow for bugfix or security updates or to extend the image with additional packages:

Architecture independent feed
Architecture dependent feed
Machine dependent feed

See Using Runtime Package Management for details.

The feeds for the first prototype will contain all the packages which are supported by the poky-altcfg distribution as produced by the bitbake world command for the set of layers supplied by Poky.

Available reference package formats

To keep the first prototype simple, we will only release images and package feeds based on the "ipk" format supported by opkg package manager.

Of course, the OpenEmbedded build system also supports the deb and rpm package formats, so users will be able to generate their own images and feeds based on these other formats.

Another reason for not providing "deb" and "rpm" packages is that we do not want inexperienced users to expect to be able to grab and use such packages produced for regular Linux distributions (Debian, Fedora, etc.). Being built with different C library versions and toolchains, the corresponding binaries are most likely to be broken anyway.

Available reference binary artifacts and deliverables

The first prototype will make the below binary artifacts publicly available:

Root filesystem images (as specified above)
Binary package feeds (as specified above)
PR database, so that people can manage updates to packages, by importing the database in a local PR server.
Prebuilt object data ("Shared State cache) through a public server (see SSTATE_MIRRORS), to reuse binary output already built by the Yocto Project autobuilders, to reduce the compile time of additional packages.
Hash Equivalence data through a public server (see BB_HASHSERVE_UPSTREAM), to increase the reusability of prebuilt objects.

Share State and Hash Equivalence data would be only useful to "Intermediate" and "Advanced" use cases, to rebuild parts or the whole system from source.

Such artifacts will be served through a Content Delivery Network (CDN), to make sure that they can be downloaded fast enough from any location in the world, in a way that remains much faster than re-compiling such artifacts from sources.

Those binaries should also be released along with:

Sources corresponding to the software used to build the image
SPDX output for the initial image
The build system and layer sources. A good fit would be Alex Kanavin's build bundles.

Defining policies and processes

This reference binary distribution prototype will also drive the documentation of policies and process to address future work such as:

What criteria should a new CPU architecture or platform tuning meet to be added / tested?
What criteria should other layers meet to be tested, in addition to being already tested by the Autobuilder? Should these criteria only apply to specific recipes or to the entire layers containing them?

Testing plans

Here are the tests we plan to implement:

Producing and testing binary images through the autobuilder
Making nightly builds available to testers
Testing that package updates apply successfully:
- Within the current stable branch (nanbield): from one release update to the next, and from the latest release update to the tip of the branch
- Between release branches: from the latest update to our latest LTS (kirkstone), to the latest update to the current stable release (nanbield).
Testing that it's also possible to build and deploy packages from the meta-openembedded and meta-virtualization layers, as a way to assess the relevance of our policies and process for additional layers or recipes. This would also be a way to demonstrate the ability to build container images, which would be a great way to ship binary distributions.

We will work with recipe maintainers to publish fixes for update issues that testing will find in recipes. Each issue will be reported in Bugzilla.

Open questions

Issues with Yocto Project being pedantic in its use of arch strings such as "intel-x86-64" instead of "x86-64". This conflicts with typical use in the greater Linux and "binary" community, which may not recognize the "extended" arch strings. Examples: libresolv and libdnf (reported by Mark Asselstine).

Achievements

Fixed bugs

Bug 15292 - systemd-compat-units: error at package post-install time

Missing features

Not implemented yet:

Ability to update the system SPDX description after installing extra packages or package updates. The current way SPDX is generated doesn't allow to generate -spdx packages, which would have made this possible (discussion).

Out of scope

Features outside of the scope of this effort, but definitely worth keeping track of for future efforts, or for people already maintaining binary distributions.

Binary distribution installer: not absolutely necessary but users may expect one. For example the Yoe distro offers one, and so does Raspberry Pi.