Binary Distro Prototype

Scope of a Yocto Binary Distribution Prototype

The Yocto Project has received funding for developing the processes and tooling for enabling binary distributions (see the Engineering Request for Quotation). As part of that, the project would need some form of prototype test binary distro to develop and test this. The main goal of this would be to develop the tools, and documentation needed to build a binary distribution and allow application and system developers to progress through a hierarchy of different uses of the resulting environment:

• Beginner use case: begin by fetching a binary image, ready to boot on a target machine. Extend the system by installing extra applications through package feeds.
• Intermediate use case: tweak and rebuild existing packages, compile new applications and modify images by using an eSDK and other binary artifacts.
• Advanced use case: completely reorganize and optimize the system for a specific target, switching to building from source

The first step in this effort is to define the scope of the prototype we will develop. We are looking for feedback and comments about issues we may not have considered.

Bear in mind that a prototype has to be limited in scope. However, the developed tools, procedures and documentation will open a pathway for the Yocto Project and OpenEmbedded community to experiment with more architectures, features, recipes and images.

Also bear in mind that the effort is not planning to adopt, replace or repeat any existing distributions such as Ångström or Yoe.

Notes from discussions:

• Alex Kanavin suggests as "Intermediate" step to use the build bundles that he submitted. However, this is closer to the "Advanced" use case as this already uses BitBake and OE/Yocto (discussion).

Target architectures and machines

The first prototype would only target the below architectures:

• "x86-64" architecture through the qemux86-64 machine.
• "arm64" architecture through the qemuarm64 machine.

Testing would happen through QEMU system emulation. The emulated architectures will be sufficient to test the outputs of this effort, but the developed tooling and documentation should allow the community to experiment with other target architectures and machines.

Target tunes

The binary images and other artifacts should be generated with the default tunes of the qemux86-64 and qemuarm64 machines.

See DEFAULTTUNE for details.

Distribution, Init Manager, C library

Images and package feeds will be generated for the poky-altcfg distribution, using the systemd init manager.

Choosing systemd makes the binary reference distribution as generic as possible, enabling the widest set of features, and being more familiar to more users. Of course, switching to other init managers and more restricted features will be possible by later compiling the system from source (advanced use case).

This also implies the choice of the GNU C library (glibc). Selecting glibc also makes sense to enable the widest possible selection of packages, and some packages have limited or no support for the Musl C library. That's typically the case of systemd, which upstream only supports glibc.

Last but not least, people who want to use the Musl C library probably do that to optimize the size of their system, which is a use case not well adapted to using ready-made binary images. Such an advanced use case will be addressed by re-compiling from source, after testing initial prototypes with the binary image based on glibc.

Binary images

The prototype will provide as a reference, for each targeted architecture:

• The root filesystem binary image, generated by building the core-image-full-cmdline target.
• An image containing on-target development tools (e.g compiler, debugger)
• The corresponding SD card images to be booted through QEMU. Other boot methods will be possible, but will be outside of the scope of the reference distribution.

Package feeds

For each targeted architecture, several package feeds will be available to allow for bugfix or security updates or to extend the image with additional packages:

• Architecture independent feed
• Architecture dependent feed
• Machine dependent feed

See Using Runtime Package Management for details.

The feeds for the first prototype will contain all the packages which are supported by the poky-altcfg distribution as produced by the bitbake world command for the set of layers supplied by Poky.

Available reference package formats

To keep the first prototype simple, we will only release images and package feeds based on the "ipk" format supported by opkg package manager.

Of course, the OpenEmbedded build system also supports the deb and rpm package formats, so users will be able to generate their own images and feeds based on these other formats.

Another reason for not providing "deb" and "rpm" packages is that we do not want inexperienced users to expect to be able to grab and use such packages produced for regular Linux distributions (Debian, Fedora, etc.). Being built with different C library versions and toolchains, the corresponding binaries are most likely to be broken anyway.

Available reference binary artifacts and deliverables

The first prototype will make the below binary artifacts publicly available:

• Root filesystem images (as specified above)
• Binary package feeds (as specified above)
• PR database, so that people can manage updates to packages, by importing the database in a local PR server.
• Prebuilt object data ("Shared State cache) through a public server (see SSTATE_MIRRORS), to reuse binary output already built by the Yocto Project autobuilders, to reduce the compile time of additional packages.
• Hash Equivalence data through a public server (see BB_HASHSERVE_UPSTREAM), to increase the reusability of prebuilt objects.

Share State and Hash Equivalence data would be only useful to "Intermediate" and "Advanced" use cases, to rebuild parts or the whole system from source.

Such artifacts will be served through a Content Delivery Network (CDN), to make sure that they can be downloaded fast enough from any location in the world, in a way that remains much faster than re-compiling such artifacts from sources.

Those binaries should also be released along with:

• Sources corresponding to the software used to build the image, as produced by the archiver class. Unlike standard GNU/Linux distributions, we won't ship source package feeds (such as SRPMs).
• SPDX output for the initial image
• The build system and layer sources. A good fit would be Alex Kanavin's build bundles.

Defining policies and processes

This reference binary distribution prototype will also drive the documentation of policies and process to address future work such as:

• What criteria should a new CPU architecture or platform tuning meet to be added / tested?
• What criteria should other layers meet to be tested, in addition to being already tested by the Autobuilder? Should these criteria only apply to specific recipes or to the entire layers containing them?

See the Policies and Processes.

Testing plans

Here are the tests we plan to implement:

• Producing and testing binary images through the autobuilder
• Making nightly builds available to testers
• Testing that package updates apply successfully:
  • Within the current stable branch (nanbield): from one release update to the next, and from the latest release update to the tip of the branch
  • Between release branches: from the latest update to our latest LTS (kirkstone), to the latest update to the current stable release (nanbield).
• Testing that it's also possible to build and deploy packages from the meta-openembedded and meta-virtualization layers, as a way to assess the relevance of our policies and process for additional layers or recipes. This would also be a way to demonstrate the ability to build container images, which would be a great way to ship binary distributions.

Here are other tests that would be worth implementing:

• Check that package upgrades don't cause configuration file changes to be lost. See this discussion.
• Check that the latest package upgrades successfully apply to any previous release, at least on the same branch. For example, upgrading directly from release x.y.n-2 to x.y may not work without going through x.y.n-1 as an intermediate. This could happen if a package has a really broken post-install script in n-2, which damage is repaired in n-1. Should n repair it too in case n-1 is skipped?

We will work with recipe maintainers to publish fixes for update issues that testing will find in recipes. Each issue will be reported in Bugzilla.

Open questions

• Issues with Yocto Project being pedantic in its use of arch strings such as "intel-x86-64" instead of "x86-64". This conflicts with typical use in the greater Linux and "binary" community, which may not recognize the "extended" arch strings. Examples: libresolv and libdnf (reported by Mark Asselstine).

PR server passthrough

The goal of this part is to enable a local distribution to use packages from an "upstream" distribution, but customize specific ones so that the locally built packages have priority over the upstream ones.

As for the Hash Equivalence server, we know have a patchset enabling a local PR server to contact an "upstream" server to compute the local PR value. Therefore, if an upstream package bears revision <x>, the local version of it can have revision <x>.<y>, having priority other the upstream one.

Policies and Processes

Open questions:

• When do we run updates? What does process look like?
• How long are the feeds maintained?
• How do we handle PR and PE changes wrt to the core layers?
• Do we add a default user? How do we handle image passwords? root password?

Criteria for adding and testing a new CPU architecture or platform tuning

• Only tunes in Openembedded-Core should be supported.

Process and policy for proposing adding new layers to the binary distro test artifacts process

• Layers must be Yocto Project Compatible
• In addition, layers must be well written and be targetting project best practices like reproducibility, no network access outside fetching so mirroring and licensing auditing works and not skipping QA tests without good reason.
• Any dependency layers would already have to be included in the testing already or make a separate application for addition first.
• The layer being added must be suitably generic with a clear need/usage of the components provided by the layer in the ecosystem
• The layer maintainer must have a commitment to keeping recipe upgrades functional
• The proposal must make it clear which recipes within the layer are being proposed to be built
• Proposals would be made to the Yocto Project TSC covering the above topics

Process and policy for proposing addition of a new MACHINE target

• Layers to build the machine need to be Yocto Project Compatible
• Layers need to be buildable on the project autobuilder
• Needs to be Yocto Project membership sponsorship of the machine due to the project resource usage
• Needs to be a commitment to testing the images for the platform for releases and upgrades
• Proposals would be made to the Yocto Project TSC covering the above topics

Process and policy for proposing addition of new recipe to the build

• This process assumes the layer is already present but the recipe is not currently included in the test matrix
• The recipe would need to have demonstrated the ability to pass on target installation and upgrade
• Proposals would be made to the Yocto Project TSC covering the above topics
• The software needs to have a good maintenance track record including reasonable CVE response
• Note: The TSC is aiming to appoint a maintainer to take the lead on resolving binary distro testing issues and handling recipe addition requests.

TSC Criteria for change evaluation

• Does the project have resources (human and compute for build/test/QA/bugfixing) to support the change?
• Is the change widely useful to a significant portion of the user base?
• Does the change enhance the project's test matrix?
• Does the change significantly improve developer productivity?

Achievements

Fixed bugs

• Bug 15292 - systemd-compat-units: error at package post-install time

Yocto Project Documentation improvements

• Fixes and updates to the Test Environment Manual

Project status

Here is the status of the various deliverables, according to the project specifications:

1. Define the scope of a binary distro prototype (which recipes and machines to target)
   Completed - See the top of this page: #Scope of a Yocto Binary Distribution Prototype
2. Add automated testing of upgrading an image from our previous release to the current version using a package feed.
   Ongoing - A prototype based on the "testimage" class has been submitted.
3. Implement PR server functionality to handle “pass through” mode and hierarchy data.
   Third [1] patch series submitted for the "passthrough" functionality, with support for multiple upstream servers. The patches to modernize the existing PR server code have already been merged. BitBake selftests have also been implemented for most of the code functionality (database, client, server, support for upstream server, read-only mode, "history" and "no history" modes). A fourth patch series is expected by Apr. 22 after real-life testing (generation of a full image, not only BitBake selftests).
4. Automated testing of the ability to assemble images quickly from sstate/PRServ/Hashequiv data.
   . Possible thanks to the use of the PR server in read-only mode.
5. Propose policies/requirements on how a binary reference distro for the project would behave. Need to include proposed policies for including new recipes and extending to cover new platforms/architectures.
   Proposal available on #Policies_and_Processes, thanks to Richard and the TSC.

Missing features

Not implemented yet:

• Ability to update the system SPDX description after installing extra packages or package updates. The current way SPDX is generated doesn't allow to generate -spdx packages, which would have made this possible (discussion).

Out of scope

Features outside of the scope of this effort, but definitely worth keeping track of for future efforts, or for people already maintaining binary distributions.

• Binary distribution installer: not absolutely necessary but users may expect one. For example the Yoe distro offers one, and so does Raspberry Pi.

FAQ

Q: Can my product point at your package feed? A: No. The aim of this work is demonstrate the tools, processes and policies needed to build a binary distribution using the project. There is no commitment or guarantee to maintain the feeds for any product use case or with any guarantees around security issues. Please do not point products at these package feeds.