Binary Distro Process

From Yocto Project
Revision as of 15:45, 18 December 2019 by Mark Hatle (talk | contribs)
Jump to navigationJump to search

Using the Yocto Project to define and build a binary distribution is possible, but to be useful and maintainable requires careful considerations for the configuration of the configuration, build process, and test procedures. This document will cover the general process, some of the recommended configurations and suggest where to insert your test procedures into the process.

High Level Architecture

A binary distribution begins with creating a process that allows you to do consistent, deterministic builds. This process becomes cyclical over the life of the product. The general procedure involves: Setup the build (new configuration or restoration of the prior build configuration and artifacts), performing the build, verifying the results of the build, storing the artifacts for the next iteration, and publishing the artifacts for the end user to use.

Binary Workflow.JPG Everything starts with an overall system configuration. The following items will need to be captured as part of the Configs.

Description of Process

Layer Setup & Configuration

  • Layer Setup
    • URLs
    • Commit IDs
  • Configuration
    • bblayers.conf
    • local.conf
  USER_CLASSES += "reproducible_build"
  USER_CLASSES += "packagefeed-stability"
  USER_CLASSES += "buildhistory"
  DEFAULTTUNE = "<common tune>"
  DISTRO = "..."
  PACKAGE_CLASSES = "package_rpm"
  EXTRA_IMAGE_FEATURES – REMOVE debug-tweaks!
  BB_HASHSERVER = "auto"
  BB_SIGNATURE_HANDLER = "OEEquivHash"
  PRSERV_HOST = "localhost:0"
  INHERIT += "archiver"
  ARCHIVER_MODE[src] = "original"
  ARCHIVER_MODE[recipe] = "1"
  ARCHIVER_MODE[dumpdata] = "1"
  ARCHIVER_MODE[srpm] = "1"

Build Project

In order to get a consistent set of updated packages over the life of binary distribution, it is necessary to define which recipe targets will be built. These should be defined and captured as part of the overall build configuration. Each defined recipe target should be carefully considered, as it will be very difficult to remove a recipe target over the life of the binary distribution.

The targets may include individual package recipes, packagegroup recipes, image recipes or even world builds. It is recommended that packagegroups, and images are the usual defined targets. If you are expecting to use world builds, it may be necessary to add specific recipe blacklists to remove packages that should not be part of the binary distribution.

Removal of targets will result in binary packages that are stagnant, as there is no reasonable way to remove packages, as a distribution, once it has been made available to an user system. There are ways to deprecate and replace installed software during an upgrade but these should only be used if specifically needed and will require extensive usage testing to ensure that they do not create any runtime / software upgrade issues. These tests will need to be repeated for all future releases as well, possibly complicating testing.

The results of the build will be a series of artifacts.

Artifacts

The following artifacts will be generated and stored as part of the build:

  • Configuration
    • Layer URLs/Commit IDs (or copies of)
    • project configuration files (local.conf/bblayers.conf)
    • List of target recipes for the build
      • This may be captured in various logs
  • Log(s)
    • Console Logs
    • Individual Build Logs
    • buildhistory
  • Sources
    • Downloads
    • ARCHIVER output
  • Sstate
    • prserver database
    • sstate-cache
    • hash equivalency database
  • Package Repository
    • Repository Configuration
    • RPM Packages
  • images
  • eSDK / SDK

Test

A test pass is then executed. The first step should be an inspection of the logs (including build history) to verify the output is as expected. Next a series of tests will need to verify that the image(s) boot, RPM packages can be installed, upgrades from the first and last releases work properly, etc. Over time this test suite will need to be expanded to deal with issues found along the way.

Fix

In the case of a test failure, a fix for the failure will need to be created and committed. Failed artifacts will be discarded (not released) as part of this.

Release

In the case of a test pass, assuming it has been decided to release, the artifacts are published. Note, prior artifacts should never be removed from the publishing location. Some artifacts, like build history may be determined to be internal only, but generally everything can be released.

Restore Last Released Artifacts

Once a new build is determined to be needed, layers are updated, last released artifacts are restored and a new build is performed repeating the process.

EOL

Product End-Of-Life is declared and no further updates are released.

Requirements

The following requirements were identified as being needed for a binary distribution. Note, a specific distribution may not require all of these items, but they should all be considered as part of this development.

Stage 1 requirements:

  • Prebuilt/bootable image for each board
  • A repeatable process to generate a binary package feeds
    • Process documentation
      • Each step of the process, as implemented, should be documented to ensure it can be reproduced in the future
      • Initial configuration, target recipes, and specific commands should be included in this documentation
    • Process shall be worked on in conjunction with the community
    • Process shall allow updated packages to be built and released in addition to prior builds
  • A mechanism to capture the source to binary build information so a user can transition from binary distribution to source distribution
    • local.conf
    • prserver
      • Must be configurable to NOT clash with the prserver values for the official feed
    • sstate-cache
      • hash equivalency
    • documentation
      • how to transition to custom builds
        • eSDK, full bitbake
      • how to puyblish custom components (feeds)
  • Should use Yocto Project helper classes
    • reproducible_build
    • packagefeed-stability (may not be required with hash equivalency)
  • Must be secure!
    • The default image must not have predefined passwords!
      • Passwords, ssh keys, etc should not be predefined
      • A 'first boot' mechanism to generate device specific information is needed
        • Host keys
        • root and/or user passwords
    • All services (within reason) should be disabled by default
      • initial attack surface should be as small as reasonable
  • Package feeds must be generic (reusable), and only 'specific' when necessary (machine)
    • Generic architecture feeds
      • x86_64
      • armv7 hard-float
      • armv8 -- aarch64
      • etc
    • SOC_FAMILY specific feeds (as necessary)
      • device drivers and other soc family items for groups of boards
    • MACHINE (board) specific feeds
      • machine specific packages
    • Supports one or more custom feeds (as configured and enabled by the device admin)
  • Package Feeds must be cumulative, permitting a user to revert to a specific version
  • Downloadable image or other mechanism to quickly boot, without needing an SDK
    • Must be preconfigured to use the package feeds (where applicable)
      • System upgrade should be able to be an automated process (cronjob)
      • Manual updates must be supported
  • PSIRT / Security Response process
    • Define and document product lifespan (EOL)
    • CVE Monitoring
      • Product a report of public known issues, issues under investigation, and issues that have been fixed
      • SLA (Service Level Agreements) defined for security fix/release to binary feed
      • Method for outside people to notify us of security issues (may not be CVE)
      • On-going kernel upgrades to deal with kernel defects
        • May be Yocto Project, LTS or 'master' kernels
  • Test strategy for validating binary packages
    • Buildhistory used to allow architects/leads to verify components before release
    • API / ABI comparison tooling to verify system is unlikely to break due to a change
    • Package upgrade testing
      • First release to current version(s)
      • Last release to current version(s)
      • Random intermediate release to current version(s)
      • Should execute ptest and oe qa tests and/or other automated tests after upgrading
        • Goal is as good or better then prior test pass

Stage 2 requirements:

  • Cryptographic signed sstate-cache
    • Sstate-cache objects are signed and verified prior to usage
  • Cryptographic Signed packages and package feeds
    • Package manager must validate signatures before installing
    • User must be able to add their own signature to extend the package feeds
  • Increase available packages to new components

Possible future work, based on user feedback:

  • Desktop environment
    • XFCE
    • KDE
    • Gnome
  • Ability to migrate an image from one board to another
    • Reconfigure feeds
    • Load new boot image
    • Swap to a new board
  • Web interfaces to install packages and get a device package manifest
    • May also involve remote device management
  • Package feed viewer
    • Something like rpmfind to view and inspect package info
  • Increase available packages to new components
Retrieved from "https://wiki.yoctoproject.org/wiki/index.php?title=Binary_Distro_Process&oldid=65881"