Binary Distro Process

From Yocto Project
Revision as of 18:43, 13 December 2019 by Mark Hatle (talk | contribs)
Jump to navigationJump to search

High Level Architecture

A binary distribution begins with creating a process that allows you to do consistent, deterministic builds. This process becomes cyclical over the life of the product. The general procedure involves: Setup the build (new configuration or restoration of the prior build configuration and artifacts), performing the build, verifying the results of the build, storing the artifacts for the next iteration, and publishing the artifacts for the end user to use.

Binary Workflow.JPG Everything starts with an overall system configuration. The following items will need to be captured as part of the Configs.

Description of Process

Layer Setup & Configuration

  • Layer Setup
    • URLs
    • Commit IDs
  • Configuration
    • bblayers.conf
    • local.conf
  USER_CLASSES += "reproducible_build"
  USER_CLASSES += "packagefeed-stability"
  USER_CLASSES += "buildhistory"
  DEFAULTTUNE = "<common tune>"
  DISTRO = "..."
  PACKAGE_CLASSES = "package_rpm"
  EXTRA_IMAGE_FEATURES – REMOVE debug-tweaks!
  BB_HASHSERVER = "auto"
  BB_SIGNATURE_HANDLER = "OEEquivHash"
  PRSERV_HOST = "localhost:0"
  INHERIT += "archiver"
  ARCHIVER_MODE[src] = "original"
  ARCHIVER_MODE[recipe] = "1"
  ARCHIVER_MODE[dumpdata] = "1"
  ARCHIVER_MODE[srpm] = "1"

Build Project

The project is built using a standard process/target. Depending on the purposes of the binary distribution, it may require one or more recipe targets or even a world build.

The results of the build will be a series of artifacts.

Artifacts

The following artifacts will be generated and stored as part of the build:

  • Configuration
    • Layer URLs/Commit IDs (or copies of)
    • project configuration files (local.conf/bblayers.conf)
  • Log(s)
    • Console Logs
    • Individual Build Logs
    • buildhistory
  • Sources
    • Downloads
    • ARCHIVER output
  • Sstate
    • prserver database
    • sstate-cache
    • hash equivalency database
  • Package Repository
    • Repository Configuration
    • RPM Packages
  • images
  • eSDK / SDK

Test

A test pass is then executed. The first step should be an inspection of the logs (including build history) to verify the output is as expected. Next a series of tests will need to verify that the image(s) boot, RPM packages can be installed, upgrades from the first and last releases work properly, etc. Over time this test suite will need to be expanded to deal with issues found along the way.

Fix

In the case of a test failure, a fix for the failure will need to be created and committed. Failed artifacts will be discarded (not released) as part of this.

Release

In the case of a test pass, assuming it has been decided to release, the artifacts are published. Note, prior artifacts should never be removed from the publishing location. Some artifacts, like build history may be determined to be internal only, but generally everything can be released.

Restore Last Released Artifacts

Once a new build is determined to be needed, layers are updated, last released artifacts are restored and a new build is performed repeating the process.

EOL

Product End-Of-Life is declared and no further updates are released.

Requirements

The following requirements were identified as being needed for a binary distribution. Note, a specific distribution may not require all of these items, but they should all be considered as part of this development.

Stage 1 requirements:

  • Prebuilt/bootable image for each board
  • A repeatable process to generate a binary package feeds
    • Process shall be worked on in conjunction with the community
    • Process shall allow updated packages to be built and released in addition to prior builds
  • A mechanism to capture the source to binary build information so a user can transition from binary distribution to source distribution
    • local.conf
    • prserver
      • Must be configurable to NOT clash with the prserver values for the official feed
    • sstate-cache
      • hash equivalency
    • documentation
      • how to transition to custom builds
        • eSDK, full bitbake
      • how to puyblish custom components (feeds)
  • Should use Yocto Project helper classes
    • reproducible_build
    • packagefeed-stability (may not be required with hash equivalency)
  • Must be secure!
    • The default image must not have predefined passwords!
      • Passwords, ssh keys, etc should not be predefined
      • A 'first boot' mechanism to generate device specific information is needed
        • Host keys
        • root and/or user passwords
    • All services (within reason) should be disabled by default
      • initial attack surface should be as small as reasonable
  • Package feeds must be generic (reusable), and only 'specific' when necessary (machine)
    • Generic architecture feeds
      • x86_64
      • armv7 hard-float
      • armv8 -- aarch64
      • etc
    • SOC_FAMILY specific feeds (as necessary)
      • device drivers and other soc family items for groups of boards
    • MACHINE (board) specific feeds
      • machine specific packages
    • Supports one or more custom feeds (as configured and enabled by the device admin)
  • Package Feeds must be cumulative, permitting a user to revert to a specific version
  • Downloadable image or other mechanism to quickly boot, without needing an SDK
    • Must be preconfigured to use the package feeds (where applicable)
      • System upgrade should be able to be an automated process (cronjob)
      • Manual updates must be supported
  • PSIRT / Security Response process
    • Define and document product lifespan (EOL)
    • CVE Monitoring
      • Product a report of public known issues, issues under investigation, and issues that have been fixed
      • SLA (Service Level Agreements) defined for security fix/release to binary feed
      • Method for outside people to notify us of security issues (may not be CVE)
      • On-going kernel upgrades to deal with kernel defects
        • May be Yocto Project, LTS or 'master' kernels
  • Test strategy for validating binary packages
    • Buildhistory used to allow architects/leads to verify components before release
    • API / ABI comparison tooling to verify system is unlikely to break due to a change
    • Package upgrade testing
      • First release to current version(s)
      • Last release to current version(s)
      • Random intermediate release to current version(s)
      • Should execute ptest and oe qa tests and/or other automated tests after upgrading
        • Goal is as good or better then prior test pass

Stage 2 requirements:

  • Cryptographic signed sstate-cache
    • Sstate-cache objects are signed and verified prior to usage
  • Cryptographic Signed packages and package feeds
    • Package manager must validate signatures before installing
    • User must be able to add their own signature to extend the package feeds
  • Increase available packages to new components

Possible future work, based on user feedback:

  • Desktop environment
    • XFCE
    • KDE
    • Gnome
  • Ability to migrate an image from one board to another
    • Reconfigure feeds
    • Load new boot image
    • Swap to a new board
  • Web interfaces to install packages and get a device package manifest
    • May also involve remote device management
  • Package feed viewer
    • Something like rpmfind to view and inspect package info
  • Increase available packages to new components
Retrieved from "https://wiki.yoctoproject.org/wiki/index.php?title=Binary_Distro_Process&oldid=65538"