Binary Distro Process - Revision history

Mark Hatle at 16:57, 20 December 2019

2019-12-20T16:57:50Z

← Older revision		Revision as of 16:57, 20 December 2019
Line 1:		Line 1:
	Using the Yocto Project to define and build a binary distribution is possible, but to be useful and maintainable requires careful considerations for the configuration of the configuration, build process, and test procedures. This document will cover the general process, some of the recommended configurations and suggest where to insert your test procedures into the process.		Using the Yocto Project to define and build a binary distribution is possible, but to be useful and maintainable requires careful considerations for the configuration of the configuration, build process, and test procedures. This document will cover the general process, some of the recommended configurations and suggest where to insert your test procedures into the process.

			This page covers the high level architecture, along with various requirements that need to be considered to create a successful binary distribution. The actual implementation of this architecture and requirements is left to future documents.

	'''High Level Architecture'''		'''High Level Architecture'''
Line 113:		Line 115:
	*** how to transition to custom builds		*** how to transition to custom builds
	**** eSDK, full bitbake		**** eSDK, full bitbake
	*** how to ~~puyblish~~ custom components (feeds)		*** how to publish custom components (feeds)
	* Should use Yocto Project helper classes		* Should use Yocto Project helper classes
	** reproducible_build		** reproducible_build

Mark Hatle at 20:33, 19 December 2019

2019-12-19T20:33:57Z

← Older revision		Revision as of 20:33, 19 December 2019
Line 6:		Line 6:

	[[File:Binary_Workflow.JPG]]		[[File:Binary_Workflow.JPG]]

	Everything starts with an overall system configuration. The following items will need to be captured as part of the Configs.		Everything starts with an overall system configuration. The following items will need to be captured as part of the Configs.

Mark Hatle at 15:45, 18 December 2019

2019-12-18T15:45:50Z

← Older revision		Revision as of 15:45, 18 December 2019
Line 36:		Line 36:
	''Build Project''		''Build Project''

	~~The project~~ is built ~~using~~ a ~~standard process/~~target~~. Depending on~~ the ~~purposes~~ of the binary distribution, it may ~~require one or more~~ recipe targets ~~or even~~ a ~~world build~~.		In order to get a consistent set of updated packages over the life of binary distribution, it is necessary to define which recipe targets will be built. These should be defined and captured as part of the overall build configuration. Each defined recipe target should be carefully considered, as it will be very difficult to remove a recipe target over the life of the binary distribution.

			The targets may include individual package recipes, packagegroup recipes, image recipes or even world builds. It is recommended that packagegroups, and images are the usual defined targets. If you are expecting to use world builds, it may be necessary to add specific recipe blacklists to remove packages that should not be part of the binary distribution.

			Removal of targets will result in binary packages that are stagnant, as there is no reasonable way to remove packages, as a distribution, once it has been made available to an user system. There are ways to deprecate and replace installed software during an upgrade but these should only be used if specifically needed and will require extensive usage testing to ensure that they do not create any runtime / software upgrade issues. These tests will need to be repeated for all future releases as well, possibly complicating testing.

	The results of the build will be a series of artifacts.		The results of the build will be a series of artifacts.
Line 47:		Line 51:
	** Layer URLs/Commit IDs (or copies of)		** Layer URLs/Commit IDs (or copies of)
	** project configuration files (local.conf/bblayers.conf)		** project configuration files (local.conf/bblayers.conf)
			** List of target recipes for the build
			*** This may be captured in various logs
	* Log(s)		* Log(s)
	** Console Logs		** Console Logs
Line 92:		Line 98:
	* Prebuilt/bootable image for each board		* Prebuilt/bootable image for each board
	* A repeatable process to generate a binary package feeds		* A repeatable process to generate a binary package feeds
			** Process documentation
			*** Each step of the process, as implemented, should be documented to ensure it can be reproduced in the future
			*** Initial configuration, target recipes, and specific commands should be included in this documentation
	** Process shall be worked on in conjunction with the community		** Process shall be worked on in conjunction with the community
	** Process shall allow updated packages to be built and released in addition to prior builds		** Process shall allow updated packages to be built and released in addition to prior builds

Mark Hatle at 15:30, 18 December 2019

2019-12-18T15:30:11Z

← Older revision		Revision as of 15:30, 18 December 2019
Line 1:		Line 1:
			Using the Yocto Project to define and build a binary distribution is possible, but to be useful and maintainable requires careful considerations for the configuration of the configuration, build process, and test procedures. This document will cover the general process, some of the recommended configurations and suggest where to insert your test procedures into the process.

	'''High Level Architecture'''		'''High Level Architecture'''

Mark Hatle at 18:43, 13 December 2019

2019-12-13T18:43:48Z

@@ Line 3: / Line 3: @@
 A binary distribution begins with creating a process that allows you to do consistent, deterministic builds.  This process becomes cyclical over the life of the product.  The general procedure involves: Setup the build (new configuration or restoration of the prior build configuration and artifacts), performing the build, verifying the results of the build, storing the artifacts for the next iteration, and publishing the artifacts for the end user to use.
-PICTURE HERE
+[[File:Binary_Workflow.JPG]]
 Everything starts with an overall system configuration.  The following items will need to be captured as part of the Configs.
@@ Line 82: / Line 81: @@
 Product End-Of-Life is declared and no further updates are released.

Mark Hatle at 18:23, 13 December 2019

2019-12-13T18:23:47Z

← Older revision		Revision as of 18:23, 13 December 2019
Line 6:		Line 6:

	Everything starts with an overall system configuration. The following items will need to be captured as part of the Configs.		Everything starts with an overall system configuration. The following items will need to be captured as part of the Configs.

			'''Description of Process'''

	''Layer Setup & Configuration''		''Layer Setup & Configuration''

Mark Hatle: Binary Distribution Process

2019-12-13T18:21:04Z

Binary Distribution Process

New page

'''High Level Architecture'''

A binary distribution begins with creating a process that allows you to do consistent, deterministic builds. This process becomes cyclical over the life of the product. The general procedure involves: Setup the build (new configuration or restoration of the prior build configuration and artifacts), performing the build, verifying the results of the build, storing the artifacts for the next iteration, and publishing the artifacts for the end user to use.

PICTURE HERE

Everything starts with an overall system configuration. The following items will need to be captured as part of the Configs.

''Layer Setup & Configuration''

* Layer Setup
** URLs
** Commit IDs
* Configuration
** bblayers.conf
** local.conf
USER_CLASSES += "reproducible_build"
USER_CLASSES += "packagefeed-stability"
USER_CLASSES += "buildhistory"
DEFAULTTUNE = "<common tune>"
DISTRO = "..."
PACKAGE_CLASSES = "package_rpm"
EXTRA_IMAGE_FEATURES – REMOVE debug-tweaks!
BB_HASHSERVER = "auto"
BB_SIGNATURE_HANDLER = "OEEquivHash"
PRSERV_HOST = "localhost:0"
INHERIT += "archiver"
ARCHIVER_MODE[src] = "original"
ARCHIVER_MODE[recipe] = "1"
ARCHIVER_MODE[dumpdata] = "1"
ARCHIVER_MODE[srpm] = "1"

''Build Project''

The project is built using a standard process/target. Depending on the purposes of the binary distribution, it may require one or more recipe targets or even a world build.

The results of the build will be a series of artifacts.

''Artifacts''

The following artifacts will be generated and stored as part of the build:

* Configuration
** Layer URLs/Commit IDs (or copies of)
** project configuration files (local.conf/bblayers.conf)
* Log(s)
** Console Logs
** Individual Build Logs
** buildhistory
* Sources
** Downloads
** ARCHIVER output
* Sstate
** prserver database
** sstate-cache
** hash equivalency database
* Package Repository
** Repository Configuration
** RPM Packages
* images
* eSDK / SDK

''Test''

A test pass is then executed. The first step should be an inspection of the logs (including build history) to verify the output is as expected. Next a series of tests will need to verify that the image(s) boot, RPM packages can be installed, upgrades from the first and last releases work properly, etc. Over time this test suite will need to be expanded to deal with issues found along the way.

''Fix''

In the case of a test failure, a fix for the failure will need to be created and committed. Failed artifacts will be discarded (not released) as part of this.

''Release''

In the case of a test pass, assuming it has been decided to release, the artifacts are published. Note, prior artifacts should never be removed from the publishing location. Some artifacts, like build history may be determined to be internal only, but generally everything can be released.

''Restore Last Released Artifacts''

Once a new build is determined to be needed, layers are updated, last released artifacts are restored and a new build is performed repeating the process.

''EOL''

Product End-Of-Life is declared and no further updates are released.