Meta-swupd - Revision history

Joshua Lock: /* Using swupd client */ add a brief section about using swupd-client

2016-04-19T14:15:00Z

Using swupd client: add a brief section about using swupd-client

← Older revision		Revision as of 14:15, 19 April 2016
Line 146:		Line 146:
	== Using swupd client ==		== Using swupd client ==

	''~~TODO~~''		The swupd-client package installs a binary named ''swupd'' which can be used to perform updates, etc on the target.

			Take extra care with swupd-client versions prior to v3.4.0 as they hard-code the update URLs to clearlinux.org. When steps haven't been taken to change the default update URLs be sure to operate swupd-client with the '''-u''' switch and pass a URL to your update server.

			Examples of swupd-client use follow:

			<pre>swupd check-update -u example.com/updates
			swupd update -u example.com/updates
			swupd verify -u example.com/updates</pre>

Joshua Lock: /* Quick steps */ flesh out step 7 on building extra images with SWUPD_IMAGES

2016-04-19T14:14:23Z

Quick steps: flesh out step 7 on building extra images with SWUPD_IMAGES

← Older revision		Revision as of 14:14, 19 April 2016
Line 128:		Line 128:
	<pre>SWUPD_BUNDLES = "feature_one feature_two"		<pre>SWUPD_BUNDLES = "feature_one feature_two"
	BUNDLE_CONTENTS[feature_one] = "package_one package_three package_six"</pre>		BUNDLE_CONTENTS[feature_one] = "package_one package_three package_six"</pre>
	'''NOTE''': beware of reserved bundle names, ~~the names~~ '''full''' and '''mega''' ~~are reserved~~ and cannot be used for bundle names.</li>		'''NOTE''': beware of reserved bundle names, both '''full''' and '''mega''' have special meaning and cannot be used for bundle names.</li>
	<li><p>~~''TODO'' mention~~ optional images ~~generated from~~ os-core + additional bundles ~~via~~ <code>SWUPD_IMAGES</code></p></li></ol>		<li><p>(optional) Define extra images, consisting of the os-core with any number of additional bundles installed, which can be built. Do this by setting the <code>SWUPD_IMAGES</code> variable to a list of additional image name suffixes and assigning the names of bundles to inclde to a varflag matching the defined name. For example:</p>
			<pre>SWUPD_IMAGES = "product1"
			SWUPD_IMAGES[product1] = "product1"</pre>
			<p>defined in an image named ''myco-image-core'' would enable bitbake to be invoked with the target ''myco-image-core-product1''. The ''myco-image-core-product1'' image would consist of the base OS (''os-core'' bundle) and the contents of the ''product1'' bundle and might, for example, be used to provide an image which can be directly flashed to the product hardware.</p></li></ol>

	== Control variables ==		== Control variables ==

Joshua Lock: /* Stateful */ update to reflect recent changes which allow updating of files in /etc

2016-04-19T14:13:32Z

Stateful: update to reflect recent changes which allow updating of files in /etc

← Older revision		Revision as of 14:13, 19 April 2016
Line 93:		Line 93:
	Linux-based OS built with OpenEmbedded and the Yocto Project Tools are stateful OS, that is the default configuration is provided by files in <code>/etc</code> and without extra work on top of the default packages the OS won't boot without a populated <code>/etc</code>. This contradicts the swupd way and means that currently any files a package installs into <code>/etc</code> won't be updated between OS version.		Linux-based OS built with OpenEmbedded and the Yocto Project Tools are stateful OS, that is the default configuration is provided by files in <code>/etc</code> and without extra work on top of the default packages the OS won't boot without a populated <code>/etc</code>. This contradicts the swupd way and means that currently any files a package installs into <code>/etc</code> won't be updated between OS version.

	~~Unfortunately this means~~ the ~~solution~~ in ~~meta~~-~~swupd does not support adding users or groups in future~~ updates. ~~It's possible to solve~~ this on a ~~per-OS basis by implementing custom swupd helper scripts~~ (~~as described below), we are investigating methods to solve this within meta-swupd by patching~~ swupd-client ~~(see [https://bugzilla.yoctoproject.org~~/~~show_bug.cgi?id=9199 Yocto #9199]) as well as investigating providing easier support for developing stateless OS with OpenEmbedded~~/~~Yocto Project tools~~.		swupd-client checks the <code>VERSION_ID</code> in the the <code>os-release</code> file when checking for updates. As this file is usually installed in <code>/etc</code> we include a <code>bbappend</code> which instead creates this file in <code>/usr/lib/</code> (a location swupd-client prefers) and symlink it to <code>/etc</code>.

	swupd-client ~~checks the~~ <code>~~VERSION_ID~~</code> in ~~the the~~ <code>~~os-release~~</code> ~~file when checking for updates~~. ~~As this file~~ is ~~usually installed~~ in <code>/etc</code> ~~we include a~~ <code>~~bbappend~~</code> which ~~instead creates this file~~ in <code>/~~usr/lib~~/</code> ~~(a location~~ swupd-client ~~prefers) and symlink it to~~ <code>/~~etc~~</code>.		In order to support deploying updates to files in <code>/etc</code> via swupd we have developed patches for both swupd-client and swupd-server to include files in <code>/etc</code> in updates. These changes are enabled in swupd-server (resulting in swupd manifests which list files in <code>/etc</code>) but disabled by default in swupd-client.

			This is because enabling updating of files in <code>/etc</code> requires some forethought by the OS developer to ensure that files in <code>/etc</code> which shouldn't be updated are not included in the bundle directories. For example if you would like users of your OS to be able to add users and groups without OS updates overwriting those users and groups you would need to ensusre that the passwd and group files are not included in the bundle directories during the <code>do_swupd_update</code> task.

			You might achieve this by having a <code>prefunc</code> for the <code>do_swupd_update</code> task which removes said files from the ''os-core'' bundle directory.

			To enable stateful OS support in swupd-client disable the <code>stateless</code> <code>PACKAGE_CONFIG</code> option for the swupd-client recipe, i.e. with a <code>swupd-client_%.bbappend</code> file:

	~~'''NOTE''': once a version of swupd-client which supports updating configuration files in~~ <~~code~~>~~/etc~~</~~code~~> has been developed it will be possible to deploy support for additional users and groups to an existing swupd-based OS in a 2-step fashion: first deploy an update which includes the new version of swupd-client, after which any future updates will be able to include modified configuration files.		<pre>PACKAGE_CONFIG_remove = "stateless"</pre>

	=== swupd helper scripts ===		=== swupd helper scripts ===

Joshua Lock: Add note on systemd requirement

2016-04-14T13:58:53Z

Add note on systemd requirement

← Older revision		Revision as of 13:58, 14 April 2016
Line 50:		Line 50:

	When building a distribution with swupd the OS developer should be aware of the following requirements:		When building a distribution with swupd the OS developer should be aware of the following requirements:

			=== systemd ===

			swupd assumes the use of systemd both by using systemd units to trigger update checks and by explicitly calling the systemd binary to restart the <code>update-triggers.target</code> after an OS update has completed (see the swupd helper scripts section for more information about this target).

			Therefore at this time one must use systemd as init manager in order to make use of swupd.

	=== Version number ===		=== Version number ===

Joshua Lock: Add link to the layer

2016-04-14T08:29:52Z

Add link to the layer

← Older revision		Revision as of 08:29, 14 April 2016
Line 1:		Line 1:
			This page provides preliminary documentation for using the [http://git.yoctoproject.org/cgit/cgit.cgi/meta-swupd meta-swupd] layer.

	= Building a swupd-based OS =		= Building a swupd-based OS =

Joshua Lock: /* useradd */

2016-04-13T21:43:14Z

useradd

← Older revision		Revision as of 21:43, 13 April 2016
Line 71:		Line 71:
	swupd_check_dangling_symlinks \		swupd_check_dangling_symlinks \
	"</pre>		"</pre>
	=== ~~useradd~~ ===		=== Users and groups ===

	Another aspect of the immutable os-core bundle is that the account information in the passwd and group files installed in the os-core bundle must include user and group information required for all bundles provided by the OS.		Another aspect of the immutable os-core bundle is that the account information in the passwd and group files installed in the os-core bundle must include user and group information required for all bundles provided by the OS.

Joshua Lock: /* Control variables */

2016-04-13T21:29:57Z

Control variables

← Older revision		Revision as of 21:29, 13 April 2016
Line 125:		Line 125:
	* <code>SWUPD_N_DELTAPACKS</code> — the number of previous releases against which to generate delta-packs, defaults to 2.		* <code>SWUPD_N_DELTAPACKS</code> — the number of previous releases against which to generate delta-packs, defaults to 2.
	* <code>SWUPD_VERSION_STEP</code> — Amount the OS_VERSION should be increased by for each release. Used by the delta pack looping to generate delta packs going back up to SWUPD_N_DELTAPACK releases.		* <code>SWUPD_VERSION_STEP</code> — Amount the OS_VERSION should be increased by for each release. Used by the delta pack looping to generate delta packs going back up to SWUPD_N_DELTAPACK releases.
	* <code>SWUPD_IMAGES</code> ''TODO''		* <code>SWUPD_IMAGES</code> — ''TODO''

	== Using swupd client ==		== Using swupd client ==

	''TODO''		''TODO''

Joshua Lock: Improve the swupd howto guide

2016-04-13T21:29:25Z

Improve the swupd howto guide

Show changes

Joshua Lock: Initial, WIP, docs for meta-swupd

2016-03-22T22:27:48Z

Initial, WIP, docs for meta-swupd

New page

= What is swupd? =

The [https://clearlinux.org/features/software-update SoftWare UPDater] — swupd — from [https://clearlinux.org/ Clear Linux] provides a new way of adding functionality to and updating a Linux-based OS.

swupd uses binary-delta technology to efficiently update only the files that have changed between OS updates. This means that updates are small, resulting in fast downloads, and fast to apply.

Bundles allow a system adminstrator to easily add or remove a complete set of functionality to the OS image, rather than worrying about individual packages.

= What is this meta-swupd? =

[http://git.yoctoproject.org/cgit/cgit.cgi/meta-swupd meta-swupd] provides a metadata layer to aide Open Embedded and Yocto Project users in developing Linux-based operating systems built around swupd.

meta-swupd can be used to implement both of the key features of swupd; efficient binary-delta based updates and bundle-based feature management.

meta-swupd can easily be used to only deploy efficient update streams ''or'' to deploy a bundle-based OS with efficient updates.

= Swupd Concepts =

== Single, OS-wide, version number ==

swupd based operating systems atomically upgrade the entire OS from one version to the next, rather than upgrading at a more granular, package-based, level. This means that the entire contents of the OS can be determined from a single version number.

= How do I use it? =

The following steps are required to develop an OS that uses a swupd update stream: * add the meta-swupd layer to bblayers.conf * designate an image recipe as the base OS image (os-core, in swupd parlance) and <code>inherit swupd-image</code> in that recipe * ensure the <code>OS_VERSION</code> variable is assigned an integer value which matches the <code>VERSION_ID</code> value in the ''os-release'' recipe and that this number is increased before each build which should generate swupd update artefacts

In addition, to make use of swupd’s bundle functionality, you will need to: * assign a list of bundle names to <code>SWUPD_BUNDLES</code> i.e:

<pre>```SWUPD_BUNDLES = "feature_one feature_two"```</pre>
<ul>
<li><p>for each named bundle, assign a list of packages for which their content should be included in the bundle to a varflag of <code>BUNDLE_CONTENTS</code> which matches the bundle name i.e:</p>
<p><code>BUNDLE_CONTENTS[feature_one] = "package_one package_three package_six"</code></p></li></ul>

= What might go wrong? (a.k.a. known issues) =

* refuses to overwrite generated data — because the act of generating swupd updates is time-consuming meta-swupd, by design, refuses to overwrite an existing update stream warning that the <code>OS_VERSION</code> number should be incremented.
* hardlinks and extended attributes — there’s a bug in [https://www.yoctoproject.org/tools-resources/projects/pseudo pseudo] which results in the extended attributes for a hardlink being incorrectly associated with the directory entry, rather than the inode. See [https://bugzilla.yoctoproject.org/show_bug.cgi?id=9317 bug #9317]
* alternatives — when a bundle adds a package which pulls in an alternative with a higher priority than the one in the base image (i.e. coreutils takes precedence over busybox) we end up with a situation where the base image contains the symlink to the higher priority binary, but not the binaries themselves. See [https://bugzilla.yoctoproject.org/show_bug.cgi?id=9320 bug #9320]
* potentially copies a lot of duplicate files — when copying bundle contents around it’s very likely that the same file will exist in the swupd working directory multiple times. We may be able to make this less disk space intensive by using a utility like hardlink to deduplicate identical files. See [https://bugzilla.yoctoproject.org/show_bug.cgi?id=9189 bug #9189]
* copies then deletes, rather than just copying — the current/initial implementation copies the entire mega-image rootfs for each bundle chroot-like directory and then prunes away unwanted files. It will be much more efficient to only copy the files which are required in the first instance. See [https://bugzilla.yoctoproject.org/show_bug.cgi?id=9325 bug #9325]

= Control Variables =

Several variables can be set to tune the way swupd-image works: * <code>SWUPD_GENERATE</code> — if set to ''0'' i.e. <code>SWUPD_GENERATE = "0"</code> swupd update artefact processing will be skipped but all tasks of the <code>swupd-image</code> class will be executed. This is useful both for debugging the <code>swupd-image</code> class and in a scenario where it might be desirable to generate the chroot-like bundle directories without performing an processing with swupd. * <code>SWUPD_DELTAPACKS</code> — if set to ''0'' i.e. <code>SWUPD_DELTAPACKS="0"</code> swupd delta-packs will not be generated. * <code>SWUPD_N_DELTAPACKS</code> — the number of previous releases against which to generate delta-packs, defaults to 2. * <code>SWUPD_VERSION_STEP</code> — Amount the OS_VERSION should be increased by for each release. Used by the delta pack looping to generate delta packs going back up to SWUPD_N_DELTAPACK releases. This could be replaced by a more elegant algorithm, see [https://bugzilla.yoctoproject.org/show_bug.cgi?id=9322 bug #9322].

= Implementation Overview =

An image that inherits this class will automatically have bundle ‘chroots’ created which contain the filesystem contents of the specified bundles. The mechanism to achieve this is that several virtual image recipes are created, one for each defined bundle plus a ‘mega’ image recipe. The ‘mega’ image contains the base image plus all of the bundles (and their contents), whilst bundle images contain only the base image plus the contents of a single bundle.

We build the mega image first, then the base image (the one which inherits the <code>swupd-image</code> class) and finally all of the bundle images . Each non-mega image has a manifest generated that lists all of the file contents of the image.

Each bundle ‘chroot’-like directory and the rootfs of the base image are all populated from the contents of the mega image’s rootfs. The reason for this is to ensure that all files which are modified during some kind of post-processing step, i.e. passwd and groups updated during postinsts, are fully populated. This may not be an ideal compromise and requires further thought (i.e. see the known issue about alternatives above).

Once the images and their manifests have been created each bundle image manifest is compared to the base image manifest in order to generate a delta list of files in the bundle image which don’t exist in the base image. Files in this list are then preserved in the bundle directory for processing by swupd-server in order to generate update artefacts.