Yocto Project - User contributions [en]

System Update

2025-07-29T12:04:54Z

Thomas Perrot: Update RAUC features

== Introduction ==

This page compares different system update mechanisms. The purpose is to help the project with picking a suitable mechanism that the project then will support going forward. Users may find this page relevant for picking a mechanism that suits their specific needs.

A system update mechanism must ensure that a device running an older release of the operating systems runs with a more recent release when the update mechanism is done. This includes updating everything that defines the system (rootfs, kernel, bootloader, etc.), restarting running processes and potentially a reboot. An ideal mechanism:
* never ends up in an inconsistent state (atomic update),
* always keeps the device usable (fallback to previous state when there are problems, or at least supporting a recovery mode),
* requires little additional resources (disk space, RAM),
* minimizes downtime while updating,
* works in combination with security technology (integrity protection),
* is secure (does not install or execute software created by an attacker).

These are conflicting requirements. Different mechanisms will have different strengths and weaknesses. Therefore the first chapter provides a more detailed definition of the different aspects and has a table comparing the mechanisms. The following sections then describe each mechanism in more detail.

Some talks at Embedded Linux Conference presented an overview of the current mechanisms:
* How do you update your embedded Linux devices? by Daniel Sangorrin / Keijiro Yano http://events.linuxfoundation.org/sites/events/files/slides/linuxcon-japan-2016-softwre-updates-sangorrin.pdf
* Comparison of Linux Software Update Technologies by Matt Porter http://events.linuxfoundation.org/sites/events/files/slides/Comparison%20of%20Linux%20Software%20Update%20Technologies_0.pdf. Video at https://youtu.be/pdHV9H9nZks?list=PLbzoR-pLrL6pRFP6SOywVJWdEHlmQE51q. Full paper done for Automotive Grade Linux (AGL) here: https://lists.linuxfoundation.org/pipermail/automotive-discussions/2016-May/002061.html
* Software update for IoT: the current state of play by Chris Simmonds http://de.slideshare.net/chrissimmonds/software-update-for-iot-the-current-state-of-play
* OSS Remote Firmware Updates for IoT-like Projects by Silvano Cirujano Cuesta http://events.linuxfoundation.org/sites/events/files/slides/OSS_Remote_Firmware_Updates_for_IoT-like_Projects.pdf
* "Surviving in the Wilderness: Integrity Protection and System Update" by Patrick Ohly: [https://openiotelcna2017.sched.com/event/9J5i/surviving-in-the-wilderness-integrity-protection-and-system-update-patrick-ohly-intel-gmbh abstract and slides], [https://www.youtube.com/watch?v=N8V0W0p3YBU video recording of talk]

== Comparison ==

;Type
: ''Block-based'' update mechanisms directly modify blocks in the partition(s) that they update, without going through the filesystem. This implies that the partition has to be the same for all devices and that devices must use exactly the same partition size. ''File-based'' update mechanisms modify files and directories. Therefore devices with different partition sizes can use the same update data and it may be possible to update without a reboot.
;Disk layout
: Dependencies on boot loader, number and kind of partitions, etc. Flexible mechanisms make no or only few assumptions about the system, but typically then require additional work to integrate with a specific setup.
;Rootfs
: The partition which contains the OS. May be strictly read-only (block-based update mechanisms) or read/write (file-based). Some update mechanisms support installing and updating a subset of the full OS.
;Updates from
: describes from where the update mechanism gets the update.
;Updates what
: describes which parts of the overall system the mechanism updates.
;Code stability
: Based on how long the code has been in use, personal experience, security track record in existing deployments, etc.
;OE/Yocto integration
: Whether the mechanism is already available and who supports it.
;Resource requirements on server
: affect both build time and long-term storage capacity. Likely to depend on the complexity of the changes.
;Resource requirements on client
: Amount of temporary disk space, CPU/network load, ..., again for different scenarios.
;Failure resilience
: Summarizes how the mechanism copes with potential problems.
;Complexity
: Some mechanisms are harder to use correctly than others (usability). Also includes how difficult is to set up the mechanism because of dependencies.
;Downtime
: How long normal operation of the device needs to be interrupted for an update.
;Security
: Compatibility with other technology, protection of the update mechanism itself.
;License
: License of the client running on the target device.
;Backend
: Back end solution for rolling out software updates.

{| class="wikitable"
! rowspan="2"|Mechanism
! rowspan="2"|Type
! rowspan="2"|Disk layout
! rowspan="2"|Rootfs
! rowspan="2"|Updates from
! rowspan="2"|Updates what
! rowspan="2"|Code stability
! rowspan="2"|OE/Yocto integration
! colspan="2"|Resource requirements
! rowspan="2"|Failure resilience
! rowspan="2"|Complexity
! rowspan="2"|Downtime
! rowspan="2"|Security
! rowspan="2"|License
! rowspan="2"|Backend
|-
! on server
! on client
|-
! [https://clearlinux.org/documentation/swupdate_about_sw_update.html swupd]
| file-based
| flexible
| read/write
| HTTP(S) server, local media
| depends on setup
| relatively stable, under active development
| [http://git.yoctoproject.org/cgit/cgit.cgi/meta-swupd meta-swupd]
| moderate, suitable for frequent updates
| minimal download, needs sufficient free space in rootfs
| favors fast updates over failure resilience
| some planning required
| minimal, reboot optional
| Compatible with Linux IMA, Smack, SELinux. Signed update data, HTTPS transfer protection.
| GPL-2.0-or-later
|
|-
! [https://github.com/sbabic/swupdate sbabic's swupdate]
| block-based / file based
| flexible
| depends on setup (read-only supported)
| local and remote (plain HTTP(S) or custom server)
| depends on setup
| Code relatively stable, 6 months release cycle
| [https://github.com/sbabic/meta-swupdate meta-swupdate], [https://github.com/sbabic/meta-swupdate-boards meta-swupdate-boards]
| archives full image per build
| download and write full (compressed) image, zero-copy
| integrated rollback (requires bootloader support)
| easy to use (but also very customizable)
| reboot required
| signed and encrypted images, HTTPS
| GPL-2.0-only
| [https://github.com/eclipse/hawkbit hawkBit], [https://github.com/siemens/wfx wfx], HTTP(S) server
|-
! [https://github.com/mendersoftware Mender]
| block-based / file based
| [https://docs.mender.io/Devices/Partition-layout flexible (minimum four partitions)], U-Boot or GRUB as boot loader
| supports read-write and read-only
| remote using Mender management server ([https://docs.mender.io/Architecture/Overview#modes-of-operation managed mode]) or local using CLI ([https://docs.mender.io/Architecture/Overview#modes-of-operation standalone mode])
| Complete rootfs, including kernel (built-in). Customizable with [https://docs.mender.io/artifacts/update-modules Update Modules].
| relatively stable, fully supported and tested [https://docs.mender.io/administration/upgrading upgrade path]
| [https://github.com/mendersoftware/meta-mender meta-mender]
| compressed rootfs per build
| download and write [https://docs.mender.io/architecture/mender-artifacts#streaming-and-compression compressed rootfs]
| integrated rollback
| easy when using meta-mender
| reboot required
| HTTPS enforced, [https://docs.mender.io/artifacts/signing-and-verification signed images]
| Apache 2.0
| [https://hosted.mender.io/ Hosted Mender] or self-hosted Mender server
|-
! [https://github.com/ostreedev/ostree OSTree]
| file-based
| flexible, but supports only limited set of bootloaders
| read/write, OS trees bind mounted read-only, /etc and /var writable
| local and remote repositories
| kernel and filesystem
| relatively stable, significant user base, under active development
| meta-ostree (WIP), [https://github.com/advancedtelematic/meta-updater meta-updater] (public)
| generating commits based on new builds, storing them in repository
| updating local repository, hard links for sharing unchanged content between deployments
| rollback to a different deployed OS tree
| some work required
| reboot required
| GPG-signed commits
| LGPL-2.0-or-later
| rpm-ostree?
|-
|-
! [https://github.com/rauc/rauc RAUC]
| block based / file-based (tar)
| flexible (block-device/MTD)
| depends on setup (read-only supported)
| depends on setup
| depends on setup (any storage device)
| relatively stable, under active development
| [https://github.com/rauc/meta-rauc meta-rauc]
| archives full (compressed) image per build
| download and write full (compressed) image
| integrated rollback and/or rescue (requires bootloader support)
| some customization required
| reboot required
| X509-signed update bundles
| LGPL-2.1-only
| [https://github.com/eclipse/hawkbit hawkBit] or HTTP(S) server (for example lighttpd) or USB Stick
|}

== swupd ==

;Disk layout
: swupd updates files in a single partition. Other than that, it makes no assumptions about disk layout, filesystem and boot mechanism.
;Rootfs
: Files provided by the OS are read-only, everything else is read/write (/etc, /var) and preserved during updates. OS can be split up into a core OS (always installed) and optional bundles which may or may not be installed.
;Updates from
: Uses libcurl to fetch update data and thus supports all URL schemes supported by libcurl, in particular http(s) and local files.
;Updates what
: swupd itself updates files in the rootfs, other components then need to be updated (boot loader, kernel, firmware, ...) then need be updated via custom plugins.
;Code stability
: Used and maintained in Clear Linux OS. Code relatively stable, but would benefit from a rewrite (evolved from a prototype). Code changes not written for Clear Linux OS tend to get merged only slowly or not at all (examples: [https://github.com/clearlinux/swupd-server/pull/25 assumption about filesystem], [https://github.com/clearlinux/swupd-server/pull/26 error checking], [https://github.com/clearlinux/swupd-client/pull/71 path configuration]).
;OE/Yocto integration
: Layer available, not part of Yocto releases, experimental. Supports building incremental updates (= only changes since last build are stored for new build) and deltas (= optimized update pack for specific previous builds, typically major milestones).
;Resource requirements on server
: Build time and storage for each update linear with total number of files (file system analysis, zero packs) plus linear with number of modified files (compression). When using swupd purely as update mechanism (i.e. no bundles), space requirement on the server could be reduced to linear with the number of modified files by not creating the zero packs. Optionally can prepare deltas from certain previous builds, which is linear with the number of modified files since each of those builds.
;Resource requirements on client
: In the best case (delta prepared by server), a single archive with just some file diffs gets downloaded, unpacked and applied. In other cases, each new or modified file gets downloaded and unpacked. Staging new content needs free space in the rootfs partition, i.e. partition must be at least twice as large as the base OS.
;Failure resilience
: No recovery mechanism built into swupd itself. Short period of time where interrupted update may leave behind inconsistent rootfs. No updates possible when there is not enough free space left. Updating files while system services are running reduces downtime, but is also more risky if system services aren't prepared for it. Could be extended to do updates without that risk, at the expense of increased downtime.
;Complexity
: Upgrade path must be considered as part of release process (deltas, incompatible changes).
;Downtime
: Downloading and staging in parallel to normal operation. Services are kept running until after the update, at which point the device admin needs to restart services or reboot depending on what was updated (not automated at the moment in meta-swupd layer).
;Security
: Compatible with Linux IMA, Smack, SELinux. Conceptually incompatible with dm-verity. Relies on HTTPS and (optionally) [https://clearlinux.org/blogs/security-software-update-clear-linux-os-intel%C2%AE-architecture signing] to protect integrity of update data (not integrated into meta-swupd yet).

== babic's SWUpdate ==

;Disk layout
: There is no constraint how software is stored. SWUPdate supports raw flash (NOR, NAND), UBI volumes, disk partitions or can update files (provided in a tarball) into an existing filesystem. Each artifact can be stored on a different storage device.
;Rootfs
: No constrain where software is stored. During an update, a single partition, multiple partitions or generically multiple different storages can be updated.
:
: SWUpdate is often used in one of the following setup:
:
:* '''rescue''' : The system reboots in maintenance mode and SWUpdate is started from a Ramdisk. Just one copy of the Software is stored into the system.
:* '''dual-copy''' : two copies of the software (rootfs, kernel) are stored into the system and SWUpdate installs the stand-by copy.
;Updates from
:
:* local provisioning : USB, SD, etc.
:* generic URL : HTTP(S), FTP. It uses the libcurl library and supports what libcurl provides.
:* Webserver : SWUpdate integrates a Webserver (mongoose)
:* External Backend connector (suricatta mode) to bind with an external backend server. Currently, the Hawkbit server is supported https://github.com/eclipse/hawkbit. Open to further backends.
:* Custom Protocol: SWUPdate provides a library (LGPLv2.1) in case of custom protocol.
;Updates what
:
:* bootloader (risky !)
:* kernel
:* interface to bootloader (U-Boot) Allows to change u-boot variables and allow to use plugins to make changes to other bootloaders.
;* disk partitions
;* provide interface to update FPGAs, external microcontrollers, etc.
;* custom handlers: an interface allows to add own installers written in C or in LUA.
;* system update: atomic update of network connected devices as they are just one.
;Resources on client
:
:* '''rescue''' : meta-swupdate provides recipe to generate a compressed ramdisk with small footprint. Including support for signed image, the whole rootfs is ~4MB. The minimal requirement for a complete rescue (bootloader, kernel for SWUpdate and ramdisk) is 8MB. This allows to put the rescue in a small storage like a SPI-NOR, while the software is stored on another and bigger device (NAND, eMMC).
:* '''dual-copy''' : Needs active/passive rootfs partition, and bandwidth to download compressed rootfs image. No additional space is required if the image is directly streamed to the stand-by copy.
;Failure resilience
: SWUpdate provides interface to bootloaders (U-Boot, GRUB, EFIBootGuard). Update can be marked good or bad and it starts a fallback (support from bootloader required).
; Downtime
:
:* '''rescue''' : There is need to reboot in maintenance mode and once the image is installed, needs to reboot again with the new production image.
:* '''dual-copy''' : Update is downloaded and applied during normal operation. Afterwards one reboot is required, no other downtime.
;Security
:
:* Connection with HTTPS to the external the server.
:* Signed images : it is possible to sign the images used for the update in order to check its integrity. X.509 cryptography (CMS) or RSA keys.
:* Split in several processes: connection to the internet can run with a different userid / groupid as the installer (privilege separation). The installer runs often with high privileges because it has to write the hardware.
:* Support for encrypted artifacts
:* Signing / Encryption with openSSL or MbedTLS.

== Mender ==

;Disk layout
: No upper limit on partitions, [https://docs.mender.io/devices/partition-layout minimum four]. Dual rootfs partitions with two extra partitions, a boot and a data partition. Depends on either U-Boot or GRUB as boot loader for the automatic rollback.
;Rootfs
: Stored in one active and one passive partition, read/write while in use, but modifications to it get lost during updates when switching partitions, so persistent data should be stored in the data partition. Kernel is stored on rootfs.
;Updates from
: Mender Server over HTTPS ([https://docs.mender.io/architecture/overview#modes-of-operation managed mode])
: manual provisioning : local file system/URL, e.g. USB, SD, HTTP(S) ([https://docs.mender.io/architecture/overview#modes-of-operation standalone mode])
;Update what
: Built-in support for rootfs update, including kernel. Can be customized via [https://docs.mender.io/artifacts/update-modules Update Modules] to update files, packages or external components.
;Code stability
: Stable release. Upgrade path fully supported and tested.
;Resources on server
: Needs to store one compressed rootfs image for each update, plus small meta data section.
;Resources on client
: Needs active/passive rootfs partition, and bandwidth to download compressed rootfs image. Needs no additional space on device beyond the partitions (update is streamed), space for the Mender binary, and a tiny local database.
;Failure resilience
: Automatic rollback if the device either fails to boot, or the Mender daemon cannot connect to the Mender server afterwards.
;Complexity
: Relatively easy to [https://docs.mender.io/Artifacts/Building-Mender-Yocto-image build with Yocto]. More complex if [https://mender.io/blog/porting-mender-to-a-non-yocto-build-system not using Yocto].
;Downtime
: Update is downloaded and applied during normal operation. Afterwards one reboot is required, no other downtime.
;Security
: Secure connection to the server (TLS). [https://docs.mender.io/1.1/artifacts/signing-and-verification Artifact signing] with RSA or ECDSA.

== OSTree ==

;Disk layout
:Only limited set of bootloaders supported (those that support [http://www.freedesktop.org/wiki/Specifications/BootLoaderSpec The Boot Loader Specification] and some exceptions). The root file system has to have the /boot directory managed by OSTree. The actual deployed OS trees have certain conventions and requirements. For example, /etc and /var need to be writable, while /usr is mounted read-only.
;Rootfs
:Files provided by OS are expected to be in /usr, which is mounted read-only. /etc and /var are writable.
;Updates from
:Local and remote repostories. OSTree works by replicating a repository to the target device and then "checking out" deployment directories from the repository. Local and HTTP methods for replicating the repository are available.
;Updates what
:OSTree (atomically) updates root file system trees and the kernel. If anything else needs to be updated, it needs to happen by running custom code on the target device.
;Code stability
:OSTree is relatively stable. It's used by many distributions and projects, such as Fedora Atomic and Flatpak. OSTree is under active development and has an open-source community around it.
;OE/Yocto integration
:Meta-ostree layer is work in progress.
;Resource requirements on server
:OSTree generates "commit objects" based on the filesystem changes between builds. These commit objects are stored in a commit chain the same way as git does. The commit objects transmitted over the network as binary diffs when the remote and local repositories are synchronized.
;Resource requirements on client
:Client has the local copy of the repository. The repository contains the objects, which map to the files on the OS trees. The deployments of the OS trees (or "checkouts" in git termininology) are made using hard links to the repository content. This means that the required space is only increased when the data changes: the static data files are shared between deployments.
:OSTree allows deleting of data. This means that the full operating system history doesn't need to be stored in the repository. A typical OSTree-based system might have two deployed OS trees: one that is being currently used and one fallback.
;Failure resilience
:OSTree controls the booting to the different deployed trees using bootloader entries. If booting a deployed OS tree fails, a different bootloader entry can be chosen for booting into a different OS tree.
;Complexity
:Some work is needed to fit the root filesystem into the conventions that OSTree likes. Some documentation of the needed steps [https://ostree.readthedocs.io/en/latest/manual/adapting-existing/ is available here].
;Downtime
:Reboot is required after updates. Changing between deployed OS trees is done by selecting a different boot loader entry.
;Security
:The commits can be signed with GPG-based signatures. OSTree repositories store file extended attributes, which means that the security mechanisms using extended attributes should be functional with OSTree.

== RAUC ==

;Type
: RAUC supports filesystems on block devices and MTD (NAND/NOR) as well as raw images (bootloaders, FPGA bitstreams).
;Disk layout
: RAUC does not depend on a fixed disk layout but allows to flexibly [https://rauc.readthedocs.io/en/latest/reference.html#sec-ref-slot-config configure] one. It supports A+B scenarios as well as A+recovery or more complex setups.
;Rootfs
: There are no limitations on the root file system, it can be read-only or read-write. It has to contain the RAUC system configuration file, a keyring for verification and the small RAUC binary itself.
;Updates from
: RAUC is the updating core that is meant to be integrated in a custom environment. Thus it supports installing bundles from local file paths, while another application is responsible fetching the update.
: External storage (USB, SD): direct access, no copying required
: Webserver: Application downloads Bundle to local tmpfs / storage.
: Demo-applications planned.
;Updates what
:
:* everything which is updatable
:* default handler for common storage types and file systems (ext4, NAND, UBI, raw copy)
:* allows [https://rauc.readthedocs.io/en/latest/using.html#customizing-the-update custom handler] for each update target (script or application, information passed as environment variables)
;Resources on client
: At least one production rootfs and a rescue system. Temporary storage for update Bundle (compressed) if provided by a webserver.
;Failure resilience
: Interface to [https://rauc.readthedocs.io/en/latest/integration.html#interfacing-with-the-bootloader Booloader] (Barebox, U-Boot, GRUB, UEFI) allows atomic updates (correct fallback handling is up to the bootloader). Interface to mark updates good or bad after certain checks in the newly booted rootfs (must be supported by bootloader).
;Complexity
: Integration into the rootfs and generating update Bundles is relatively easy with Yocto ([https://github.com/rauc/meta-rauc meta-rauc]) or [https://www.mail-archive.com/ptxdist@pengutronix.de/msg11471.html PTXdist]. Manual integration described [https://rauc.readthedocs.io/en/latest/integration.html here].
;Downtime
: Normally, downloading and installing will be perforemd as background operation. A reboot is needed to make the new system active (might be scheduled).
;Security
: Signed images: Signing images is mandatory. X.509 cryptography (CMS) is used to sign the bundle and a full PKI with revocations is supported. An example script allows creating key/cert/keychain for test purposes.
: Verified boot: When using image updates, compatible with IMA, SELinux, dm-verity, ... (as it does require write access to the filesystems).

System Update

2025-07-29T11:56:51Z

Thomas Perrot:

== Introduction ==

This page compares different system update mechanisms. The purpose is to help the project with picking a suitable mechanism that the project then will support going forward. Users may find this page relevant for picking a mechanism that suits their specific needs.

A system update mechanism must ensure that a device running an older release of the operating systems runs with a more recent release when the update mechanism is done. This includes updating everything that defines the system (rootfs, kernel, bootloader, etc.), restarting running processes and potentially a reboot. An ideal mechanism:
* never ends up in an inconsistent state (atomic update),
* always keeps the device usable (fallback to previous state when there are problems, or at least supporting a recovery mode),
* requires little additional resources (disk space, RAM),
* minimizes downtime while updating,
* works in combination with security technology (integrity protection),
* is secure (does not install or execute software created by an attacker).

These are conflicting requirements. Different mechanisms will have different strengths and weaknesses. Therefore the first chapter provides a more detailed definition of the different aspects and has a table comparing the mechanisms. The following sections then describe each mechanism in more detail.

Some talks at Embedded Linux Conference presented an overview of the current mechanisms:
* How do you update your embedded Linux devices? by Daniel Sangorrin / Keijiro Yano http://events.linuxfoundation.org/sites/events/files/slides/linuxcon-japan-2016-softwre-updates-sangorrin.pdf
* Comparison of Linux Software Update Technologies by Matt Porter http://events.linuxfoundation.org/sites/events/files/slides/Comparison%20of%20Linux%20Software%20Update%20Technologies_0.pdf. Video at https://youtu.be/pdHV9H9nZks?list=PLbzoR-pLrL6pRFP6SOywVJWdEHlmQE51q. Full paper done for Automotive Grade Linux (AGL) here: https://lists.linuxfoundation.org/pipermail/automotive-discussions/2016-May/002061.html
* Software update for IoT: the current state of play by Chris Simmonds http://de.slideshare.net/chrissimmonds/software-update-for-iot-the-current-state-of-play
* OSS Remote Firmware Updates for IoT-like Projects by Silvano Cirujano Cuesta http://events.linuxfoundation.org/sites/events/files/slides/OSS_Remote_Firmware_Updates_for_IoT-like_Projects.pdf
* "Surviving in the Wilderness: Integrity Protection and System Update" by Patrick Ohly: [https://openiotelcna2017.sched.com/event/9J5i/surviving-in-the-wilderness-integrity-protection-and-system-update-patrick-ohly-intel-gmbh abstract and slides], [https://www.youtube.com/watch?v=N8V0W0p3YBU video recording of talk]

== Comparison ==

;Type
: ''Block-based'' update mechanisms directly modify blocks in the partition(s) that they update, without going through the filesystem. This implies that the partition has to be the same for all devices and that devices must use exactly the same partition size. ''File-based'' update mechanisms modify files and directories. Therefore devices with different partition sizes can use the same update data and it may be possible to update without a reboot.
;Disk layout
: Dependencies on boot loader, number and kind of partitions, etc. Flexible mechanisms make no or only few assumptions about the system, but typically then require additional work to integrate with a specific setup.
;Rootfs
: The partition which contains the OS. May be strictly read-only (block-based update mechanisms) or read/write (file-based). Some update mechanisms support installing and updating a subset of the full OS.
;Updates from
: describes from where the update mechanism gets the update.
;Updates what
: describes which parts of the overall system the mechanism updates.
;Code stability
: Based on how long the code has been in use, personal experience, security track record in existing deployments, etc.
;OE/Yocto integration
: Whether the mechanism is already available and who supports it.
;Resource requirements on server
: affect both build time and long-term storage capacity. Likely to depend on the complexity of the changes.
;Resource requirements on client
: Amount of temporary disk space, CPU/network load, ..., again for different scenarios.
;Failure resilience
: Summarizes how the mechanism copes with potential problems.
;Complexity
: Some mechanisms are harder to use correctly than others (usability). Also includes how difficult is to set up the mechanism because of dependencies.
;Downtime
: How long normal operation of the device needs to be interrupted for an update.
;Security
: Compatibility with other technology, protection of the update mechanism itself.
;License
: License of the client running on the target device.
;Backend
: Back end solution for rolling out software updates.

{| class="wikitable"
! rowspan="2"|Mechanism
! rowspan="2"|Type
! rowspan="2"|Disk layout
! rowspan="2"|Rootfs
! rowspan="2"|Updates from
! rowspan="2"|Updates what
! rowspan="2"|Code stability
! rowspan="2"|OE/Yocto integration
! colspan="2"|Resource requirements
! rowspan="2"|Failure resilience
! rowspan="2"|Complexity
! rowspan="2"|Downtime
! rowspan="2"|Security
! rowspan="2"|License
! rowspan="2"|Backend
|-
! on server
! on client
|-
! [https://clearlinux.org/documentation/swupdate_about_sw_update.html swupd]
| file-based
| flexible
| read/write
| HTTP(S) server, local media
| depends on setup
| relatively stable, under active development
| [http://git.yoctoproject.org/cgit/cgit.cgi/meta-swupd meta-swupd]
| moderate, suitable for frequent updates
| minimal download, needs sufficient free space in rootfs
| favors fast updates over failure resilience
| some planning required
| minimal, reboot optional
| Compatible with Linux IMA, Smack, SELinux. Signed update data, HTTPS transfer protection.
| GPL-2.0-or-later
|
|-
! [https://github.com/sbabic/swupdate sbabic's swupdate]
| block-based / file based
| flexible
| depends on setup (read-only supported)
| local and remote (plain HTTP(S) or custom server)
| depends on setup
| Code relatively stable, 6 months release cycle
| [https://github.com/sbabic/meta-swupdate meta-swupdate], [https://github.com/sbabic/meta-swupdate-boards meta-swupdate-boards]
| archives full image per build
| download and write full (compressed) image, zero-copy
| integrated rollback (requires bootloader support)
| easy to use (but also very customizable)
| reboot required
| signed and encrypted images, HTTPS
| GPL-2.0-only
| [https://github.com/eclipse/hawkbit hawkBit], [https://github.com/siemens/wfx wfx], HTTP(S) server
|-
! [https://github.com/mendersoftware Mender]
| block-based / file based
| [https://docs.mender.io/Devices/Partition-layout flexible (minimum four partitions)], U-Boot or GRUB as boot loader
| supports read-write and read-only
| remote using Mender management server ([https://docs.mender.io/Architecture/Overview#modes-of-operation managed mode]) or local using CLI ([https://docs.mender.io/Architecture/Overview#modes-of-operation standalone mode])
| Complete rootfs, including kernel (built-in). Customizable with [https://docs.mender.io/artifacts/update-modules Update Modules].
| relatively stable, fully supported and tested [https://docs.mender.io/administration/upgrading upgrade path]
| [https://github.com/mendersoftware/meta-mender meta-mender]
| compressed rootfs per build
| download and write [https://docs.mender.io/architecture/mender-artifacts#streaming-and-compression compressed rootfs]
| integrated rollback
| easy when using meta-mender
| reboot required
| HTTPS enforced, [https://docs.mender.io/artifacts/signing-and-verification signed images]
| Apache 2.0
| [https://hosted.mender.io/ Hosted Mender] or self-hosted Mender server
|-
! [https://github.com/ostreedev/ostree OSTree]
| file-based
| flexible, but supports only limited set of bootloaders
| read/write, OS trees bind mounted read-only, /etc and /var writable
| local and remote repositories
| kernel and filesystem
| relatively stable, significant user base, under active development
| meta-ostree (WIP), [https://github.com/advancedtelematic/meta-updater meta-updater] (public)
| generating commits based on new builds, storing them in repository
| updating local repository, hard links for sharing unchanged content between deployments
| rollback to a different deployed OS tree
| some work required
| reboot required
| GPG-signed commits
| LGPL-2.0-or-later
| rpm-ostree?
|-
|-
! [https://github.com/rauc/rauc RAUC]
| block based / file-based (tar)
| flexible (block-device/MTD)
| depends on setup (read-only supported)
| depends on setup
| depends on setup (any storage device)
| relatively stable, under active development
| [https://github.com/rauc/meta-rauc meta-rauc]
| archives full (compressed) image per build
| download and write full (compressed) image
| integrated rollback (requires bootloader support)
| some customization required
| reboot required
| X509-signed update bundles
| LGPL-2.1-only
| [https://github.com/eclipse/hawkbit hawkBit] or HTTP(S) server (for example lighttpd) or USB Stick
|}

== swupd ==

;Disk layout
: swupd updates files in a single partition. Other than that, it makes no assumptions about disk layout, filesystem and boot mechanism.
;Rootfs
: Files provided by the OS are read-only, everything else is read/write (/etc, /var) and preserved during updates. OS can be split up into a core OS (always installed) and optional bundles which may or may not be installed.
;Updates from
: Uses libcurl to fetch update data and thus supports all URL schemes supported by libcurl, in particular http(s) and local files.
;Updates what
: swupd itself updates files in the rootfs, other components then need to be updated (boot loader, kernel, firmware, ...) then need be updated via custom plugins.
;Code stability
: Used and maintained in Clear Linux OS. Code relatively stable, but would benefit from a rewrite (evolved from a prototype). Code changes not written for Clear Linux OS tend to get merged only slowly or not at all (examples: [https://github.com/clearlinux/swupd-server/pull/25 assumption about filesystem], [https://github.com/clearlinux/swupd-server/pull/26 error checking], [https://github.com/clearlinux/swupd-client/pull/71 path configuration]).
;OE/Yocto integration
: Layer available, not part of Yocto releases, experimental. Supports building incremental updates (= only changes since last build are stored for new build) and deltas (= optimized update pack for specific previous builds, typically major milestones).
;Resource requirements on server
: Build time and storage for each update linear with total number of files (file system analysis, zero packs) plus linear with number of modified files (compression). When using swupd purely as update mechanism (i.e. no bundles), space requirement on the server could be reduced to linear with the number of modified files by not creating the zero packs. Optionally can prepare deltas from certain previous builds, which is linear with the number of modified files since each of those builds.
;Resource requirements on client
: In the best case (delta prepared by server), a single archive with just some file diffs gets downloaded, unpacked and applied. In other cases, each new or modified file gets downloaded and unpacked. Staging new content needs free space in the rootfs partition, i.e. partition must be at least twice as large as the base OS.
;Failure resilience
: No recovery mechanism built into swupd itself. Short period of time where interrupted update may leave behind inconsistent rootfs. No updates possible when there is not enough free space left. Updating files while system services are running reduces downtime, but is also more risky if system services aren't prepared for it. Could be extended to do updates without that risk, at the expense of increased downtime.
;Complexity
: Upgrade path must be considered as part of release process (deltas, incompatible changes).
;Downtime
: Downloading and staging in parallel to normal operation. Services are kept running until after the update, at which point the device admin needs to restart services or reboot depending on what was updated (not automated at the moment in meta-swupd layer).
;Security
: Compatible with Linux IMA, Smack, SELinux. Conceptually incompatible with dm-verity. Relies on HTTPS and (optionally) [https://clearlinux.org/blogs/security-software-update-clear-linux-os-intel%C2%AE-architecture signing] to protect integrity of update data (not integrated into meta-swupd yet).

== babic's SWUpdate ==

;Disk layout
: There is no constraint how software is stored. SWUPdate supports raw flash (NOR, NAND), UBI volumes, disk partitions or can update files (provided in a tarball) into an existing filesystem. Each artifact can be stored on a different storage device.
;Rootfs
: No constrain where software is stored. During an update, a single partition, multiple partitions or generically multiple different storages can be updated.
:
: SWUpdate is often used in one of the following setup:
:
:* '''rescue''' : The system reboots in maintenance mode and SWUpdate is started from a Ramdisk. Just one copy of the Software is stored into the system.
:* '''dual-copy''' : two copies of the software (rootfs, kernel) are stored into the system and SWUpdate installs the stand-by copy.
;Updates from
:
:* local provisioning : USB, SD, etc.
:* generic URL : HTTP(S), FTP. It uses the libcurl library and supports what libcurl provides.
:* Webserver : SWUpdate integrates a Webserver (mongoose)
:* External Backend connector (suricatta mode) to bind with an external backend server. Currently, the Hawkbit server is supported https://github.com/eclipse/hawkbit. Open to further backends.
:* Custom Protocol: SWUPdate provides a library (LGPLv2.1) in case of custom protocol.
;Updates what
:
:* bootloader (risky !)
:* kernel
:* interface to bootloader (U-Boot) Allows to change u-boot variables and allow to use plugins to make changes to other bootloaders.
;* disk partitions
;* provide interface to update FPGAs, external microcontrollers, etc.
;* custom handlers: an interface allows to add own installers written in C or in LUA.
;* system update: atomic update of network connected devices as they are just one.
;Resources on client
:
:* '''rescue''' : meta-swupdate provides recipe to generate a compressed ramdisk with small footprint. Including support for signed image, the whole rootfs is ~4MB. The minimal requirement for a complete rescue (bootloader, kernel for SWUpdate and ramdisk) is 8MB. This allows to put the rescue in a small storage like a SPI-NOR, while the software is stored on another and bigger device (NAND, eMMC).
:* '''dual-copy''' : Needs active/passive rootfs partition, and bandwidth to download compressed rootfs image. No additional space is required if the image is directly streamed to the stand-by copy.
;Failure resilience
: SWUpdate provides interface to bootloaders (U-Boot, GRUB, EFIBootGuard). Update can be marked good or bad and it starts a fallback (support from bootloader required).
; Downtime
:
:* '''rescue''' : There is need to reboot in maintenance mode and once the image is installed, needs to reboot again with the new production image.
:* '''dual-copy''' : Update is downloaded and applied during normal operation. Afterwards one reboot is required, no other downtime.
;Security
:
:* Connection with HTTPS to the external the server.
:* Signed images : it is possible to sign the images used for the update in order to check its integrity. X.509 cryptography (CMS) or RSA keys.
:* Split in several processes: connection to the internet can run with a different userid / groupid as the installer (privilege separation). The installer runs often with high privileges because it has to write the hardware.
:* Support for encrypted artifacts
:* Signing / Encryption with openSSL or MbedTLS.

== Mender ==

;Disk layout
: No upper limit on partitions, [https://docs.mender.io/devices/partition-layout minimum four]. Dual rootfs partitions with two extra partitions, a boot and a data partition. Depends on either U-Boot or GRUB as boot loader for the automatic rollback.
;Rootfs
: Stored in one active and one passive partition, read/write while in use, but modifications to it get lost during updates when switching partitions, so persistent data should be stored in the data partition. Kernel is stored on rootfs.
;Updates from
: Mender Server over HTTPS ([https://docs.mender.io/architecture/overview#modes-of-operation managed mode])
: manual provisioning : local file system/URL, e.g. USB, SD, HTTP(S) ([https://docs.mender.io/architecture/overview#modes-of-operation standalone mode])
;Update what
: Built-in support for rootfs update, including kernel. Can be customized via [https://docs.mender.io/artifacts/update-modules Update Modules] to update files, packages or external components.
;Code stability
: Stable release. Upgrade path fully supported and tested.
;Resources on server
: Needs to store one compressed rootfs image for each update, plus small meta data section.
;Resources on client
: Needs active/passive rootfs partition, and bandwidth to download compressed rootfs image. Needs no additional space on device beyond the partitions (update is streamed), space for the Mender binary, and a tiny local database.
;Failure resilience
: Automatic rollback if the device either fails to boot, or the Mender daemon cannot connect to the Mender server afterwards.
;Complexity
: Relatively easy to [https://docs.mender.io/Artifacts/Building-Mender-Yocto-image build with Yocto]. More complex if [https://mender.io/blog/porting-mender-to-a-non-yocto-build-system not using Yocto].
;Downtime
: Update is downloaded and applied during normal operation. Afterwards one reboot is required, no other downtime.
;Security
: Secure connection to the server (TLS). [https://docs.mender.io/1.1/artifacts/signing-and-verification Artifact signing] with RSA or ECDSA.

== OSTree ==

;Disk layout
:Only limited set of bootloaders supported (those that support [http://www.freedesktop.org/wiki/Specifications/BootLoaderSpec The Boot Loader Specification] and some exceptions). The root file system has to have the /boot directory managed by OSTree. The actual deployed OS trees have certain conventions and requirements. For example, /etc and /var need to be writable, while /usr is mounted read-only.
;Rootfs
:Files provided by OS are expected to be in /usr, which is mounted read-only. /etc and /var are writable.
;Updates from
:Local and remote repostories. OSTree works by replicating a repository to the target device and then "checking out" deployment directories from the repository. Local and HTTP methods for replicating the repository are available.
;Updates what
:OSTree (atomically) updates root file system trees and the kernel. If anything else needs to be updated, it needs to happen by running custom code on the target device.
;Code stability
:OSTree is relatively stable. It's used by many distributions and projects, such as Fedora Atomic and Flatpak. OSTree is under active development and has an open-source community around it.
;OE/Yocto integration
:Meta-ostree layer is work in progress.
;Resource requirements on server
:OSTree generates "commit objects" based on the filesystem changes between builds. These commit objects are stored in a commit chain the same way as git does. The commit objects transmitted over the network as binary diffs when the remote and local repositories are synchronized.
;Resource requirements on client
:Client has the local copy of the repository. The repository contains the objects, which map to the files on the OS trees. The deployments of the OS trees (or "checkouts" in git termininology) are made using hard links to the repository content. This means that the required space is only increased when the data changes: the static data files are shared between deployments.
:OSTree allows deleting of data. This means that the full operating system history doesn't need to be stored in the repository. A typical OSTree-based system might have two deployed OS trees: one that is being currently used and one fallback.
;Failure resilience
:OSTree controls the booting to the different deployed trees using bootloader entries. If booting a deployed OS tree fails, a different bootloader entry can be chosen for booting into a different OS tree.
;Complexity
:Some work is needed to fit the root filesystem into the conventions that OSTree likes. Some documentation of the needed steps [https://ostree.readthedocs.io/en/latest/manual/adapting-existing/ is available here].
;Downtime
:Reboot is required after updates. Changing between deployed OS trees is done by selecting a different boot loader entry.
;Security
:The commits can be signed with GPG-based signatures. OSTree repositories store file extended attributes, which means that the security mechanisms using extended attributes should be functional with OSTree.

== RAUC ==

;Type
: RAUC supports filesystems on block devices and MTD (NAND/NOR) as well as raw images (bootloaders, FPGA bitstreams).
;Disk layout
: RAUC does not depend on a fixed disk layout but allows to flexibly [https://rauc.readthedocs.io/en/latest/reference.html#sec-ref-slot-config configure] one. It supports A+B scenarios as well as A+recovery or more complex setups.
;Rootfs
: There are no limitations on the root file system, it can be read-only or read-write. It has to contain the RAUC system configuration file, a keyring for verification and the small RAUC binary itself.
;Updates from
: RAUC is the updating core that is meant to be integrated in a custom environment. Thus it supports installing bundles from local file paths, while another application is responsible fetching the update.
: External storage (USB, SD): direct access, no copying required
: Webserver: Application downloads Bundle to local tmpfs / storage.
: Demo-applications planned.
;Updates what
:
:* everything which is updatable
:* default handler for common storage types and file systems (ext4, NAND, UBI, raw copy)
:* allows [https://rauc.readthedocs.io/en/latest/using.html#customizing-the-update custom handler] for each update target (script or application, information passed as environment variables)
;Resources on client
: At least one production rootfs and a rescue system. Temporary storage for update Bundle (compressed) if provided by a webserver.
;Failure resilience
: Interface to [https://rauc.readthedocs.io/en/latest/integration.html#interfacing-with-the-bootloader Booloader] (Barebox, U-Boot, GRUB, UEFI) allows atomic updates (correct fallback handling is up to the bootloader). Interface to mark updates good or bad after certain checks in the newly booted rootfs (must be supported by bootloader).
;Complexity
: Integration into the rootfs and generating update Bundles is relatively easy with Yocto ([https://github.com/rauc/meta-rauc meta-rauc]) or [https://www.mail-archive.com/ptxdist@pengutronix.de/msg11471.html PTXdist]. Manual integration described [https://rauc.readthedocs.io/en/latest/integration.html here].
;Downtime
: Normally, downloading and installing will be perforemd as background operation. A reboot is needed to make the new system active (might be scheduled).
;Security
: Signed images: Signing images is mandatory. X.509 cryptography (CMS) is used to sign the bundle and a full PKI with revocations is supported. An example script allows creating key/cert/keychain for test purposes.
: Verified boot: When using image updates, compatible with IMA, SELinux, dm-verity, ... (as it does require write access to the filesystems).

User:Thomas Perrot

2025-02-04T08:45:39Z

Thomas Perrot:

Embedded Linux and kernel engineer at Bootlin

pub rsa3072/0x9FC00B0571FE0AED Thomas Perrot <thomas.perrot@bootlin.com>
Key fingerprint = 8740 77C6 A6A3 0A23 03A8 1221 9FC0 0B05 71FE 0AED

pub rsa4096/0x53A3D309F9177FB2 Thomas Perrot <thomas.perrot@tupi.fr>
Key fingerprint = 088F DE87 B7F1 F018 B520 666B 53A3 D309 F917 7FB2

On Matrix as @tperrot:matrix.org.

<s>On IRC Libera as tprrt or tperrot.</s>

In Southern France, close to Toulouse.

User:Thomas Perrot

2021-06-06T17:47:33Z

Thomas Perrot:

Embedded Linux and kernel engineer at Bootlin

pub rsa3072/0x9FC00B0571FE0AED Thomas Perrot <thomas.perrot@bootlin.com>
Key fingerprint = 8740 77C6 A6A3 0A23 03A8 1221 9FC0 0B05 71FE 0AED

pub rsa4096/0x53A3D309F9177FB2 Thomas Perrot <thomas.perrot@tupi.fr>
Key fingerprint = 088F DE87 B7F1 F018 B520 666B 53A3 D309 F917 7FB2

On IRC Libera as tprrt or tperrot.

In Southern France, close to Toulouse.

User:Thomas Perrot

2021-06-06T17:44:20Z

Thomas Perrot:

Yocto Build Failure Swat Team

2021-03-15T07:21:08Z

Thomas Perrot:

== Overview ==

All builds that are run on the public autobuilder are important for the Yocto Project, whether they be routine validation runs or pre-integration test builds. Random failures if ignored accumulate and can result in a significant number of builds failing.

The role of the Bug Swat Team is to monitor the autobuilder and do preliminary investigation of failures, to ensure that they are logged and brought to the attention of the appropriate owner.

Importantly, the Swat Team '''isn't responsible for resolving issues''' encountered on the autobuilder, simply just enough analysis so that it can be logged for later analysis and ideally make the right people aware of them.

Each week a different member of the team is on call. Every build that fails on the autobuilder should be monitored unless stated otherwise. The rotation happens at the end of Friday (deliberately vague), any failures over the weekend should be triaged by the incoming member on Monday.

The Swat Chairs are the primary contact for the Swat Team. The current Swat Chairs are [[User:RossBurton | Ross Burton]] and [[User:Rpurdie | Richard Purdie]]. The Chairs are assisted by Stephen K. Jolley who handles the rotation process. If the person currently on call, or about to be on call, can no longer perform their duty then they should contact Stephen to arrange a replacement.

== Process ==

The SWAT process is now using a specific tool, [https://swatbot.yoctoproject.org/ swatbot]. Swatbot has a filter which will list all the pending issues that need to be triaged using [https://swatbot.yoctoproject.org/mainindex/swat/ this link ] or the "SWAT Pending Builds" link on the left hand menu. Each issue has links to the autobuilder logs for the failing step (e.g. usually stdio and warning/error logs).

The builds are shown in a tree like structure with the parent build and any child builds under it. The builds are edited as a group under the parent as quite often a failure might be common to the child builds. Each failure does need to be triaged individually although multiple builds can be changed at once to a given resolution.

Swatbot can filter pending issues to be triaged using the [https://swatbot.yoctoproject.org/mainindex/swat/ SWAT Pending Builds] link. Once you have selected an issue to triage, you will have to take the correct reporting action and finally edit the entry to indicate what was done.

You can also be notified when a build fails by subscribing to the [https://lists.yoctoproject.org/g/yocto-builds yocto-builds] mailing list. This is sending a mail when a build fails, including direct links to the [https://autobuilder.yoctoproject.org/ autobuilder job summary] and the [https://errors.yoctoproject.org/Errors/Latest/Autobuilder/ Error Reporting Service]. The mail will also state if it is expected that the build is triaged by Swat, so check this to see if the build can be ignored as the owner is taking full responsibility. Currently, swatbot will not give you this information so you have to get it from the autobuilder build entry (the <tt>Build properties</tt> tab should have: <tt>swat_monitor true</tt>), the autobuilder API, or the notification email.

Another tool that can be used to monitor builds is the [https://autobuilder.yoctoproject.org/typhoon/#/console Autobuilder 'Yocto Console View'] which is an overview of the top-level builds (''a-full'' and ''a-quick'') and the sub-builds they trigger.

Both the top-level build entry and the mail notification will include notes from the build owner, so check this for any useful context. For example, it may request that failures are reported directly to a specific person instead of bugs created, or that particular failures that are expected.

=== Report ===

There are two categories of builds that Swat will be monitoring: official branches and staging branches. The official branches are the primary top-level branches in Poky, that is master and all of the release branches (gatesgarth, dunfell, etc). The staging branches are where patches are held for testing, such as master-next, stable/dunfell-nut, or ross/mut.

Communication is important: if the build owner is on IRC then it's always worth discussing issues with them first as they may have further context and directions. Also, if the build owner triages the build failures then they must update the swatbot entries so that Swat doesn't duplicate the work.

When reporting an issue, be it in a mailing list post or a new bug, the following information should be included:
* Relevant details about the build configuration. For example: did the failure happen just once, or in all PowerPC builds? Was it specific to multilib configurations? Look across the entire build run and identify any patterns.
* The error itself. Trim the log down to just the error and any relevant context in the bug description.
* A link to the build failure. Either a link to the [http://errors.yoctoproject.org/ error reports] page (such as http://errors.yoctoproject.org/Errors/Details/199667/) or a link to the autobuilder build log (such as https://autobuilder.yoctoproject.org/typhoon/#/builders/34/builds/168).

When filing bugs, always search Bugzilla first to see if the issue is already known. For example, there are some bugs that occur intermittently and are already filed with ''AB-INT'' in the whiteboard field. They are listed here: [https://bugzilla.yoctoproject.org/buglist.cgi?quicksearch=whiteboard%3AAB-INT&list_id=640327 Autobuilder issues]

The exact progress depends on whether the branch is an official branch or a staging branch.

==== Staging Branches ====

For builds against staging branches which contain patches under test for integration (such as master-next, stable/dunfell-nut, ross/mut, etc), first attempt to identify if there is a patch in the branch that is likely to be responsible for the failure. For example, if <tt>wget</tt> fails with <tt>libgnutls</tt> errors and there is a GnuTLS upgrade in the branch, then that is a likely candidate. If a patch can be identified that hasn't yet been merged into an official branch, then reply to the patch on the mailing list with the details. If it isn't obvious which patch is responsible for the failure, or a patch can be identified but it has already been merged to the release branch, then file a bug and ensure the branch maintainer (see the [[Releases]] page for names) is on the CC list.

Most of the failures will be for staging branches as master-next is the branch that is tested the most. However, it is rebased quite frequently so it is not always easy to find which patchs were included. In that case, you have to get the actual commit hash, for example in the build properties, the variable is <tt>yp_build_revision</tt> or in the build configuration at the beginning of the stdio log. For example, this qemux86 build [https://autobuilder.yoctoproject.org/typhoon/#/builders/59/builds/3120] was master-next at revision 47482eff9897ccde946e9247724babc3a586d318.
With that information, you can then clone poky (or any other layer of interest) and fetch the proper commit and see the git log:

<pre>
$ git clone git://git.yoctoproject.org/poky
$ cd poky
$ git fetch origin 47482eff9897ccde946e9247724babc3a586d318
$ git log FETCH_HEAD
</pre>

'''If in doubt, file a bug'''. All errors must be taken care of.

If the issue is in the infrastructure or autobuilder itself then file a bug against "Infrastructure: Autobuilder", infrastructure bugs should be assigned to [[User:Halstead| Michael Halstead]] and autobuilder logic bugs to [[User:Rpurdie | Richard Purdie]].

==== Official Branches ====

For builds of official branches, that is master or a release branch, '''all failures or warnings are critical''' and must be [[#Filing_bugs | filed in Bugzilla]]. Remember to check that the issue isn't already filed. Where an issue is already filed, please do add a comment so we can assess how frequently different issues are occurring.

=== Update ===

Finally the swatbot build entry must be updated with a summary of the outcome. Three different resolutions are available:
* Mail sent: used when you replied to the problematic patch directly on the mailing list.
* Bug Opened: used when a new bug has been opened or a new comment has been added to a bug. Please add the bug number in the notes.
* Handled (other): used when the maintainer is already aware of the issue and is working on it resolution or a patch has already been sent to solve the issue.

You need an account for this step. If it hasn't been provided, please ask on the swat mailling list.

'''Every issue that is dealt with must be annotated''', so it is easy to see which issues have been handled. This includes filing new bugs, finding existing bugs, contacting the mailing list, contacting the maintainer directly on IRC, or identifying that a patch has already been sent to fix the issue.

== Tips ==

An issue will quite often repeat itself across multiple builds. It is worth looking for those repetitions as swatbot will allow you to select many builds and update them all at once.

In the stdio logs window, clicking on the looking glass icon will load the full log in the browser, allowing you to use the search feature of the browser. Quite often, you'll start by looking for "error:".

== Handoff ==

At the end of the week, the outgoing person on Swat should email swat@lists.yoctoproject.org summarising the week and noting anything that the incoming person on Swat next week should be aware of. For example, noting that there's a new intermittent bug to watch for.

== Members ==

* [[User:RossBurton | Ross Burton]]

* [[User:Leonardo_Sandoval | Leo Sandoval]]

* [[User:Anibal Limon | Anibal Limon]]

* [[User:Köry maincent | Köry Maincent]]

* [[User:Thomas Perrot | Thomas Perrot]]

* [[User:SaulWold | Saul Wold]]

* [[User:Oleksiy Obitotskyy | Oleksiy Obitotskyy]]

* [[User:Alejandro Enedino Hernandez Samaniego | Alejandro Hernandez Samaniego]]

* [[User:PaulEggleton | Paul Eggleton]]

* [[User:Naveen Kumar Saini | Naveen Saini]]

* [[User:Alexandre Belloni | Alexandre Belloni]]

* [[User:Kergoth | Christopher Larson]]

* [[User:Lee_chee_yang | Lee Chee Yang]]

* [[User:Jon Mason | Jon Mason]]

* [[User:Minjae Kim | Minjae Kim]]

* [[User:Jagadheesan | Jaga]]

Yocto Build Failure Swat Team

2021-03-08T15:52:52Z

Thomas Perrot:

== Overview ==

All builds that are run on the public autobuilder are important for the Yocto Project, whether they be routine validation runs or pre-integration test builds. Random failures if ignored accumulate and can result in a significant number of builds failing.

The role of the Bug Swat Team is to monitor the autobuilder and do preliminary investigation of failures, to ensure that they are logged and brought to the attention of the appropriate owner.

Importantly, the Swat Team '''isn't responsible for resolving issues''' encountered on the autobuilder, simply just enough analysis so that it can be logged for later analysis and ideally make the right people aware of them.

Each week a different member of the team is on call. Every build that fails on the autobuilder should be monitored unless stated otherwise. The rotation happens at the end of Friday (deliberately vague), any failures over the weekend should be triaged by the incoming member on Monday.

The Swat Chairs are the primary contact for the Swat Team. The current Swat Chairs are [[User:RossBurton | Ross Burton]] and [[User:Rpurdie | Richard Purdie]]. The Chairs are assisted by Stephen K. Jolley who handles the rotation process. If the person currently on call, or about to be on call, can no longer perform their duty then they should contact Stephen to arrange a replacement.

== Process ==

The SWAT process is now using a specific tool, [https://swatbot.yoctoproject.org/ swatbot]. Swatbot has a filter which will list all the pending issues that need to be triaged using [https://swatbot.yoctoproject.org/mainindex/swat/ this link ] or the "SWAT Pending Builds" link on the left hand menu. Each issue has links to the autobuilder logs for the failing step (e.g. usually stdio and warning/error logs).

The builds are shown in a tree like structure with the parent build and any child builds under it. The builds are edited as a group under the parent as quite often a failure might be common to the child builds. Each failure does need to be triaged individually although multiple builds can be changed at once to a given resolution.

Swatbot can filter pending issues to be triaged using the [https://swatbot.yoctoproject.org/mainindex/swat/ SWAT Pending Builds] link. Once you have selected an issue to triage, you will have to take the correct reporting action and finally edit the entry to indicate what was done.

You can also be notified when a build fails by subscribing to the [https://lists.yoctoproject.org/g/yocto-builds yocto-builds] mailing list. This is sending a mail when a build fails, including direct links to the [https://autobuilder.yoctoproject.org/ autobuilder job summary] and the [https://errors.yoctoproject.org/Errors/Latest/Autobuilder/ Error Reporting Service]. The mail will also state if it is expected that the build is triaged by Swat, so check this to see if the build can be ignored as the owner is taking full responsibility. Currently, swatbot will not give you this information so you have to get it from the autobuilder build entry (the <tt>Build properties</tt> tab should have: <tt>swat_monitor true</tt>), the autobuilder API, or the notification email.

Another tool that can be used to monitor builds is the [https://autobuilder.yoctoproject.org/typhoon/#/console Autobuilder 'Yocto Console View'] which is an overview of the top-level builds (''a-full'' and ''a-quick'') and the sub-builds they trigger.

Both the top-level build entry and the mail notification will include notes from the build owner, so check this for any useful context. For example, it may request that failures are reported directly to a specific person instead of bugs created, or that particular failures that are expected.

=== Report ===

There are two categories of builds that Swat will be monitoring: official branches and staging branches. The official branches are the primary top-level branches in Poky, that is master and all of the release branches (gatesgarth, dunfell, etc). The staging branches are where patches are held for testing, such as master-next, stable/dunfell-nut, or ross/mut.

Communication is important: if the build owner is on IRC then it's always worth discussing issues with them first as they may have further context and directions. Also, if the build owner triages the build failures then they must update the swatbot entries so that Swat doesn't duplicate the work.

When reporting an issue, be it in a mailing list post or a new bug, the following information should be included:
* Relevant details about the build configuration. For example: did the failure happen just once, or in all PowerPC builds? Was it specific to multilib configurations? Look across the entire build run and identify any patterns.
* The error itself. Trim the log down to just the error and any relevant context in the bug description.
* A link to the build failure. Either a link to the [http://errors.yoctoproject.org/ error reports] page (such as http://errors.yoctoproject.org/Errors/Details/199667/) or a link to the autobuilder build log (such as https://autobuilder.yoctoproject.org/typhoon/#/builders/34/builds/168).

When filing bugs, always search Bugzilla first to see if the issue is already known. For example, there are some bugs that occur intermittently and are already filed with ''AB-INT'' in the whiteboard field. They are listed here: [https://bugzilla.yoctoproject.org/buglist.cgi?quicksearch=AB-INT&list_id=620110 Autobuilder issues]

The exact progress depends on whether the branch is an official branch or a staging branch.

==== Staging Branches ====

For builds against staging branches which contain patches under test for integration (such as master-next, stable/dunfell-nut, ross/mut, etc), first attempt to identify if there is a patch in the branch that is likely to be responsible for the failure. For example, if <tt>wget</tt> fails with <tt>libgnutls</tt> errors and there is a GnuTLS upgrade in the branch, then that is a likely candidate. If a patch can be identified that hasn't yet been merged into an official branch, then reply to the patch on the mailing list with the details. If it isn't obvious which patch is responsible for the failure, or a patch can be identified but it has already been merged to the release branch, then file a bug and ensure the branch maintainer (see the [[Releases]] page for names) is on the CC list.

Most of the failures will be for staging branches as master-next is the branch that is tested the most. However, it is rebased quite frequently so it is not always easy to find which patchs were included. In that case, you have to get the actual commit hash, for example in the build properties, the variable is <tt>yp_build_revision</tt> or in the build configuration at the beginning of the stdio log. For example, this qemux86 build [https://autobuilder.yoctoproject.org/typhoon/#/builders/59/builds/3120] was master-next at revision 47482eff9897ccde946e9247724babc3a586d318.
With that information, you can then clone poky (or any other layer of interest) and fetch the proper commit and see the git log:

<pre>
$ git clone git://git.yoctoproject.org/poky
$ cd poky
$ git fetch origin 47482eff9897ccde946e9247724babc3a586d318
$ git log FETCH_HEAD
</pre>

'''If in doubt, file a bug'''. All errors must be taken care of.

If the issue is in the infrastructure or autobuilder itself then file a bug against "Infrastructure: Autobuilder", infrastructure bugs should be assigned to [[User:Halstead| Michael Halstead]] and autobuilder logic bugs to [[User:Rpurdie | Richard Purdie]].

==== Official Branches ====

For builds of official branches, that is master or a release branch, '''all failures or warnings are critical''' and must be [[#Filing_bugs | filed in Bugzilla]]. Remember to check that the issue isn't already filed. Where an issue is already filed, please do add a comment so we can assess how frequently different issues are occurring.

=== Update ===

Finally the swatbot build entry must be updated with a summary of the outcome. Three different resolutions are available:
* Mail sent: used when you replied to the problematic patch directly on the mailing list.
* Bug Opened: used when a new bug has been opened or a new comment has been added to a bug. Please add the bug number in the notes.
* Handled (other): used when the maintainer is already aware of the issue and is working on it resolution or a patch has already been sent to solve the issue.

You need an account for this step. If it hasn't been provided, please ask on the swat mailling list.

'''Every issue that is dealt with must be annotated''', so it is easy to see which issues have been handled. This includes filing new bugs, finding existing bugs, contacting the mailing list, contacting the maintainer directly on IRC, or identifying that a patch has already been sent to fix the issue.

== Tips ==

An issue will quite often repeat itself across multiple builds. It is worth looking for those repetitions as swatbot will allow you to select many builds and update them all at once.

In the stdio logs window, clicking on the looking glass icon will load the full log in the browser, allowing you to use the search feature of the browser. Quite often, you'll start by looking for "error:".

== Handoff ==

At the end of the week, the outgoing person on Swat should email swat@lists.yoctoproject.org summarising the week and noting anything that the incoming person on Swat next week should be aware of. For example, noting that there's a new intermittent bug to watch for.

== Members ==

* [[User:RossBurton | Ross Burton]]

* [[User:Leonardo_Sandoval | Leo Sandoval]]

* [[User:Anibal Limon | Anibal Limon]]

* [[User:Köry maincent | Köry Maincent]]

* [[User:Thomas Perrot | Thomas Perrot]]

* [[User:SaulWold | Saul Wold]]

* [[User:Alejandro Enedino Hernandez Samaniego | Alejandro Hernandez Samaniego]]

* [[User:PaulEggleton | Paul Eggleton]]

* [[User:Naveen Kumar Saini | Naveen Saini]]

* [[User:Alexandre Belloni | Alexandre Belloni]]

* [[User:Kergoth | Christopher Larson]]

* [[User:Lee_chee_yang | Lee Chee Yang]]

* [[User:Jon Mason | Jon Mason]]

* [[User:Minjae Kim | Minjae Kim]]

* [[User:Jagadheesan | Jaga]]

User:Thomas Perrot

2021-03-08T12:40:39Z

Thomas Perrot:

User:Thomas Perrot

2021-03-08T12:40:17Z

Thomas Perrot:

User:Thomas Perrot

2021-03-08T12:38:27Z

Thomas Perrot:

User:Thomas Perrot

2021-03-08T12:33:26Z

Thomas Perrot:

Embedded Linux and kernel engineer at Bootlin

tprrt or tperrot on Freenode

https://www.linkedin.com/in/tprrt/

User:Thomas Perrot

2021-03-08T11:40:22Z

Thomas Perrot:

Embedded Linux and kernel engineer at Bootlin

https://www.linkedin.com/in/tprrt/