Yocto Project - User contributions [en]

User:Anibal Limon

2020-12-22T18:19:21Z

Anibal Limon:

Software Integration Engineer at Linaro, Living in México

User:Anibal Limon

2020-12-22T18:18:43Z

Anibal Limon:

Software Integration Engineer at Linaro
Living in Mexico

Yocto Build Failure Swat Team

2020-12-22T18:18:12Z

Anibal Limon: /* Members */

== Overview ==

All builds that are run on the public autobuilder are important for the Yocto Project, whether they be routine validation runs or pre-integration test builds. Random failures if ignored accumulate and can result in a significant number of builds failing.

The role of the Bug Swat Team is to monitor the autobuilder and do preliminary investigation of failures, to ensure that they are logged and brought to the attention of the appropriate owner.

Importantly, the Swat Team '''isn't responsible for resolving issues''' encountered on the autobuilder, simply just enough analysis so that it can be logged for later analysis and ideally make the right people aware of them.

Each week a different member of the team is on call. Every build that fails on the autobuilder should be monitored unless stated otherwise. The rotation happens at the end of Friday (deliberately vague), any failures over the weekend should be triaged by the incoming member on Monday.

The Swat Chairs are the primary contact for the Swat Team. The current Swat Chairs are [[User:RossBurton | Ross Burton]], Armin Kuster and [[User:Rpurdie | Richard Purdie]]. The Chairs are assisted by Stephen K. Jolley who handles the rotation process. If the person currently on call, or about to be on call, can no longer perform their duty then they should contact Stephen to arrange a replacement.

== Process ==

The process is simply three steps:
# Identify build failures
# Report the build failures
# Update the build log

=== Identify ===

To be notified when a build fails subscribe to the [https://lists.yoctoproject.org/g/yocto-builds yocto-builds] mailing list. This is sent a mail when a build fails, including direct links to the [https://autobuilder.yoctoproject.org/ autobuilder job summary], the [[BuildLog]], and the [https://errors.yoctoproject.org/Errors/Latest/Autobuilder/ Error Reporting Service]. The mail will also state if it is expected that the build is triaged by Swat, so check this to see if the build can be ignored as the owner is taking full responsibility.

There are several services that are used when monitoring the builds:
* The [https://autobuilder.yoctoproject.org/typhoon/#/console Autobuilder 'Yocto Console View'] is an overview of the top-level builds (''a-full'' and ''a-quick'') and the sub-builds they trigger. From here you can drill down into the sub-builds and the logs for each phase of the build.
* The [https://errors.yoctoproject.org/ Error Reporting Service] archives errors from the autobuilder.
* The [[BuildLog]] is a wiki page that is updated automatically when builds fail with links to the appropriate logs, and is where Swat adds notes explaining the failures and any resolution.

Both the mail notification and the [[BuildLog]] will include notes from the build owner, so check this for any useful context. For example, it may request that failures are reported directly to a specific person instead of bugs created, or that particular failures that are expected.

=== Report ===

There are two categories of builds that Swat will be monitoring: official branches and staging branches. The official branches are the primary top-level branches in Poky, that is master and all of the release branches (gatesgarth, dunfell, etc). The staging branches are where patches are held for testing, such as master-next, stable/dunfell-nut, or ross/mut.

Communication is important: if the build owner is on IRC then it's always worth discussing issues with them first as they may have further context and directions. Also, if the build owner triages the build failures then they must update the BuildLog so that Swat doesn't duplicate the work.

When reporting an issue, be it in a mailing list post or a new bug, the following information should be included:
* Relevant details about the build configuration. For example: did the failure happen just once, or in all PowerPC builds? Was it specific to multilib configurations? Look across the entire build run and identify any patterns.
* The error itself. Trim the log down to just the error and any relevant context in the bug description.
* A link to the build failure. Either a link to the [http://errors.yoctoproject.org/ error reports] page (such as http://errors.yoctoproject.org/Errors/Details/199667/) or a link to the autobuilder build log (such as https://autobuilder.yoctoproject.org/typhoon/#/builders/34/builds/168).

When filing bugs, always search Bugzilla first to see if the issue is already known. For example, there are some bugs that occur intermittently and are already filed with ''AB-INT'' in the whiteboard field.

The exact progress depends on whether the branch is an official branch or a staging branch.

==== Official Branches ====

For builds of official branches, that is master or a release branch, '''all failures or warnings are critical''' and must be [[#Filing_bugs | filed in Bugzilla]]. Remember to check that the issue isn't already filed. Where an issue is already filed, please do add a comment so we can assess how frequently different issues are occurring.

==== Staging Branches ====

For builds against staging branches which contain patches under test for integration (such as master-next, stable/dunfell-nut, ross/mut, etc), first attempt to identify if there is a patch in the branch that is likely to be responsible for the failure. For example, if <tt>wget</tt> fails with <tt>libgnutls</tt> errors and there is a GnuTLS upgrade in the branch, then that is a likely candidate. If a patch can be identified that hasn't yet been merged into an official branch, then reply to the patch on the mailing list with the details. If it isn't obvious which patch is responsible for the failure, or a patch can be identified but it has already been merged to the release branch, then file a bug and ensure the branch maintainer (see the [[Releases]] page for names) is on the CC list.

'''If in doubt, file a bug'''. All errors must be actioned unless a patch has already been sent for the issue, in which case make note of this in the [[BuildLog]].

If the issue is in the infrastructure or autobuilder itself then file a bug against "Infrastructure: Autobuilder", infrastructure bugs should be assigned to [[User:Halstead| Michael Halstead]] and autobuilder logic bugs to [[User:Rpurdie | Richard Purdie]].

=== Update ===

Finally the [[BuildLog]] must be updated with a summary of the outcome. Some examples:
* "Glib upgrade on list causes multilib failures, replied on mailing list"
* "New qemu hangs on PPC, filed bug #111"
* "Test cases timing out, existing bug #222"

'''Every issue that is dealt with must be annotated''', so it is easy to see which issues have been handled and which is outstanding. This includes filing new bugs, finding existing bugs, contacting the mailing list, contacting the maintainer directly on IRC, or identifying that a patch has already been sent to fix the issue.

== Handoff ==

At the end of the week, the outgoing person on Swat should email swat@lists.yoctoproject.org summarising the week and noting anything that the incoming person on Swat next week should be aware of. For example, noting that there's a new intermittent bug to watch for.

== Members ==

* [[User:RossBurton | Ross Burton]]

* [[User:Leonardo_Sandoval | Leo Sandoval]]

* [[User:Anibal Limon | Anibal Limon]]

* [[User:SaulWold | Saul Wold]]

* [[User:Alejandro Enedino Hernandez Samaniego | Alejandro Hernandez Samaniego]]

* [[User:PaulEggleton | Paul Eggleton]]

* [[User:Naveen Kumar Saini | Naveen Saini]]

* Armin Kuster

* [[User:Kergoth | Christopher Larson]]

* Lee Chee Yang

* [[User:Jon Mason | Jon Mason]]

* [[User:Minjae Kim | Minjae Kim]]

Reproducible Builds

2017-03-13T21:09:37Z

Anibal Limon: /* Verification */

== Current Status ==

The Yocto Project aims to have builds which are entirely reproducible. That is; if you run a build today, then run that same configuration X time in the future, the binaries you get out of the build should be binary identical.

This implies that the host system you run the build on and the path you run the build in should not affect the target system output.

Our build system doesn't produce binary reproducible builds today, but we are actively working towards that goal and fixing issues as we idenitify them. We also plan to improve our testing to help find reproducibility issues.

The design of the system lends itself very well to producing reproducible builds, as we provide a reproducible build environment with minimal dependencies on the host/build OS.

We have several technologies in place which aide in reproducibility:

Our shared state (sstate) mechanism base our builds on hashes of input metadata, reusing the outputs if the inputs are the same.

As our sstate files need to be reusable regardless of build path and can be target or native binaries, we have mechanisms for working around various issues such as hard-coded paths (though we'd prefer to remove the need for them entirely).

A related problem to sstate is that of knowing when the input has changed, has the output changed? This is useful in the context of package feeds, amongst other things, to know whether we should update them or not.
We have binary build comparison tools at an early stage of development to allow us to reduce unnecessary churn in package feeds, however further developments are planned in this area using the tools from the reproducible-builds project.

Our SDKs need to be relocatable and run anywhere they are installed to. We include our own C library to do this in a "run anywhere" scenario and are able to generate fully relocatable toolchains.

Our new for Yocto Project 2.3 (Pyro) recipe specific sysroots ensure that the output of a recipe doesn't change depending on whether other recipes have already been built and in which order, even when the software tries to autodetect available features.

At this point the project does not intend to target timestamp levels of reproducibility so whilst the binary content should be the same, file timestamps may not be and this means package manager tarballs would not be binary identical due to timestamp differences

== Why do we want reproducible builds? ==

The many benefits of reproducible builds as listed in the [https://reproducible-builds.org/ reproducible-builds] project, not least of all the ability to verify a built output matches the source, are key motivations for our work on reproducible builds. Additonal benefits specific to the Yocto Project include:
* ability to reduce the churn in package feeds
* ability to improve reliability of allarch recipes and enable wider use of them

== Related Work ==

Some key bugs that are important in our reproducibility efforts are:
* <s>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=1560 #1560] Enable recipe specific sysroots</s> — DONE for 2.3
* [https://bugzilla.yoctoproject.org/show_bug.cgi?id=5866 #5866] Reproducible builds: identical binaries
* [https://bugzilla.yoctoproject.org/show_bug.cgi?id=10813 #10813] Replace build-compare with tools from the reproducible-builds project
* Make use of <tt>[https://reproducible-builds.org/specs/source-date-epoch/ SOURCE_DATE_EPOCH]</tt>, most likely with patches from [https://wiki.debian.org/ReproducibleBuilds/TimestampsProposal Debian] to ensure tools are using it.
* Develop tests to catch issues with reproducibility

== Verification ==

To ensure that our builds are reproducible we need to implement an infrastructure of verification over the build systems outputs Images, SDKs, Binary packages, etc.

The idea is to run diffoscope over target shared states (populate_sysroot) because is the first output on what the other artifacts (Images, SDK's, Binary packages) are generated. The build output will be generated running bitbake world with two different Host systems, this can be done using two Virtual machines as Autobuilder workers and then launch the comparison process (diffoscope) against the two build outputs.

A script exists to call diffoscope against shared states, the AB needs to use something similar to get the comparison results. [https://github.com/alimon/yocto-reproduciblebuilds-data/blob/master/sstate_diffoscope.py]. There is a example of the output here [https://github.com/alimon/yocto-reproduciblebuilds-data/tree/master/poky_master_qemux86_d454dc2fc].

An Autobuilder development branch for reproducible builds. [http://git.yoctoproject.org/cgit/cgit.cgi/yocto-autobuilder/log/?h=contrib/alimon/reproducible_builds]

To preserve the results a web system needs to be implemented (the AB will push the results), this also will serve to publish something like www.yoctoproject.org/reproduciblebuilds to see the actual status and what recipes needs works in order to be 100% reproducible. An initial implementation only with database models. [https://github.com/alimon/reproducible-builds]

Reproducible Builds

2017-03-13T20:55:15Z

Anibal Limon:

== Current Status ==

The Yocto Project aims to have builds which are entirely reproducible. That is; if you run a build today, then run that same configuration X time in the future, the binaries you get out of the build should be binary identical.

This implies that the host system you run the build on and the path you run the build in should not affect the target system output.

Our build system doesn't produce binary reproducible builds today, but we are actively working towards that goal and fixing issues as we idenitify them. We also plan to improve our testing to help find reproducibility issues.

The design of the system lends itself very well to producing reproducible builds, as we provide a reproducible build environment with minimal dependencies on the host/build OS.

We have several technologies in place which aide in reproducibility:

Our shared state (sstate) mechanism base our builds on hashes of input metadata, reusing the outputs if the inputs are the same.

As our sstate files need to be reusable regardless of build path and can be target or native binaries, we have mechanisms for working around various issues such as hard-coded paths (though we'd prefer to remove the need for them entirely).

A related problem to sstate is that of knowing when the input has changed, has the output changed? This is useful in the context of package feeds, amongst other things, to know whether we should update them or not.
We have binary build comparison tools at an early stage of development to allow us to reduce unnecessary churn in package feeds, however further developments are planned in this area using the tools from the reproducible-builds project.

Our SDKs need to be relocatable and run anywhere they are installed to. We include our own C library to do this in a "run anywhere" scenario and are able to generate fully relocatable toolchains.

Our new for Yocto Project 2.3 (Pyro) recipe specific sysroots ensure that the output of a recipe doesn't change depending on whether other recipes have already been built and in which order, even when the software tries to autodetect available features.

At this point the project does not intend to target timestamp levels of reproducibility so whilst the binary content should be the same, file timestamps may not be and this means package manager tarballs would not be binary identical due to timestamp differences

== Why do we want reproducible builds? ==

The many benefits of reproducible builds as listed in the [https://reproducible-builds.org/ reproducible-builds] project, not least of all the ability to verify a built output matches the source, are key motivations for our work on reproducible builds. Additonal benefits specific to the Yocto Project include:
* ability to reduce the churn in package feeds
* ability to improve reliability of allarch recipes and enable wider use of them

== Related Work ==

Some key bugs that are important in our reproducibility efforts are:
* <s>[https://bugzilla.yoctoproject.org/show_bug.cgi?id=1560 #1560] Enable recipe specific sysroots</s> — DONE for 2.3
* [https://bugzilla.yoctoproject.org/show_bug.cgi?id=5866 #5866] Reproducible builds: identical binaries
* [https://bugzilla.yoctoproject.org/show_bug.cgi?id=10813 #10813] Replace build-compare with tools from the reproducible-builds project
* Make use of <tt>[https://reproducible-builds.org/specs/source-date-epoch/ SOURCE_DATE_EPOCH]</tt>, most likely with patches from [https://wiki.debian.org/ReproducibleBuilds/TimestampsProposal Debian] to ensure tools are using it.
* Develop tests to catch issues with reproducibility

== Verification ==

To ensure that our builds are reproducible we need to implement an infrastructure of verification over the build systems outputs Images, SDKs, Binary packages, etc.

The idea is to run diffoscope over target shared states (populate_sysroot) because is the first output on what the other artifacts (Images, SDK's, Binary packages) are generated. The build output will be generated running bitbake world with two different Host systems, this can be done using two Virtual machines as Autobuilder workers and then launch the comparison process (diffoscope) against the two build outputs.

An Autobuilder development branch for reproducible builds. [http://git.yoctoproject.org/cgit/cgit.cgi/yocto-autobuilder/log/?h=contrib/alimon/reproducible_builds]

To preserve the results a web system needs to be implemented (the AB will push the results), this also will serve to publish something like www.yoctoproject.org/reproduciblebuilds to see the actual status and what recipes needs works in order to be 100% reproducible. An initial implementation only with database models. [https://github.com/alimon/reproducible-builds]