Yocto Build Failure Swat Team

Overview

The role of the Bug Swat Team is to monitor the autobuilder and do preliminary investigation of failures, to ensure that they are logged and brought to the attention of the appropriate owner.

All builds that are run on the public autobuilder are important for the Yocto Project, whether they be routine validation runs (master or release branches) or a pre-integration test builds (master-next, stable/*, and others). Random failures if ignored accumulate and can result in most builds failing.

Each week a different member of the team is on call. Every build that fails on the autobuilder should be monitored unless told otherwise.

Importantly, the Swat Team isn't responsible for resolving issues encountered on the autobuilder, just enough analysis that it can be logged and the appropriate owner notified.

The Swat Chairs are the primary contact for the Swat Team. The current Swat Chairs are Ross Burton, Armin Kuster and Richard Purdie. The Chairs are assisted by Stephen K. Jolley who handles the rotation process.

Process

The BuildLog wiki page is automatically updated by the autobuilder when a new build is triggered. This often includes guidance for SWAT (the "Reason" field when triggering the build) such as to ignore it (if the build is expected to break and the owner is monitoring themselves) or a request to report failures straight to a specific person (if the build only contains changes from a single person, such as a GCC upgrade). Unless when told otherwise, the usual process is as follows:

For builds against master or a release branch, any issues observed should be filed in Bugzilla.

For builds against other branches (master-next, ross/mut, -next branches for stable releases, etc.), attempt to identify what patch in the branch is likely responsible for the failure. For example, if wget fails with libgnutls errors and there is a GnuTLS upgrade in the branch, that is the likely candidate. If a patch can be identified that isn't also in master or release branch, reply on the mailing list with the failure details. If it isn't obvious which patch is responsible for the failure then file a bug and ensure the branch owner is either the assignee or on the CC list.

If in doubt, file a bug. All observed errors must be actioned unless a patch has already been sent for the issue, in which case please make note of this in the BuildLog.

If the issue is in the infrastructure or autobuilder itself then file a bug against , infrastructure bugs should be assigned to Michael Halstead and autobuilder logic bugs to Richard Purdie.

The results of pre-triage for an issue should be added to the corresponding entry in the BuildLog, including a link to the resolution (patch name, bug link, etc) and a brief summary of the issue. Every issue should be added to the build log so it acts as a build status report.

Note that some builds, in particular the "perf" builds are not listed on BuildLog unless the build fails (to try and reduce noise on the log). Failures in performance test builds should be handled like any other build.

The net result is all failures listed in BuildLog should have outcomes listed against them from the person on SWAT at the time.

Filing bugs

When filing the bug, several items must be included:

• The error itself. Trim the log down to just the error and any relevant context in the bug description, and also attach the complete log file. Autobuilder logs are not persistent so the full log must be attached.
• A link to the build failure. Ideally a link to the error reports page (such as http://errors.yoctoproject.org/Errors/Details/199667/) but a link to the autobuilder build log is acceptable (such as https://autobuilder.yoctoproject.org/typhoon/#/builders/34/builds/168).

Process summary

• Monitor builds via one or more of:
  • the autobuilder: main page, console view. Orange builds have warnings, red builds failed with errors.
  • the Error Reports web page which has an entry for each recipe that fails. A link to the appropriate search is on the autobuilder's Console View.
  • the BuildLog wiki page. This has links to each build log that wasn't successful.
  • the yocto-builds mailing list. This gets a mail for every unsuccessful build.
• Pre-triage each failure:
  • File a bug or respond to a patch or note known issues
  • Update the BuildLog with the result of pre-triage, linking to the resolution where possible.

Questions / Contact

If you have queries about the SWAT process you may reach out to the SWAT Facilitator Stephen K. Jolley or the SWAT Chair Ross Burton.

Members

Ross Burton

Leo Sandoval

Saul Wold

Anibal Limon

Alejandro Hernandez Samaniego

Paul Eggleton

Naveen Saini

Armin Kuster (place me anywhere)

Christopher Larson

Lee Chee Yang