Yocto Build Failure Swat Team

Overview

All builds that are run on the public autobuilder are important for the Yocto Project, whether they be routine validation runs or pre-integration test builds. Random failures if ignored accumulate and can result in a significant number of builds failing.

The role of the Bug Swat Team is to monitor the autobuilder and do preliminary investigation of failures, to ensure that they are logged and brought to the attention of the appropriate owner.

Importantly, the Swat Team isn't responsible for resolving issues encountered on the autobuilder, simply just enough analysis so that it can be logged for later analysis.

Each week a different member of the team is on call. Every build that fails on the autobuilder should be monitored unless stated otherwise. The rotation happens at the end of Friday (deliberately vague), any failures over the weekend should be triaged by the incoming member on Monday.

The Swat Chairs are the primary contact for the Swat Team. The current Swat Chairs are Ross Burton, Armin Kuster and Richard Purdie. The Chairs are assisted by Stephen K. Jolley who handles the rotation process. If the person currently on call, or about to be on call, can no longer perform their duty then they should contact Stephen to arrange a replacement.

Process

The process is simply three steps:

Identify build failures
Report the build failures
Update the build log

Identify

To be notified when a build fails subscribe to the yocto-builds mailing list. This is sent a mail when a build fails, including direct links to the autobuilder job summary, the BuildLog, and the Error Reporting Service. The mail will also state if it is expected that the build is triaged by Swat, so check this to see if the build can be ignored as the owner is taking full responsibility.

There are several services that are used when monitoring the builds:

The Autobuilder 'Yocto Console View' is an overview of the top-level builds (a-full and a-quick) and the sub-builds they trigger.
The Error Reporting Service archives errors from the autobuilder.
The BuildLog is a wiki page that is updated automatically when builds fail with links to the appropriate logs, and is where Swat adds notes explaining the failures and any resolution.

Both the mail notification and the BuildLog will include notes from the build owner, so check this for any useful context. For example, it may request that failures are reported directly to a specific person instead of bugs created, or that particular failures that are expected.

Report

There are two categories of builds that Swat will be monitoring: official branches and staging branches. The official branches are the primary top-level branches in Poky, that is master and all of the release branches (gatesgarth, dunfell, etc). The staging branches are where patches are held for testing, such as master-next, stable/dunfell-nut, or ross/mut.

Communication is important: if the build owner is on IRC then it's always worth discussing issues with them first as they may have further context and directions. Also, if the build owner triages the build failures then they must update the BuildLog so that Swat doesn't duplicate the work.

When filing bugs, always search Bugzilla first to see if the issue is already known. For example, there are some bugs that occur intermittently and are already filed with AB-INT in the whiteboard field.

When reporting an issue, be it in a mailing list post or a new bug, the following information should be included:

Relevant details about the build configuration. For example: did the failure happen just once, or in all PowerPC builds? Was it specific to multilib configurations? Look across the entire build run and identify any patterns.
The error itself. Trim the log down to just the error and any relevant context in the bug description.
A link to the build failure. Either a link to the error reports page (such as http://errors.yoctoproject.org/Errors/Details/199667/) or a link to the autobuilder build log (such as https://autobuilder.yoctoproject.org/typhoon/#/builders/34/builds/168).

The exact progress depends on whether the branch is an official branch or a staging branch.

Official Branches

For builds of official branches, that is master or a release branch, all failures or warnings are critical and must be filed in Bugzilla. Remember to check that the issue isn't already filed.

Staging Branches

For builds against staging branches which contain patches under test for integration into an official branch (such as master-next, stable/dunfell-nut, ross/mut, etc), first attempt to identify if there is a patch in the branch that is likely to be responsible for the failure. For example, if wget fails with libgnutls errors and there is a GnuTLS upgrade in the branch, then that is a likely candidate. If a patch can be identified that hasn't yet been merged into an official branch, then reply to the patch on the mailing list with the details. If it isn't obvious which patch is responsible for the failure, or a patch can be identified but it has already been merged to the release branch, then file a bug and ensure the branch maintainer (see the Releases page for names) is either the assignee or on the CC list. In both situations the BuildLog must be updated with details of what action was taken (mailed the list, filed a bug, and so on).

If in doubt, file a bug. All errors must be actioned unless a patch has already been sent for the issue, in which case please make note of this in the BuildLog.

If the issue is in the infrastructure or autobuilder itself then file a bug against "Infrastructure: Autobuilder", infrastructure bugs should be assigned to Michael Halstead and autobuilder logic bugs to Richard Purdie.