Yocto Build Failure Swat Team: Difference between revisions

From Yocto Project
Jump to navigationJump to search
No edit summary
 
(69 intermediate revisions by 13 users not shown)
Line 1: Line 1:
== Overview ==
== Overview ==


The role of the SWAT team is to monitor the autobuilder and investigate all failures to ensure that they are logged and brought to the attention of the appropriate owner.
All builds that are run on the public autobuilder are important for the Yocto Project, whether they be routine validation runs or pre-integration test builds. Random failures if ignored accumulate and can result in a significant number of builds failing.


All builds that are run on the [https://autobuilder.yoctoproject.org public autobuilder] are important for the Yocto Project, whether they be a post-merge validation run (for master or a release branch) or a pre-merge test build (for master-next, ross/mut, stable/* and others).  SWAT doesn't cover company-internal autobuilders.
The role of the Bug Swat Team is to monitor the autobuilder and do preliminary investigation of failures, to ensure that they are logged and brought to the attention of the appropriate owner.


Every build should be monitored by the SWAT team unless the [[BuildLog]] entry for that build indicates otherwise.  SWAT is opt-out by whomever triggers a build on the autobuilder, not opt-in.
Importantly, the Swat Team '''isn't responsible for resolving issues''' encountered on the autobuilder, simply just enough analysis so that it can be logged for later analysis and ideally make the right people aware of them.


SWAT isn't responsible for resolving any issues encountered on the autobuilder. Their focus is on performing minimal analysis of a failure in order to ensure that it is logged and brought to the attention of a appropriate owner, a process we'll refer to as pre-triage.
Each week a different member of the team is on call. Every build that fails on the autobuilder should be monitored unless stated otherwise. The rotation happens at the end of Friday (deliberately vague), any failures over the weekend should be triaged by the incoming member on Monday.  


== Roles ==
The Swat Chairs are the primary contact for the Swat Team. The current Swat Chairs are [[User:RossBurton | Ross Burton]] and [[User:Rpurdie | Richard Purdie]]. The Chairs are assisted by Stephen K. Jolley who handles the rotation process.  If the person currently on call, or about to be on call, can no longer perform their duty then they should contact Stephen to arrange a replacement.


Active member: the currently active member of the SWAT team is expected to monitor the autobuilder and pre-triage failures in a timely fashion. Team members are active for one week at a time and rotation is usually simple round robin through the members listIf for some reason the next person isn't available then they can be defer a week until they are.
== Process ==
 
The SWAT process is now using a specific tool, [https://swatbot.yoctoproject.org/ swatbot]. Swatbot has a filter which will list all the pending issues that need to be triaged using [https://swatbot.yoctoproject.org/mainindex/swat/  this link ] or the "SWAT Pending Builds" link on the left hand menu. Each issue has links to the autobuilder logs for the failing step (e.g. usually stdio and warning/error logs).
 
The builds are shown in a tree like structure with the parent build and any child builds under it. The builds are edited as a group under the parent as quite often a failure might be common to the child builds. Each failure does need to be triaged individually although multiple builds can be changed at once to a given resolution.
 
Swatbot can filter pending issues to be triaged using the [https://swatbot.yoctoproject.org/mainindex/swat/ SWAT Pending Builds] link. Once you have selected an issue to triage, you will have to take the correct reporting action and finally edit the entry to indicate what was done.
 
You can also be notified when a build fails by subscribing to the [https://lists.yoctoproject.org/g/yocto-builds yocto-builds] mailing list.  This is sending a mail when a build fails, including direct links to the [https://autobuilder.yoctoproject.org/ autobuilder job summary] and the [https://errors.yoctoproject.org/Errors/Latest/Autobuilder/ Error Reporting Service].  The mail will also state if it is expected that the build is triaged by Swat, so check this to see if the build can be ignored as the owner is taking full responsibility. Currently, swatbot will not give you this information so you have to get it from the autobuilder build entry (the <tt>Build properties</tt> tab should have: <tt>swat_monitor true</tt>), the autobuilder API, or the notification email.
 
Another tool that can be used to monitor builds is the [https://autobuilder.yoctoproject.org/typhoon/#/console Autobuilder 'Yocto Console View'] which is an overview of the top-level builds (''a-full'' and ''a-quick'') and the sub-builds they trigger.
 
Both the top-level build entry and the mail notification will include notes from the build owner, so check this for any useful context.  For example, it may request that failures are reported directly to a specific person instead of bugs created, or that particular failures that are expected.
 
=== Report ===
 
There are two categories of builds that Swat will be monitoring: official branches and staging branches.  The official branches are the primary top-level branches in Poky, that is master and all of the release branches (gatesgarth, dunfell, etc)The staging branches are where patches are held for testing, such as master-next, stable/dunfell-nut, or ross/mut.
 
Communication is important: if the build owner is on IRC then it's always worth discussing issues with them first as they may have further context and directions.  Also, if the build owner triages the build failures then they must update the swatbot entries so that Swat doesn't duplicate the work.


SWAT Chair: the SWAT chair provides backup cover for the active member and is a first point of contact for SWAT. [[User:RossBurton | Ross Burton]], Armin Kuster and Richard Purdie are the SWAT Chair people.
When reporting an issue, be it in a mailing list post or a new bug, the following information should be included:
* Relevant details about the build configuration. For example: did the failure happen just once, or in all PowerPC builds? Was it specific to multilib configurations?  Look across the entire build run and identify any patterns.
* The error itself. Trim the log down to just the error and any relevant context in the bug description.
* A link to the build failure.  Either a link to the [http://errors.yoctoproject.org/ error reports] page (such as http://errors.yoctoproject.org/Errors/Details/199667/) or a link to the autobuilder build log (such as https://autobuilder.yoctoproject.org/typhoon/#/builders/34/builds/168).


SWAT Facilitator: the SWAT facilitator is responsible for managing the rotation process. [[User:Stephen_K._Jolley | Stephen K. Jolley]] is the current SWAT Facilitator.
When filing bugs, always search Bugzilla first to see if the issue is already known.  For example, there are some bugs that occur intermittently and are already filed with ''AB-INT'' in the whiteboard field. They are listed here: [https://bugzilla.yoctoproject.org/buglist.cgi?quicksearch=whiteboard%3AAB-INT&list_id=640327 Autobuilder issues]
 
The exact progress depends on whether the branch is an official branch or a staging branch.
 
==== Staging Branches ====
 
For builds against staging branches which contain patches under test for integration (such as master-next, stable/dunfell-nut, ross/mut, etc), first attempt to identify if there is a patch in the branch that is likely to be responsible for the failure. For example, if <tt>wget</tt> fails with <tt>libgnutls</tt> errors and there is a GnuTLS upgrade in the branch, then that is a likely candidate.  If a patch can be identified that hasn't yet been merged into an official branch, then reply to the patch on the mailing list with the details. If it isn't obvious which patch is responsible for the failure, or a patch can be identified but it has already been merged to the release branch, then file a bug and ensure the branch maintainer (see the [[Releases]] page for names) is on the CC list.
 
Most of the failures will be for staging branches as master-next is the branch that is tested the most. However, it is rebased quite frequently so it is not always easy to find which patchs were included. In that case, you have to get the actual commit hash, for example in the build properties, the variable is <tt>yp_build_revision</tt> or in the build configuration at the beginning of the stdio log. For example, this qemux86 build [https://autobuilder.yoctoproject.org/typhoon/#/builders/59/builds/3120] was master-next at revision 47482eff9897ccde946e9247724babc3a586d318.
With that information, you can then clone poky (or any other layer of interest) and fetch the proper commit and see the git log:
 
<pre>
$ git clone git://git.yoctoproject.org/poky
$ cd poky
$ git fetch origin 47482eff9897ccde946e9247724babc3a586d318
$ git log FETCH_HEAD
</pre>


== Process ==


The [[BuildLog]] wiki page is automatically updated by the autobuilder when a new build is triggered. This often includes guidance for SWAT (the "Reason" field when triggering the build) such as to ignore it (if the build is expected to break and the owner is monitoring themselves) or a request to report failures straight to a specific person (if the build only contains changes from a single person, such as a GCC upgrade). Unless when told otherwise, the usual process is as follows:
'''If in doubt, file a bug'''. All errors must be taken care of.


For builds against master or a release branch, any issues observed should be [[#Filing_bugs | filed in Bugzilla]].
If the issue is in the infrastructure or autobuilder itself then file a bug against "Infrastructure: Autobuilder", infrastructure bugs should be assigned to [[User:Halstead| Michael Halstead]] and autobuilder logic bugs to [[User:Rpurdie | Richard Purdie]].


For builds against other branches (master-next, ross/mut, -next branches for stable releases, etc.), attempt to identify what patch in the branch is likely responsible for the failure. For example, if <tt>wget</tt> fails with <tt>libgnutls</tt> errors and there is a GnuTLS upgrade in the branch, that is the likely candidate.  If a patch can be identified that isn't also in master or release branch, reply on the mailing list with the failure details.  If it isn't obvious which patch is responsible for the failure then [[#Filing_bugs | file a bug]] and ensure the branch owner is either the assignee or on the CC list.
==== Official Branches ====


'''If in doubt, file a bug'''. All observed errors must be actioned unless a patch has already been sent for the issue, in which case please make note of this in the [[BuildLog]].
For builds of official branches, that is master or a release branch, '''all failures or warnings are critical''' and must be [[#Filing_bugs | filed in Bugzilla]]. Remember to check that the issue isn't already filed. Where an issue is already filed, please do add a comment so we can assess how frequently different issues are occurring.


If the issue is in the infrastructure or autobuilder itself then file a bug against , infrastructure bugs should be assigned to [[User:Halstead| Michael Halstead]] and autobuilder logic bugs to [[User:Rpurdie | Richard Purdie]].
=== Update ===


The results of pre-triage for an issue should be added to the corresponding entry in the [[BuildLog]], including a link to the resolution (patch name, bug link, etc) and a brief summary of the issue.  Every issue should be added to the build log so it acts as a build status report.
Finally the swatbot build entry must be updated with a summary of the outcome.  Three different resolutions are available:
* Mail sent: used when you replied to the problematic patch directly on the mailing list.
* Bug Opened: used when a new bug has been opened or a new comment has been added to a bug. Please add the bug number in the notes.
* Handled (other): used when the maintainer is already aware of the issue and is working on it resolution or a patch has already been sent to solve the issue.


Note that some builds, in particular the "perf" builds are not listed on [[BuildLog]] unless the build fails (to try and reduce noise on the log). Failures in performance test builds should be handled like any other build.
You need an account for this step. If it hasn't been provided, please ask on the swat mailling list.


The net result is all failures listed in [[BuildLog]] should have outcomes listed against them from the person on SWAT at the time.
'''Every issue that is dealt with must be annotated''', so it is easy to see which issues have been handled.  This includes filing new bugs, finding existing bugs, contacting the mailing list, contacting the maintainer directly on IRC, or identifying that a patch has already been sent to fix the issue.


=== Filing bugs ===
== Tips ==


When filing the bug two things are required:
An issue will quite often repeat itself across multiple builds. It is worth looking for those repetitions as swatbot will allow you to select many builds and update them all at once.
* A link to the build failure.  This can either be the build page (such as https://autobuilder.yoctoproject.org/typhoon/#/builders/34/builds/168) or the [http://errors.yoctoproject.org/Errors/ error reports] page (such as http://errors.yoctoproject.org/Errors/Details/199667/).
* The error itself. Trim the log down to just the error and any relevant context in the bug description, and also attach the complete log file.  Autobuilder logs are not persistent so the full log must be attached.


=== Process summary ===
In the stdio logs window, clicking on the looking glass icon will load the full log in the browser, allowing you to use the search feature of the browser. Quite often, you'll start by looking for "error:".


* Monitor builds via one or more of:
Be sure to check all log files, especially the testimage logs that are available for qemu_boot_log (for example in: /tmp/work/qemux86_64-poky-linux/core-image-sato-sdk/1.0-r0/testimage/qemu_boot_log.20210902120413)
** the autobuilder: [https://autobuilder.yoctoproject.org/typhoon/ main page], [https://autobuilder.yoctoproject.org/typhoon/#/console console view]. Orange builds have warnings, red builds failed with errors.
** the [http://errors.yoctoproject.org/Errors/ Error Reports] web page which has an entry for each recipe that fails. A link to the appropriate search is on the autobuilder's Console View.
** the [[BuildLog]] wiki page.  This has links to each build log that wasn't successful.
** the [https://lists.yoctoproject.org/listinfo/yocto-builds yocto-builds] mailing list. This gets a mail for every unsuccessful build.
* Pre-triage each failure:
** File a bug ''or'' respond to a patch ''or'' note known issues
** Update the [[BuildLog]] with the result of pre-triage, linking to the resolution where possible.


== Questions / Contact ==
== Handoff ==


If you have queries about the SWAT process you may reach out to the SWAT Facilitator [[User:Stephen_K._Jolley | Stephen K. Jolley]] or the SWAT Chair [[User:RossBurton | Ross Burton]].
At the end of the week, the outgoing person on Swat should email swat@lists.yoctoproject.org summarising the week and noting anything that the incoming person on Swat next week should be aware of. For example, noting that there's a new intermittent bug to watch for.


== Members ==
== Members ==


[[User:RossBurton | Ross Burton]]
* [[User:RossBurton | Ross Burton]]
 
* [[User:Leonardo_Sandoval | Leo Sandoval]]
 
* [[User:Anibal Limon | Anibal Limon]]
 
* [[User:Köry maincent | Köry Maincent]]
 
* [[User:Thomas Perrot | Thomas Perrot]]
 
* [[User:SaulWold | Saul Wold]]
 
* [[User:Oleksiy Obitotskyy | Oleksiy Obitotskyy]]
 
* [[User:Alejandro Enedino Hernandez Samaniego | Alejandro Hernandez Samaniego]]


[[User:Leonardo_Sandoval | Leo Sandoval]]
* [[User:PaulEggleton | Paul Eggleton]]


[[User:SaulWold | Saul Wold]]
* [[User:Naveen Kumar Saini | Naveen Saini]]


[[User:Anibal | Anibal Limon]]
* [[User:Alexandre Belloni | Alexandre Belloni]]


[[User:Alejandro Enedino Hernandez Samaniego | Alejandro Hernandez Samaniego]]
* [[User:Kergoth | Christopher Larson]]


[[User:PaulEggleton | Paul Eggleton]]
* [[User:Lee_chee_yang | Lee Chee Yang]]


[[User:Naveen Kumar Saini | Naveen Saini]]
* [[User:Jon Mason | Jon Mason]]


Armin Kuster (place me anywhere)
* [[User:Minjae Kim | Minjae Kim]]


[[User:Kergoth | Christopher Larson]]
* [[User:Jagadheesan | Jaga]]


Lee Chee Yang
* [[User:Valerii Chernous | Valerii Chernous]]

Latest revision as of 10:35, 5 October 2021

Overview

All builds that are run on the public autobuilder are important for the Yocto Project, whether they be routine validation runs or pre-integration test builds. Random failures if ignored accumulate and can result in a significant number of builds failing.

The role of the Bug Swat Team is to monitor the autobuilder and do preliminary investigation of failures, to ensure that they are logged and brought to the attention of the appropriate owner.

Importantly, the Swat Team isn't responsible for resolving issues encountered on the autobuilder, simply just enough analysis so that it can be logged for later analysis and ideally make the right people aware of them.

Each week a different member of the team is on call. Every build that fails on the autobuilder should be monitored unless stated otherwise. The rotation happens at the end of Friday (deliberately vague), any failures over the weekend should be triaged by the incoming member on Monday.

The Swat Chairs are the primary contact for the Swat Team. The current Swat Chairs are Ross Burton and Richard Purdie. The Chairs are assisted by Stephen K. Jolley who handles the rotation process. If the person currently on call, or about to be on call, can no longer perform their duty then they should contact Stephen to arrange a replacement.

Process

The SWAT process is now using a specific tool, swatbot. Swatbot has a filter which will list all the pending issues that need to be triaged using this link or the "SWAT Pending Builds" link on the left hand menu. Each issue has links to the autobuilder logs for the failing step (e.g. usually stdio and warning/error logs).

The builds are shown in a tree like structure with the parent build and any child builds under it. The builds are edited as a group under the parent as quite often a failure might be common to the child builds. Each failure does need to be triaged individually although multiple builds can be changed at once to a given resolution.

Swatbot can filter pending issues to be triaged using the SWAT Pending Builds link. Once you have selected an issue to triage, you will have to take the correct reporting action and finally edit the entry to indicate what was done.

You can also be notified when a build fails by subscribing to the yocto-builds mailing list. This is sending a mail when a build fails, including direct links to the autobuilder job summary and the Error Reporting Service. The mail will also state if it is expected that the build is triaged by Swat, so check this to see if the build can be ignored as the owner is taking full responsibility. Currently, swatbot will not give you this information so you have to get it from the autobuilder build entry (the Build properties tab should have: swat_monitor true), the autobuilder API, or the notification email.

Another tool that can be used to monitor builds is the Autobuilder 'Yocto Console View' which is an overview of the top-level builds (a-full and a-quick) and the sub-builds they trigger.

Both the top-level build entry and the mail notification will include notes from the build owner, so check this for any useful context. For example, it may request that failures are reported directly to a specific person instead of bugs created, or that particular failures that are expected.

Report

There are two categories of builds that Swat will be monitoring: official branches and staging branches. The official branches are the primary top-level branches in Poky, that is master and all of the release branches (gatesgarth, dunfell, etc). The staging branches are where patches are held for testing, such as master-next, stable/dunfell-nut, or ross/mut.

Communication is important: if the build owner is on IRC then it's always worth discussing issues with them first as they may have further context and directions. Also, if the build owner triages the build failures then they must update the swatbot entries so that Swat doesn't duplicate the work.

When reporting an issue, be it in a mailing list post or a new bug, the following information should be included:

When filing bugs, always search Bugzilla first to see if the issue is already known. For example, there are some bugs that occur intermittently and are already filed with AB-INT in the whiteboard field. They are listed here: Autobuilder issues

The exact progress depends on whether the branch is an official branch or a staging branch.

Staging Branches

For builds against staging branches which contain patches under test for integration (such as master-next, stable/dunfell-nut, ross/mut, etc), first attempt to identify if there is a patch in the branch that is likely to be responsible for the failure. For example, if wget fails with libgnutls errors and there is a GnuTLS upgrade in the branch, then that is a likely candidate. If a patch can be identified that hasn't yet been merged into an official branch, then reply to the patch on the mailing list with the details. If it isn't obvious which patch is responsible for the failure, or a patch can be identified but it has already been merged to the release branch, then file a bug and ensure the branch maintainer (see the Releases page for names) is on the CC list.

Most of the failures will be for staging branches as master-next is the branch that is tested the most. However, it is rebased quite frequently so it is not always easy to find which patchs were included. In that case, you have to get the actual commit hash, for example in the build properties, the variable is yp_build_revision or in the build configuration at the beginning of the stdio log. For example, this qemux86 build [1] was master-next at revision 47482eff9897ccde946e9247724babc3a586d318. With that information, you can then clone poky (or any other layer of interest) and fetch the proper commit and see the git log:

$ git clone git://git.yoctoproject.org/poky
$ cd poky
$ git fetch origin 47482eff9897ccde946e9247724babc3a586d318
$ git log FETCH_HEAD


If in doubt, file a bug. All errors must be taken care of.

If the issue is in the infrastructure or autobuilder itself then file a bug against "Infrastructure: Autobuilder", infrastructure bugs should be assigned to Michael Halstead and autobuilder logic bugs to Richard Purdie.

Official Branches

For builds of official branches, that is master or a release branch, all failures or warnings are critical and must be filed in Bugzilla. Remember to check that the issue isn't already filed. Where an issue is already filed, please do add a comment so we can assess how frequently different issues are occurring.

Update

Finally the swatbot build entry must be updated with a summary of the outcome. Three different resolutions are available:

  • Mail sent: used when you replied to the problematic patch directly on the mailing list.
  • Bug Opened: used when a new bug has been opened or a new comment has been added to a bug. Please add the bug number in the notes.
  • Handled (other): used when the maintainer is already aware of the issue and is working on it resolution or a patch has already been sent to solve the issue.

You need an account for this step. If it hasn't been provided, please ask on the swat mailling list.

Every issue that is dealt with must be annotated, so it is easy to see which issues have been handled. This includes filing new bugs, finding existing bugs, contacting the mailing list, contacting the maintainer directly on IRC, or identifying that a patch has already been sent to fix the issue.

Tips

An issue will quite often repeat itself across multiple builds. It is worth looking for those repetitions as swatbot will allow you to select many builds and update them all at once.

In the stdio logs window, clicking on the looking glass icon will load the full log in the browser, allowing you to use the search feature of the browser. Quite often, you'll start by looking for "error:".

Be sure to check all log files, especially the testimage logs that are available for qemu_boot_log (for example in: /tmp/work/qemux86_64-poky-linux/core-image-sato-sdk/1.0-r0/testimage/qemu_boot_log.20210902120413)

Handoff

At the end of the week, the outgoing person on Swat should email swat@lists.yoctoproject.org summarising the week and noting anything that the incoming person on Swat next week should be aware of. For example, noting that there's a new intermittent bug to watch for.

Members