AutoBuilder Maintenance
Overview
An AutoBuilder system consists of a controller and one or more workers. The controller reads configuration files, presents a simple web user interface and sends build step commands to the worker(s). (Note: the terms "worker" and "build slave" both refer to the machine or process performing the actual build, and are used interchangeably.)
When making changes to the AutoBuilder, it is important to remember what the different parts of the system do and which machines need to be restarted after a change (controller or workers).
- The build step configuration files contain a list of build steps to carry out for a particular build artifact or test (e.g., nightly-x86.conf or nightly-oe-selftest.conf). Some build step configuration files trigger other builds - e.g., nightly.conf triggers all the other nightly builds. A build step configuration file can also have user interface components: poky repo and branch, save build artifacts, etc. A build step configuration file also contains a list of workers (or builders) that are able to build this specific config. For example, poky-eclipse-neon requires JDK 1.8, so the list of workers is limited to those with Java 1.8 installed. Or,the list of workers for the nightly-rpm-non-rpm config is limited to those that do not use rpm as the native package manager (e.g., Debian and Ubuntu). The working set of build step config files live in the buildstep-config directory.
- The yoctoAB.conf also lives in the buildstep-config directory and contains a list of build step configurations that should be included in the web pages and their order.
- The autobuilder.conf contains variables to specify directory paths, the number of parallel makes, the list of QA email addresses, etc. Some of (but not all) of these variables are also passed to the worker(s). What is passed on is determined by the CreateAutoConf build step. The autobuilder.conf file lives in the config directory.
- The actual build step implementation files are written in Python and live in the lib/python2.7/site-packages/autobuilder and lib/python2.7/site-packages/autobuilder/buildsteps directories. There is typically a one-to-one mapping between the build steps in the build step config files and an Python implementation file. Often, the implementation consists of sending a set of shell commands for the worker to execute.
- The core buildbot code for the controller lives in lib/python2.7/site-packages/buildbot-0.8.8-py2.7.egg. It rarely needs to be changed unless a major new feature is added.
- The core buildbot code for the worker lives in lib/python2.7/site-packages/buildbot_slave-0.8.8-py2.7.egg. It rarely needs to be changed unless a major new feature is added.
The various configuration and build step implementation files are read by the controller once on startup (startup of the controller process, not the machine as a whole) and cached. If any of the files change, the controller must be stopped and restarted:
- . ./yocto-autobuilder-setup
- ./yocto-stop-autobuilder controller
- ./yocto-start-autobuilder controller
Generally, the only time a worked needs to be stopped and restarted is when the underlying buildbot_slave code changes. The procedure is the same as above for the controller, except replace "controller" with "worker".
Example Changes
This section presents a cookbook approach to common tasks often required as part of AutoBuilder updates or maintenance.
Build Additional Image for an Existing Configuration
Let's say a particular build step config had a rule to build core-image-sato and core-image-minimal images:
- {'BuildImages': {'images': 'core-image-minimal core-image-sato'}},
and you want to also build core-image-weston. Simply add the weston image to the list:
- {'BuildImages': {'images': 'core-image-minimal core-image-sato core-image-weston'}},
To include the new image in the sanity tests, just add it to the list:
- {'RunSanityTests': {'images': 'core-image-minimal core-image-sato core-image-weston'}},
After all the changes are made, restart the controller.
Add an Additional Machine Configuration
We recently expanded the list of machines that wic images were built for. The original nightly-wic.conf file had rules to build qemux86_64 and genericx86_64 images. To build images for qemux86 and genericx86, new CreateAutoConf, BuildImages and CreateWicImages steps were added. For example:
- {'CreateAutoConf':{'machine':'qemux86', 'atextappend':'\nIMAGE_FSTYPES += " hddimg"\nMACHINE_FEATURES_append = " efi"\n'}},
- {'BuildImages':{'images':'syslinux syslinux-native parted-native gptfdisk-native dosfstools-native mtools-native'}},
- {'BuildImages':{'images':'core-image-sato'}},
- {'CreateWicImages':{'wic_img_type':'directdisk', 'target_img':'core-image-sato'}},
- {'CreateWicImages':{'wic_img_type':'directdisk-gpt', 'target_img':'core-image-sato'}},
- {'CreateWicImages':{'wic_img_type':'mkefidisk', 'target_img':'core-image-sato'}},
Normally, the extra 'atextappend' entry is not required for CreateAutoConf, but in this case it is needed for the efi image type creation. When you add new machine configurations, don't forget to add the new machine type to the list of artifacts to publish in the PublishArtifacts step.
Add a New Build Config
To add a new build to the collection of build products, create a new config file and copy to the buildset-config directory, update nightly.conf and yoctoAB.conf and restart the controller. For example, eclipse-plugin-neon was recently added to the build list. Since there are other eclipse plugins, it was easiest to copy an existing config file (i.e., eclipse-plugin-mars.conf) and change the names from "mars" to neon". The list of "builders" was changed to include only those workers with native Java 1.8 support. The yoctoAB.conf file (in the buildset-config directory) was updated to add eclipse-plugin-neon to the list, and in this case, since eclipse-poky-kepler was being deprecated, it was removed from the list. In the nightly.conf file, eclipse-plugin-neon was added to the lis tof repos and to the TriggeredBuild list and eclipse-poky-kepler was removed.
Pass a New Variable to the Workers
As mentioned above, only a subset of the variables in autobuilder.conf are actually passed down to the worker. To include a new variable in the list, first add it to autobuilder.conf (e.g., MY_NEW_VAR = "somevalue"). This makes it visible to the controller. Next edit the CreateAutoConf.py build step implementation file. In the start() function, add the new variable with a construct like:
- if os.environ.get('MY_NEW_VAR') is not None:
- fout = fout + 'MY_NEW_VAR = "' + os.environ.get('MY_NEW_VAR') + '"'
Restart the controller.
Update the List of QA Email Recipients
To change the list of people receiving the QA notification emails, edit the autobuilder.conf file and change the contents of the QA_MAIL_CC variable. Restart the controller.
Add a New User Control
When starting a build via the web page, you are presented with one or more controls to pass information to the controller. Most every build set config includes text boxes for the poky repo and branch selections. Other configs, like nightly.conf have quite a few controls to select if this is a release build, if QA emails should be sent, git repos and branches for other layers or repositories (e.g., meta-intel, oecore, bitbake, etc.). The user controls can be extended to allow additional selections. For example, let's say you want to add a new selection box with values of "True" and "False". The control will be named "mycontrol".
- Add a new entry to the "props" section of nightly.conf for the new control:
- {'mycontrol':{'prop_type':'ChoiceStringParameter',
- 'choices': ['False', 'True'],
- 'name': 'mycontrol',
- label':'<hr><h3> Should mycontrol be included as part of the image creation?:</h3>'}},
- Ensure that the new property gets copied to the properties for any triggered builds by including it in the list of TriggerBuilds entries section of the buildset config:
- {'TriggerBuilds': {'copy_properties': ['custom_mycontrol'],
- Get the value of the new custom control in build step implementation file optionally include it in the objects property list. In the class's start() function, include code snippet like this:
- try:
- self.control = self.getProperty('custom_mycontrol')
- except:
- self.control = False
- try:
Note that the "custom_" name prefix is required, as parts of the buildbot code reply on it.
If you want this new control to appear on the individual build config web pages, add it to the list of "props" entries (if present) or add a new prop array entry. E.g.:
- props: [{'mycontrol':{'prop_type':'ChoiceStringParameter',
- 'choices': ['False', 'True'],
- 'name': 'mycontrol',
- label':'<hr><h3> Should mycontrol be included as part of the image creation?:</h3>'}}]
Configuration Management
Any changes to the AutoBuilder code base should follow proper configuration management guidelines per the Yocto Project. Changes should first be made and tested on a local AutoBuilder instance (physical machine or VM), or during off hours on a staging or development AutoBuilder. After verification, patches should be posted to the mailing list for review[1] and pushed to the master yoctoautobuilder repo, then merged into the running instance on the production AutoBuilders. The Yocto Project hosts two AutoBuilder git repos: yocto-autobuilder and yocto-autobuilder-production. The former is the standard public repository which includes example configuration files. The later represents the actual running configuration of the Linux Foundation public AutoBuilders.
A typical workflow uses a copy of the yocto-autobuilder repo on a developer's computer.
- A developer makes changes and tests them on a local AutoBuilder instance.
- By convention, changes to build set config files are made to the instances in the buildset-config.controller directory, then later copied to the running instance in the buildset-config directory.
- For files in the buildset-config.controller directory, ensure the "builders" list has only 'example-worker'.
- When copied to the running instance, be sure to update the list of builders for the cluster, if appropriate.
- When satisfied, commit the changes to the local repo.
- A patch set is generated and posted to the Yocto Project mailing list <yocto@yoctoproject.org>.
- Once the patch is reviewed, approved and pushed to the master branch of yocto-autobuilder, the production AutoBuilders can be updated:
- When the AutoBuilder is idle, stop the controller
- Ensure you are on the master branch, then do a 'git pull' to grab the updates
- Switch to the yoctoio branch and do a 'git merge master' to bring in the updates
- Make any additional installation specific changes and commit them
- Push the updated yoctoio branch to the production instance ('git push prod yoctoio' [2]).
- Start the controller.
- If the change affected the buildbot_slave code, the login into each worker and restart it.
- Patches for the AutoBuilder should be sent to yocto@yoctoproject.org with "[yocto-autobuilder]" in the Subject line. Please CC: pidge@toganlabs.com and joshua.g.lock@linux.intel.com
- The original public AutoBuilder does not use the yoctio branch or the production repo to keep local changes under revision control. Instead, local changes are untracked.
