AutoBuilder Cluster Setup

While the instructions below are targeted at setting up a new AutoBuilder cluster, they are equally valid for an single machine with combined controller and worker, except you probably won't use a NAS for storage. The controller portion requires only 2 GB of local disk space; the worker requires 2+ TB to hold all the build temp files and git repos. If build artifacts and a local sstate mirror are included, additional worker space is required.

As root

• Install base OS with minimal target
• Install the requirements from https://docs.yoctoproject.org/brief-yoctoprojectqs/index.html#build-host-packages
  • (Note: if you plan to build the poky-eclipse-neon plugin, openjdk-headless 1.8 is required.)
• If you plan to run the sanity tests (which is typical for an AB), the following additional packages need to be installed:
  • GitPython for Python2 and Python3
  • zstd & lz4 (or liblz4-tool)
  • xsltproc
  • gnupg
  • vncserver (whichever one is provided by the distro is fine)
• Add pokybuild user (uid=6000, or for older clusters 1006)
• Add pokybuild user to "users" group (gid 100) (if "users" group does not exist, create it)
• Following the instructions from https://wiki.yoctoproject.org/wiki/How_to_enable_KVM_for_Poky_qemu, give /dev/kvm permissions to the pokybuild user.
• If a "kvm" group does not exist, add it to /etc/group and ensure /dev/kvm gid is set to "kvm" and is mode 660. Add pokybuild user to the "kvm" group.
• git clone --single-branch https://git.yoctoproject.org/git/poky /root/poky (in /root for historical reasons, should change in the future)
• If adding tap devs for sanity testing. Install https://downloads.yoctoproject.org/releases/yocto/yocto-{latest_release}/toolchain/x86_64/poky-glibc-x86_64-core-image-sato-core2-64-toolchain-{relver}.sh to /opt/poky.
• Add startup scripts to create tap devs and start vncserver, in /etc/rc.local, /etc/init.d/boot.local, or as systemd unit files, which should perform the following:
  • cd /root/poky/scripts/; ./runqemu-gen-tapdevs 6000 100 32 /opt/poky/sysroots/x86_64-pokysdk-linux/
  • /bin/su - pokybuild -c "/usr/bin/vncserver"
  • Note that the location of the buildtools might actually be different than the example shown here, depending on the default path, etc. If you run into a tunctl error when generating the tapdevs, it's likely a path problem.
  • You will need to run vncserver once as the pokybuild user, and set the password. The password of "builder" is generally used. Sanity tests require vncserver to listen on port 5901, the default.
• Disable or edit firewall rules to allow all connections to the tap devices. This allows sanity tests to connect to booted images.
• Install nfs-utils or nfs-common, add Domain to /etc/idmapd.conf, fstab entry, mount NAS under /srv/autobuilder.
• We use more watches then the defaults generally allow so, echo "fs.inotify.max_user_watches=65536" >> /etc/sysctl.conf
• We open more files than most distros allow so adjust ulimits. Add to /etc/security/limits.conf
  • * hard nproc 515294
  • * soft nproc 514543
  • * soft nofile 131072
  • * hard nofile 131072
  • * hard memlock 16384
  • * soft memlock 16384

As pokybuild

• Set up buildset-config, either:
  • Add ssh key to clone production yocto-autobuilder
  • git clone https://git.yoctoproject.org/git/yocto-autobuilder && cd yocto-autobuilder && git remote add prod git@git.yoctoproject.org:yocto-autobuilder-production && git fetch prod && git checkout -b yoctoio prod/yoctoio
• or:
  • edit one of the several examples included in repo e.g. cp -r buildset-config.controller buildset-config

• configure git
  • git config --global gc.auto 0
  • git config --global user.name pokybuild
  • git config --global user.email pokybuild@$(hostname -f)
• update config/autobuilder.conf and yocto-worker/buildbot.tac
• Start the worker
  • touch ~/yocto-autobuilder/.setupdone
  • . yocto-autobuilder-setup
  • ./yocto-start-autobuilder worker