Build testing using Autotest
DRAFT DRAFT DRAFT
What is Autotest ?
Autotest is a framework for fully automated testing. It is designed primarily to test the Linux kernel, though it can serve many other purposes such as qualifying new hardware, virtualization testing and other general user space program testing under linux platforms.
If you want to know more about Autotest itself see the Autotest wiki. [Fedora's AutoQA] is one example of a project that uses autotest as their test automation framework.
Because of the increasing need to automate our build tests the Yocto QA team decided to use Autotest. Currently there are 12 test cases that we run using Autotest.
Before using any of the test cases or writing new tests you must understand how things work, and in order to do that you should read the Autotest documentation:
https://github.com/autotest/autotest/wiki/ClientQuickStart
https://github.com/autotest/autotest/wiki/ControlRequirements
https://github.com/autotest/autotest/wiki/ControlHowto
https://github.com/autotest/autotest/wiki/AutotestApi
https://github.com/autotest/autotest/wiki/ResultsSpecification
How to run the tests using the standalone autotest client on your machine
Types of tests and paths
If you have read the autotest docs you should know that there are more types of tests:
- client-side control files, which can be
- manually deployed and run (what this section is about)
- deployed by autotest server and run automatically on client machines (next section)
- server-side control files (when jobs need to run stuff both on the server and multiple autotest clients) - we don't care about these.
Have a look here https://github.com/autotest/autotest/wiki/DirectoryStructure to learn more about the directory structure of Autotest.
Get Autotest
- clone the autotest repo
$ git clone https://github.com/autotest/autotest $ cd autotest/client/
- get our tests (don't mind the path, just clone it to tmp/site_tests)
$ git clone -b stefans/tests git://git.yoctoproject.org/poky-contrib tmp/site_tests
Ok, so now you should have autotest installed and our tests in ~/autotest/client/tmp/site_tests.
You'll see a bunch of control.* files and a heater.py if you "ls -l ~/autotest/client/tmp/site_tests/heater"
heater.py it's the actual test code (so we only wrote one test which we reuse in different ways by running different control files).
Let's see the available tests
$ cd ~/autotest/client $ ./autotest list Remotely fetched tests (/home/stefans/autotest/client/tmp/site_tests) [Control] [Description] heater/control.172-sstate-sato-satosdk 172 - Build core-image-sato and core-image-sato-sdk heater/control.292-lib64lsbsdk 292 - Build lib64-core-image-lsb-sdk heater/control.minimal Build core-image-minimal with a clean builddir heater/control.280-lib32connman 280 - Build core-image-sato with multilib and lib32-connman-gnome installed heater/control.169-remake 169 - Build remake and remake-native heater/control.290-lib64sato 290 - Build lib64-core-image-sato heater/control.291-lib64satosdk 291 - Build lib64-core-image-sato-sdk heater/control.311-buildapp 322 - Bitbake build-appliance-image heater/control.options This is just a dumb control file used for global options heater/control.296-perf 296 - Build perf recipe heater/control.281x-minx32 almost 281 - Build core-image-minimal with x32 tune heater/control.325-emgd 325 - Build emgd-driver-bin #let's say we want to build core-image-minimal (aka bitbake core-image-minimal) $ ./autotest --verbose run heater/control.minimal #or another one : bitbake core-image-minimal with x32 tune $ ./autotest --verbose run heater/control.281x-minx32
You'll start to see a lot of output from Autotest and the tests themselves (remove --verbose for less noise but you'll loose bitbake's output too (it's logged anyway))
Here is what autotest shows for heater/control.minimal:
[stefans@firebird client]$ ./autotest run heater/control.minimal
18:14:52 INFO | Writing results to /home/stefans/autotest/client/results/default
18:14:52 INFO | Could not symlink init scripts (lack of permissions)
18:14:52 INFO | Not logging /proc/slabinfo (lack of permissions)
18:14:52 INFO | START ---- ---- timestamp=1362413692 localtime=Mar 04 18:14:52
18:14:52 INFO | START heater.minimal heater.minimal timestamp=1362413692 localtime=Mar 04 18:14:52
18:14:52 INFO | Not logging /proc/slabinfo (lack of permissions)
18:14:52 INFO | Removing clone directory if it exists...
18:14:52 INFO | Getting repo...
18:14:52 INFO | Fetching git [REP 'git://git.yoctoproject.org/poky' BRANCH 'master'] -> /home/stefans/autotest/client/tmp/heater/src/poky
18:15:08 INFO | git commit ID is 93ec7b4d1550e07caec86e2998d0f94a01c7e785 (tag 1.4_M4.rc1-142-g93ec7b4)
18:15:08 INFO | The repo dir used is: /home/stefans/autotest/client/tmp/heater/src/poky
18:15:08 INFO | The build dir used is: /home/stefans/autotest/client/tmp/heater/src/build
18:15:08 INFO | Removing build directory if it exists...
18:15:08 INFO | Config file is:
BB_NUMBER_THREADS = "8"
PARALLEL_MAKE = "-j 8"
MACHINE ??= "qemux86-64"
DISTRO ?= "poky"
PACKAGE_CLASSES ?= "package_rpm"
EXTRA_IMAGE_FEATURES = "debug-tweaks"
USER_CLASSES ?= "buildstats image-mklibs image-prelink"
PATCHRESOLVE = "noop"
CONF_VERSION = "1"
BB_DISKMON_DIRS = "\
STOPTASKS,${TMPDIR},1G,100K \
STOPTASKS,${DL_DIR},1G,100K \
STOPTASKS,${SSTATE_DIR},1G,100K \
ABORT,${TMPDIR},100M,1K \
ABORT,${DL_DIR},100M,1K \
ABORT,${SSTATE_DIR},100M,1K"
SSTATE_DIR ?= "/data/yocto/sstate-cache"
DL_DIR ?= "/data/yocto/downloads"
18:15:08 INFO | Command issued is: source ./oe-init-build-env /home/stefans/autotest/client/tmp/heater/src/build; export CLONEDIR=/home/stefans/autotest/client/tmp/heater/src/poky; bitbake core-image-minimal
18:46:52 INFO | Command returned zero exit status... That's GOOD!
18:46:52 INFO | Not logging /proc/slabinfo (lack of permissions)
18:46:52 INFO | Not logging /var/log/messages (lack of permissions)
18:46:52 INFO | GOOD heater.minimal heater.minimal timestamp=1362415612 localtime=Mar 04 18:46:52 completed successfully
18:46:52 INFO | END GOOD heater.minimal heater.minimal timestamp=1362415612 localtime=Mar 04 18:46:52
18:46:52 INFO | END GOOD ---- ---- timestamp=1362415612 localtime=Mar 04 18:46:52
How to write a new test aka control file
What's a control file?
The key defining component of a job is its control file; this file defines all aspects of a jobs life cycle. The control file is a python script which directly drives execution of the tests in question.
Autotest supplies you with a job object which drives the job and supplies services to the control file. A control file can be as simple as: job.run_test('test')
Control files should always start with control.XXXXX. XXXXX is up to you, but we use something like control.<testopia-case-id>-<short descriptive name>
Stuff to know
- heater.py is the actual code
- control files call heater with different args
- heater.py will take care of cloning poky and setting up local.conf and running a build
- all tests have a default location for the git clone and build dir (which is shared among tests).
- the clone dir and build dir can be overwritten by the control files themselves with the right args, but we usually don't want to do that.
- there is a default local.conf used which can pe appended
In the future there will be more classes which can be used for different tests but for the most basic ones what we have now should suffice.
Examples
One of the simplest example is control.minimal which does a bitbake core-image-minimal with a clean build dir.
[stefans@firebird client]$ cat tmp/site_tests/heater/control.minimal
AUTHOR = "Stefan Stanacar"
NAME = "Build core-image-minimal with a clean builddir"
TIME = "MEDIUM"
TEST_CATEGORY = "Functional"
TEST_CLASS = "General"
TEST_TYPE = "client"
DOC = """
Build core-image-minimal
"""
job.run_test('heater', tag='minimal', cmd='bitbake core-image-minimal', clean=True)
Notes:
- AUTHOR, NAME, TIME, TEST_* and DOC are mandatory in a control file (autotest requirement, see https://github.com/autotest/autotest/wiki/ControlRequirements )
- the rest is just python code (and in our case a call to run_test('heater') with some args)
What happens behind the scene on my machine (step by step):
- the clone dir used is: /home/stefans/autotest/client/tmp/heater/src/poky
- the build dir used is: /home/stefans/autotest/client/tmp/heater/src/build
- the clone dir is removed if it exists (because of clean=True)
- git init/fetch/checkout master for git://git.yoctoproject.org/poky (because no branch or commit args were passed to the call it points to master head)
- the build dir is removed if it exists (becasue of clean=True)
- a local.conf will be created from a default one (hardcoded in heater.py) which can pe appended or overwritten (not the case here)
- from inside the clone dir a command is passed to the shell: source ./oe-init-build-env /home/stefans/autotest/client/tmp/heater/src/build; export CLONEDIR=/home/stefans/autotest/client/tmp/heater/src/poky; bitbake core-image-minimal
Another example.
Let's say we want to automate this test case https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=325
In this case we would write something like this:
[stefans@firebird ~]$ cd ~/autotest/client/tmp/site_tests/heater/
[stefans@firebird heater]$ cat control.325-emgd
AUTHOR = "Stefan Stanacar"
NAME = "325 - Build emgd-driver-bin"
TIME = "MEDIUM"
TEST_CATEGORY = "Functional"
TEST_CLASS = "General"
TEST_TYPE = "client"
DOC = """
Set LICENSE_FLAGS_WHITELIST, add meta-intel layer and build emgd-driver-bin for crownbay
"""
appendconf ="""LICENSE_FLAGS_WHITELIST = "license_emgd-driver-bin_1.14"
MACHINE = "crownbay"
"""
if job.run_test('heater', tag='emgd01', gitsrc="git://git.yoctoproject.org/meta-intel", meta=True):
job.run_test('heater', tag='emgd02', cmd='bitbake emgd-driver-bin', appendconf=appendconf, addlayers="meta-intel meta-intel/meta-crownbay")
Steps:
- git init or reset /fetch/checkout master for git://git.yoctoproject.org/meta-intel (because no branch or commit args were passed to the call it points to master head)
- a local.conf will be created from a default one (hardcoded in heater.py) and appendconf is appended to it
..... TBD
Installing Autotest server with Web frontend
Intro
Tests can be run on standalone machines, or in a server-client mode. While running the standalone autotest client it's a convient way to run a single test the machine where you've clone autotest, it's not efficient. You might want to run multiple instances of autotest running jobs consisting of many tests. (a pool of vms running different distros for example)
The server interaction is simple:
- Copy the control file across
- Execute the control file repeatedly until it completes
- Client notifies server of any reboot for monitoring
- Upon completion of control script, server pulls results back (not client push)
The server consists of three main parts:
- A mysql database that holds information on all jobs, clients (test machines), users and tests.
- The dispatcher (monitor_db) - chooses the jobs from the database to run. It's input is the database, pretty much all it does is start autoserv processes to service requests.
- There is normally one dispatcher process per machine
- Client side jobs are run asyncronously (as client machines become available)
- Server side jobs are run syncronously
- Autoserv: the server manages clients via autoserv processes - there will be one autoserv process per running job?. Each autoserv process:
- controls and monitors one or more clients
- verifies clients are working properly, and if it fails verification, attempts to repair it
- manages the execution of a job
- updates the autotest software on each client before commencing work.
Installing
On https://github.com/autotest/autotest/wiki/SysAdmin you can find documentation on how to setup an autotest server infrastructure on your distro. They provide a script for Fedora/Ubuntu which automates the installation. However I recommend you follow the manual steps to get a better understanding of what's happening behind the scenes.
Here is a step by step install on my Fedora 18 machine, which replicates pretty close the docs (or script) from the autotest wiki with some small changes.
Installing autotest server on a Fedora 18 host: Note: currently autotest has a lot of problems if SELinux is enabled and enforcing (so disable it before anything else, sorry) # yum install git make wget python-devel unzip # yum install httpd mod_wsgi mysql-server MySQL-python gnuplot python-crypto python-paramiko java-1.7.0-openjdk-devel python-httplib2 # yum install numpy python-matplotlib libpng-devel freetype-devel python-imaging - enable and start MySQL # systemctl enable mysqld.service # systemctl start mysqld.service - set a MySQL pass if you haven't done it yet # mysqladmin -u root password MYSQL_ROOT_PASS - clone autotest in /usr/local/autotest (if you want to change that be prepared to edit the apache config file and others) # cd /usr/local # git clone git://github.com/autotest/autotest.git - set apache configs # ln -s /usr/local/autotest/apache/conf /etc/httpd/autotest.d # ln -s /usr/local/autotest/apache/apache-conf /etc/httpd/conf.d/z_autotest.conf # ln -s /usr/local/autotest/apache/apache-web-conf /etc/httpd/conf.d/z_autotest-web.conf - create a user for autotest install (you'll want to use /usr/local/autotest as home dir, otherwise you'll have edit a lot of config files) # useradd -d /usr/local/autotest autotest # passwd autotest # chown -R autotest:autotest /usr/local/autotest # su - autotest - edit ~autotest/.bashrc and add your export {http,ftp,http}_proxy=http://proxy/ there if you need a proxy - edit ~autotest/.gitconfig and add gitproxy config fragment if you need a proxy - set autotest db (replace MYSQL_ROOT_PASS with the pasword you set earlier and MYSQL_AUTOTEST_PASS wit a passor for the autotest db $ ./installation_support/autotest-database-turnkey -s --root-password MYSQL_ROOT_PASS -p MYSQL_AUTOTEST_PASS - as autotest user $ ./utils/build_externals.py - compile frontend $ ./utils/compile_gwt_clients.py -a - Now you need to edit global_config.ini and set the password usef for autotest db (earlier MYSQL_AUTOTEST_PASS) - and set a path for autotest clients install (where autotest server will deploy it's code and tests on the machines/VMs) - Make sure that path it's on a large partition because that's where the builds take place $ vi global_config.ini --- password: please_set_this_password [scroll down] [AUTOSERV] # Autotest potential install paths client_autodir_paths: /data/autotest
$ exit # systemctl enable httpd.service # systemctl start httpd.service # cp /usr/local/autotest/utils/autotestd.service /etc/systemd/system # systemctl enable autotestd.service # systemctl start autotestd.service # su - autotest $ cd client - add tests and import them $ mv site_tests site_tests.orig $ git clone -b stefans/tests git://git.yoctoproject.org/poky-contrib site_tests $ ~/utils/test_importer.py Each time you want to update the tests you need to go to /usr/local/autotest/client/site_tests and do a git pull and then run /usr/local/autotest/utils/test_importer.py (so they show up in the web interface) - How to add clients:
