Build testing using Autotest

DRAFT DRAFT DRAFT

What is Autotest ?

Autotest is a framework for fully automated testing. It is designed primarily to test the Linux kernel, though it can serve many other purposes such as qualifying new hardware, virtualization testing and other general user space program testing under linux platforms.

If you want to know more about Autotest itself see the Autotest wiki. [Fedora's AutoQA] is one example of a project that uses autotest as their test automation framework.

Because of the increasing need to automate our build tests the Yocto QA team decided to use Autotest. Currently there are 12 test cases that we run using Autotest.

Before using any of the test cases or writing new tests you must understand how things work, and in order to do that you should read the Autotest documentation:

https://github.com/autotest/autotest/wiki/ClientQuickStart

https://github.com/autotest/autotest/wiki/ControlRequirements

https://github.com/autotest/autotest/wiki/ControlHowto

https://github.com/autotest/autotest/wiki/AutotestApi

https://github.com/autotest/autotest/wiki/ResultsSpecification

How to run the tests using the standalone autotest client on your machine

Types of tests and paths

If you have read the autotest docs you should know that there are more types of tests:

client-side control files, which can be
- manually deployed and run (what this section is about)
- deployed by autotest server and run automatically on client machines (next section)
server-side control files (when jobs need to run stuff both on the server and multiple autotest clients) - we don't care about these.

Have a look here https://github.com/autotest/autotest/wiki/DirectoryStructure to learn more about the directory structure of Autotest.

Get Autotest

clone the autotest repo

$ git clone https://github.com/autotest/autotest 
$ cd autotest/client/

get our tests (don't mind the path, just clone it to tmp/site_tests)

$ git clone -b stefans/tests git://git.yoctoproject.org/poky-contrib tmp/site_tests

Ok, so now you should have autotest installed and our tests in ~/autotest/client/tmp/site_tests.

You'll see a bunch of control.* files and a heater.py if you "ls -l ~/autotest/client/tmp/site_tests/heater"

heater.py it's the actual test code (so we only wrote one test which we reuse in different ways by running different control files).

Let's see the available tests:

$ cd ~/autotest/client
$ ./autotest list
Remotely fetched tests (/home/stefans/autotest/client/tmp/site_tests)
[Control]                                          [Description]
heater/control.172-sstate-sato-satosdk             172 - Build core-image-sato and core-image-sato-sdk
heater/control.292-lib64lsbsdk                     292 - Build lib64-core-image-lsb-sdk
heater/control.minimal                             Build core-image-minimal with a clean builddir
heater/control.280-lib32connman                    280 - Build core-image-sato with multilib and lib32-connman-gnome installed
heater/control.169-remake                          169 - Build remake and remake-native
heater/control.290-lib64sato                       290 - Build lib64-core-image-sato
heater/control.291-lib64satosdk                    291 - Build lib64-core-image-sato-sdk
heater/control.311-buildapp                        322 - Bitbake build-appliance-image
heater/control.options                             This is just a dumb control file used for global options
heater/control.296-perf                            296 - Build perf recipe
heater/control.281x-minx32                         almost 281 - Build core-image-minimal with x32 tune
heater/control.325-emgd                            325 - Build emgd-driver-bin

Here is what autotest shows for heater/control.minimal (add --verbose more noise and bitbake's output)

[stefans@firebird client]$ ./autotest run heater/control.minimal
18:14:52 INFO | Writing results to /home/stefans/autotest/client/results/default
18:14:52 INFO | Could not symlink init scripts (lack of permissions)
18:14:52 INFO | Not logging /proc/slabinfo (lack of permissions)
18:14:52 INFO | START	----	----	timestamp=1362413692	localtime=Mar 04 18:14:52	
18:14:52 INFO | 	START	heater.minimal	heater.minimal	timestamp=1362413692	localtime=Mar 04 18:14:52	
18:14:52 INFO | Not logging /proc/slabinfo (lack of permissions)
18:14:52 INFO | Removing clone directory if it exists...
18:14:52 INFO | Getting repo...
18:14:52 INFO | Fetching git [REP 'git://git.yoctoproject.org/poky' BRANCH 'master'] -> /home/stefans/autotest/client/tmp/heater/src/poky
18:15:08 INFO | git commit ID is 93ec7b4d1550e07caec86e2998d0f94a01c7e785 (tag 1.4_M4.rc1-142-g93ec7b4)
18:15:08 INFO | The repo dir used is: /home/stefans/autotest/client/tmp/heater/src/poky
18:15:08 INFO | The build dir used is: /home/stefans/autotest/client/tmp/heater/src/build
18:15:08 INFO | Removing build directory if it exists...
18:15:08 INFO | Config file is:
BB_NUMBER_THREADS = "8"
PARALLEL_MAKE = "-j 8"
MACHINE ??= "qemux86-64"
DISTRO ?= "poky"
PACKAGE_CLASSES ?= "package_rpm"
EXTRA_IMAGE_FEATURES = "debug-tweaks"
USER_CLASSES ?= "buildstats image-mklibs image-prelink"
PATCHRESOLVE = "noop"
CONF_VERSION = "1"
BB_DISKMON_DIRS = "\
            STOPTASKS,${TMPDIR},1G,100K \
            STOPTASKS,${DL_DIR},1G,100K \
            STOPTASKS,${SSTATE_DIR},1G,100K \
            ABORT,${TMPDIR},100M,1K \
            ABORT,${DL_DIR},100M,1K \
            ABORT,${SSTATE_DIR},100M,1K" 

SSTATE_DIR ?= "/data/yocto/sstate-cache"
DL_DIR ?= "/data/yocto/downloads"

18:15:08 INFO | Command issued is: source ./oe-init-build-env /home/stefans/autotest/client/tmp/heater/src/build; export CLONEDIR=/home/stefans/autotest/client/tmp/heater/src/poky; bitbake core-image-minimal
18:46:52 INFO | Command returned zero exit status... That's GOOD!
18:46:52 INFO | Not logging /proc/slabinfo (lack of permissions)
18:46:52 INFO | Not logging /var/log/messages (lack of permissions)
18:46:52 INFO | 		GOOD	heater.minimal	heater.minimal	timestamp=1362415612	localtime=Mar 04 18:46:52	completed successfully
18:46:52 INFO | 	END GOOD	heater.minimal	heater.minimal	timestamp=1362415612	localtime=Mar 04 18:46:52	
18:46:52 INFO | END GOOD	----	----	timestamp=1362415612	localtime=Mar 04 18:46:52

How to write a new test aka control file

What's a control file?

The key defining component of a job is its control file; this file defines all aspects of a jobs life cycle. The control file is a python script which directly drives execution of the tests in question.

Autotest supplies you with a job object which drives the job and supplies services to the control file. A control file can be as simple as: job.run_test('test')

Control files should always start with control.XXXXX. XXXXX is up to you, but we use something like control.<testopia-case-id>-<short descriptive name>

Stuff to know

heater.py is the actual code
control files call heater with different args
heater.py will take care of cloning poky and setting up local.conf and running a build
all tests have a default location for the git clone and build dir (which is shared among tests).
the clone dir and build dir can be overwritten by the control files themselves with the right args, but we usually don't want to do that.
there is a default local.conf used which can pe appended

In the future there will be more classes which can be used for different tests but for the most basic ones what we have now should suffice.

Examples

One of the simplest example is control.minimal which does a bitbake core-image-minimal with a clean build dir.

[stefans@firebird client]$ cat tmp/site_tests/heater/control.minimal 
AUTHOR = "Stefan Stanacar"
NAME = "Build core-image-minimal with a clean builddir"
TIME = "MEDIUM"
TEST_CATEGORY = "Functional"
TEST_CLASS = "General"
TEST_TYPE = "client"

DOC = """
Build core-image-minimal
"""

job.run_test('heater', tag='minimal', cmd='bitbake core-image-minimal', clean=True)

Notes:

AUTHOR, NAME, TIME, TEST_* and DOC are mandatory in a control file (autotest requirement, see https://github.com/autotest/autotest/wiki/ControlRequirements )
the rest is just python code (and in our case a call to run_test('heater') with some args)

What happens behind the scene on my machine (step by step):

the clone dir used is: /home/stefans/autotest/client/tmp/heater/src/poky
the build dir used is: /home/stefans/autotest/client/tmp/heater/src/build
the clone dir is removed if it exists (because of clean=True)
git init/fetch/checkout master for git://git.yoctoproject.org/poky (because no branch or commit args were passed to the call it points to master head)
the build dir is removed if it exists (becasue of clean=True)
a local.conf will be created from a default one (hardcoded in heater.py) which can pe appended or overwritten (not the case here)
from inside the clone dir a command is passed to the shell: source ./oe-init-build-env /home/stefans/autotest/client/tmp/heater/src/build; export CLONEDIR=/home/stefans/autotest/client/tmp/heater/src/poky; bitbake core-image-minimal

Another example.

Let's say we want to automate this test case https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=325

In this case we would write something like this:

[stefans@firebird ~]$ cd ~/autotest/client/tmp/site_tests/heater/
[stefans@firebird heater]$ cat control.325-emgd 
AUTHOR = "Stefan Stanacar"
NAME = "325 - Build emgd-driver-bin"
TIME = "MEDIUM"
TEST_CATEGORY = "Functional"
TEST_CLASS = "General"
TEST_TYPE = "client"

DOC = """
Set LICENSE_FLAGS_WHITELIST, add meta-intel layer and build emgd-driver-bin for crownbay
"""

appendconf ="""LICENSE_FLAGS_WHITELIST = "license_emgd-driver-bin_1.14"
MACHINE = "crownbay"
"""


if job.run_test('heater', tag='emgd01', gitsrc="git://git.yoctoproject.org/meta-intel", meta=True):
    job.run_test('heater', tag='emgd02', cmd='bitbake emgd-driver-bin', appendconf=appendconf, addlayers="meta-intel meta-intel/meta-crownbay")

Steps:

git init or reset /fetch/checkout master for git://git.yoctoproject.org/meta-intel (because no branch or commit args were passed to the call it points to master head)
a local.conf will be created from a default one (hardcoded in heater.py) and appendconf is appended to it

..... TBD

Installing Autotest server with Web frontend

Intro

Tests can be run on standalone machines, or in a server-client mode. While running the standalone autotest client it's a convient way to run a single test the machine where you've clone autotest, it's not efficient. You might want to run multiple instances of autotest running jobs consisting of many tests. (a pool of vms running different distros for example)

The server interaction is simple:

Copy the control file across
Execute the control file repeatedly until it completes
Client notifies server of any reboot for monitoring
Upon completion of control script, server pulls results back (not client push)

The server consists of three main parts:

A mysql database that holds information on all jobs, clients (test machines), users and tests.
The dispatcher (monitor_db) - chooses the jobs from the database to run. It's input is the database, pretty much all it does is start autoserv processes to service requests.
- There is normally one dispatcher process per machine
- Client side jobs are run asyncronously (as client machines become available)
- Server side jobs are run syncronously
Autoserv: the server manages clients via autoserv processes - there will be one autoserv process per running job?. Each autoserv process:
- controls and monitors one or more clients
- verifies clients are working properly, and if it fails verification, attempts to repair it
- manages the execution of a job
- updates the autotest software on each client before commencing work.

Installing and configuring

On https://github.com/autotest/autotest/wiki/SysAdmin you can find documentation on how to setup an autotest server infrastructure on your distro. They provide a script for Fedora/Ubuntu which automates the installation. However I recommend you follow the manual steps to get a better understanding of what's happening behind the scenes.

Here is a step by step install on my Fedora 18 machine, which replicates pretty close the docs (or script) from the autotest wiki with some small changes.


Installing autotest server on a Fedora 18 host:
Note: currently autotest has a lot of problems if SELinux is enabled and enforcing (so disable it before anything else, sorry)

# yum install git make wget python-devel unzip
# yum install httpd mod_wsgi mysql-server MySQL-python gnuplot python-crypto python-paramiko java-1.7.0-openjdk-devel python-httplib2
# yum install numpy python-matplotlib libpng-devel freetype-devel python-imaging

- enable and start MySQL
# systemctl enable mysqld.service
# systemctl start mysqld.service
- set a MySQL pass if you haven't done it yet
# mysqladmin -u root password MYSQL_ROOT_PASS

- clone autotest in /usr/local/autotest (if you want to change that be prepared to edit the apache config file and others)
# cd /usr/local
# git clone git://github.com/autotest/autotest.git

- set apache configs
# ln -s /usr/local/autotest/apache/conf /etc/httpd/autotest.d
# ln -s /usr/local/autotest/apache/apache-conf /etc/httpd/conf.d/z_autotest.conf
# ln -s /usr/local/autotest/apache/apache-web-conf /etc/httpd/conf.d/z_autotest-web.conf

- create a user for autotest install (you'll want to use /usr/local/autotest as home dir, otherwise you'll have edit a lot of config files)
# useradd -d /usr/local/autotest autotest
# passwd autotest
# chown -R autotest:autotest /usr/local/autotest
# su - autotest

- edit ~autotest/.bashrc and add your export {http,ftp,http}_proxy=http://proxy/ there if you need a proxy
- edit ~autotest/.gitconfig and add gitproxy config fragment if you need a proxy

- set autotest db (replace MYSQL_ROOT_PASS with the pasword you set earlier and MYSQL_AUTOTEST_PASS wit a passor for the autotest db
$ ./installation_support/autotest-database-turnkey -s --root-password MYSQL_ROOT_PASS -p MYSQL_AUTOTEST_PASS
- as autotest user
$ ./utils/build_externals.py
- compile frontend
$ ./utils/compile_gwt_clients.py -a

- Now you need to edit global_config.ini and set the password usef for autotest db (earlier MYSQL_AUTOTEST_PASS)
- and set a path for autotest clients install (where autotest server will deploy it's code and tests on the machines/VMs)
- Make sure that path it's on a large partition because that's where the builds take place
$ vi global_config.ini
---
password: please_set_this_password

[scroll down]

 [AUTOSERV]
 # Autotest potential install paths
client_autodir_paths: /data/autotest

----

$ exit
# systemctl enable httpd.service
# systemctl start httpd.service
# cp /usr/local/autotest/utils/autotestd.service /etc/systemd/system
# systemctl enable autotestd.service
# systemctl start autotestd.service

That's it, now go to http://localhost/afe and you should see a web interface.

Adding the tests

# su - autotest
$ cd client
$ mv site_tests site_tests.orig
$ git clone -b stefans/tests git://git.yoctoproject.org/poky-contrib site_tests
$ ~/utils/test_importer.py

Each time you want to update the tests you need to go to /usr/local/autotest/client/site_tests and do a git pull and then run /usr/local/autotest/utils/test_importer.py (so they show up in the web interface)

Add clients

Setup password-less ssh connection from the server to the client machine (which can be a VM running on the same host as the autotest server)

As autotest user, on the server, create a RSA key

ssh-keygen -t rsa

Then, still on the server, and as autotest, copy it to the host:

ssh-copy-id root@<client machine>

From the web interface go to http://localhost/afe/server/admin and add some hosts. Then they would show up when you create a job.

Build testing using Autotest

Contents

What is Autotest ?

How to run the tests using the standalone autotest client on your machine

How to write a new test aka control file

What's a control file?

Stuff to know

Examples

Installing Autotest server with Web frontend

Intro

Installing and configuring

Adding the tests

Add clients

Navigation menu

Page actions

Page actions

Personal tools

Navigation

Search

Tools