Build testing using Autotest

DRAFT DRAFT DRAFT

What is Autotest ?

Autotest is a framework for fully automated testing. It is designed primarily to test the Linux kernel, though it can serve many other purposes such as qualifying new hardware, virtualization testing and other general user space program testing under linux platforms.

If you want to know more about Autotest itself see the Autotest wiki

Because of the increasing need to automate our build tests the Yocto QA team decided to use Autotest. Currently there are 12 test cases that we run using Autotest.

Before using them and/or writing new tests you must read some Autotest documentation to understand how some things work:

https://github.com/autotest/autotest/wiki/ClientQuickStart

https://github.com/autotest/autotest/wiki/ControlRequirements

https://github.com/autotest/autotest/wiki/ControlHowto

https://github.com/autotest/autotest/wiki/AutotestApi

https://github.com/autotest/autotest/wiki/ResultsSpecification

How to run the tests using the standalone autotest client on your machine

clone the autotest repo
$ git clone https://github.com/autotest/autotest 
$ cd autotest/client/
get our tests
$ git clone -b stefans/tests git://git.yoctoproject.org/poky-contrib tmp/site_tests

Ok, so now you should have autotest installed and our tests in ~/autotest/client/tmp/site_tests.

You'll see a bunch of control.* files and a heater.py if you "ls -l ~/autotest/client/tmp/site_tests"

That's the actual test code (so we only wrote one test which we reuse in different ways).

Let's see the available tests

$ cd ~/autotest/client
$ ./autotest list
Remotely fetched tests (/home/stefans/autotest/client/tmp/site_tests)
[Control]                                          [Description]
heater/control.172-sstate-sato-satosdk             172 - Build core-image-sato and core-image-sato-sdk
heater/control.292-lib64lsbsdk                     292 - Build lib64-core-image-lsb-sdk
heater/control.minimal                             Build core-image-minimal with a clean builddir
heater/control.280-lib32connman                    280 - Build core-image-sato with multilib and lib32-connman-gnome installed
heater/control.169-remake                          169 - Build remake and remake-native
heater/control.290-lib64sato                       290 - Build lib64-core-image-sato
heater/control.291-lib64satosdk                    291 - Build lib64-core-image-sato-sdk
heater/control.311-buildapp                        322 - Bitbake build-appliance-image
heater/control.options                             This is just a dumb control file used for global options
heater/control.296-perf                            296 - Build perf recipe
heater/control.281x-minx32                         almost 281 - Build core-image-minimal with x32 tune
heater/control.325-emgd                            325 - Build emgd-driver-bin

let's say we want to build core-image-minimal (aka bitbake core-image-minimal)
$ ./autotest --verbose run heater/control.minimal
or another one : bitbake core-image-minimal with x32 tune
$ ./autotest --verbose run heater/control.281x-minx32

You'll start to see a lot of output from Autotest and the tests themselves (remove --verbose for less noise but you'll loose bitbake's output too (it's logged anyway))

Here's some output from ./autotest run heater/control.minimal

[stefans@firebird client]$ ./autotest run heater/control.minimal
18:14:52 INFO | Writing results to /home/stefans/autotest/client/results/default
18:14:52 INFO | Could not symlink init scripts (lack of permissions)
18:14:52 INFO | Not logging /proc/slabinfo (lack of permissions)
18:14:52 INFO | START	----	----	timestamp=1362413692	localtime=Mar 04 18:14:52	
18:14:52 INFO | 	START	heater.minimal	heater.minimal	timestamp=1362413692	localtime=Mar 04 18:14:52	
18:14:52 INFO | Not logging /proc/slabinfo (lack of permissions)
18:14:52 INFO | Removing clone directory if it exists...
18:14:52 INFO | Getting repo...
18:14:52 INFO | Fetching git [REP 'git://git.yoctoproject.org/poky' BRANCH 'master'] -> /home/stefans/autotest/client/tmp/heater/src/poky
18:15:08 INFO | git commit ID is 93ec7b4d1550e07caec86e2998d0f94a01c7e785 (tag 1.4_M4.rc1-142-g93ec7b4)
18:15:08 INFO | The repo dir used is: /home/stefans/autotest/client/tmp/heater/src/poky
18:15:08 INFO | The build dir used is: /home/stefans/autotest/client/tmp/heater/src/build
18:15:08 INFO | Removing build directory if it exists...
18:15:08 INFO | Config file is:
BB_NUMBER_THREADS = "8"
PARALLEL_MAKE = "-j 8"
MACHINE ??= "qemux86-64"
DISTRO ?= "poky"
PACKAGE_CLASSES ?= "package_rpm"
EXTRA_IMAGE_FEATURES = "debug-tweaks"
USER_CLASSES ?= "buildstats image-mklibs image-prelink"
PATCHRESOLVE = "noop"
CONF_VERSION = "1"
BB_DISKMON_DIRS = "\
            STOPTASKS,${TMPDIR},1G,100K \
            STOPTASKS,${DL_DIR},1G,100K \
            STOPTASKS,${SSTATE_DIR},1G,100K \
            ABORT,${TMPDIR},100M,1K \
            ABORT,${DL_DIR},100M,1K \
            ABORT,${SSTATE_DIR},100M,1K" 

SSTATE_DIR ?= "/data/yocto/sstate-cache"
DL_DIR ?= "/data/yocto/downloads"

18:15:08 INFO | Command issued is: source ./oe-init-build-env /home/stefans/autotest/client/tmp/heater/src/build; export CLONEDIR=/home/stefans/autotest/client/tmp/heater/src/poky; bitbake core-image-minimal
18:46:52 INFO | Command returned zero exit status... That's GOOD!
18:46:52 INFO | Not logging /proc/slabinfo (lack of permissions)
18:46:52 INFO | Not logging /var/log/messages (lack of permissions)
18:46:52 INFO | 		GOOD	heater.minimal	heater.minimal	timestamp=1362415612	localtime=Mar 04 18:46:52	completed successfully
18:46:52 INFO | 	END GOOD	heater.minimal	heater.minimal	timestamp=1362415612	localtime=Mar 04 18:46:52	
18:46:52 INFO | END GOOD	----	----	timestamp=1362415612	localtime=Mar 04 18:46:52	

How to write a new test aka control file

What's a control file?

The key defining component of a job is its control file; this file defines all aspects of a jobs life cycle. The control file is a python script which directly drives execution of the tests in question.

Autotest supplies you with a job object which drives the job and supplies services to the control file. A control file can be as simple as: job.run_test('test')

Control files should always start with control.XXXXX. XXXXX is up to you, but we use something like control.<testopia-case-id>-<short descriptive name>

Stuff to know

• heater.py is the actual code
• control files call heater with different args
• heater.py will take care of cloning poky and setting up local.conf and running a build
• all tests have a default location for the git clone and build dir (which is shared among tests).
• the clone dir and build dir can be overwritten by the control files themselves with the right args, but we usually don't want to do that.
• there is a default local.conf used which can pe appended

In the future there will be more classes which can be used for different tests but for the most basic ones what we have now should suffice.

Examples

One of the simplest example is control.minimal which does a bitbake core-image-minimal with a clean build dir.

[stefans@firebird client]$ cat tmp/site_tests/heater/control.minimal 
AUTHOR = "Stefan Stanacar"
NAME = "Build core-image-minimal with a clean builddir"
TIME = "MEDIUM"
TEST_CATEGORY = "Functional"
TEST_CLASS = "General"
TEST_TYPE = "client"

DOC = """
Build core-image-minimal
"""

job.run_test('heater', tag='minimal', cmd='bitbake core-image-minimal', clean=True)

Notes:

• AUTHOR, NAME, TIME, TEST_* and DOC are mandatory in a control file (autotest requirement, see https://github.com/autotest/autotest/wiki/ControlRequirements )
• the rest is just python code (and in our case a call to run_test('heater') with some args

What happens behind the scene on my machine (step by step):

• the clone dir used is: /home/stefans/autotest/client/tmp/heater/src/poky
• the build dir used is: /home/stefans/autotest/client/tmp/heater/src/build
• the clone dir is removed if it exists (because of clean=True)
• git init/fetch/checkout master for git://git.yoctoproject.org/poky (because no branch or commit args were passed to the call it points to master head)
• the build dir is removed if it exists (becasue of clean=True)
• a local.conf will be created from a default one (hardcoded in heater.py) which can pe appended or overwritten (not the case here)
• from inside the clone dir a command is passed to the shell: source ./oe-init-build-env /home/stefans/autotest/client/tmp/heater/src/build; export CLONEDIR=/home/stefans/autotest/client/tmp/heater/src/poky; bitbake core-image-minimal

Another example.

Let's say we want to automate this test case https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=325

In this case we would write something like this:

[stefans@firebird ~]$ cd ~/autotest/client/tmp/site_tests/heater/
[stefans@firebird heater]$ cat control.325-emgd 
AUTHOR = "Stefan Stanacar"
NAME = "325 - Build emgd-driver-bin"
TIME = "MEDIUM"
TEST_CATEGORY = "Functional"
TEST_CLASS = "General"
TEST_TYPE = "client"

DOC = """
Set LICENSE_FLAGS_WHITELIST, add meta-intel layer and build emgd-driver-bin for crownbay
"""

appendconf ="""LICENSE_FLAGS_WHITELIST = "license_emgd-driver-bin_1.14"
MACHINE = "crownbay"
"""


if job.run_test('heater', tag='emgd01', gitsrc="git://git.yoctoproject.org/meta-intel", meta=True):
    job.run_test('heater', tag='emgd02', cmd='bitbake emgd-driver-bin', appendconf=appendconf, addlayers="meta-intel meta-intel/meta-crownbay")

Steps:

• git init or reset /fetch/checkout master for git://git.yoctoproject.org/meta-intel (because no branch or commit args were passed to the call it points to master head)
• a local.conf will be created from a default one (hardcoded in heater.py) and appendconf is appended to it

..... TBD