Build testing using Autotest

DRAFT

What is Autotest ?

Autotest is a framework for fully automated testing. It is designed primarily to test the Linux kernel, though it is useful for many other purposes such as qualifying new hardware, virtualization testing and other general user space program testing under linux platforms. If you want to know more about Autotest itself see the Autotest wiki

Because of the increasing need to automate our build tests the Yocto QA team decided to use Autotest. Currently there are 12 test cases that we run using Autotest.

Before using them and/or writing new tests you must read some Autotest documentation to understand how some things work:

https://github.com/autotest/autotest/wiki/ClientQuickStart

https://github.com/autotest/autotest/wiki/ControlRequirements

https://github.com/autotest/autotest/wiki/ControlHowto

https://github.com/autotest/autotest/wiki/AutotestApi

https://github.com/autotest/autotest/wiki/ResultsSpecification

How to run the tests using the standalone autotest client on your machine

clone the autotest repo
$ git clone https://github.com/autotest/autotest 
$ cd autotest/client/
get our tests
$ git clone -b stefans/tests git://git.yoctoproject.org/poky-contrib tmp/site_tests

Ok, so now you should have autotest installed and our tests in ~/autotest/client/tmp/site_tests.

You'll see a bunch of control.* files and a heater.py if you "ls -l ~/autotest/client/tmp/site_tests"

That's the actual test code (so we only wrote one test which we reuse in different ways).

Let's see the available tests

$ cd ~/autotest/client
$ ./autotest list
Remotely fetched tests (/home/stefans/autotest/client/tmp/site_tests)
[Control]                                          [Description]
heater/control.172-sstate-sato-satosdk             172 - Build core-image-sato and core-image-sato-sdk
heater/control.292-lib64lsbsdk                     292 - Build lib64-core-image-lsb-sdk
heater/control.minimal                             Build core-image-minimal with a clean builddir
heater/control.280-lib32connman                    280 - Build core-image-sato with multilib and lib32-connman-gnome installed
heater/control.169-remake                          169 - Build remake and remake-native
heater/control.290-lib64sato                       290 - Build lib64-core-image-sato
heater/control.291-lib64satosdk                    291 - Build lib64-core-image-sato-sdk
heater/control.311-buildapp                        322 - Bitbake build-appliance-image
heater/control.options                             This is just a dumb control file used for global options
heater/control.296-perf                            296 - Build perf recipe
heater/control.281x-minx32                         almost 281 - Build core-image-minimal with x32 tune
heater/control.325-emgd                            325 - Build emgd-driver-bin

let's say we want to build core-image-minimal (aka bitbake core-image-minimal)
$ ./autotest --verbose run heater/control.minimal
or another one : bitbake core-image-minimal with x32 tune
$ ./autotest --verbose run heater/control.281x-minx32

You'll start to see a lot of output from Autotest and the tests themselves (remove --verbose for less noise but you'll loose bitbake's output too (it's logged anyway))

Here some output from ./autotest run heater/control.minimal

[stefans@firebird client]$ ./autotest run heater/control.minimal
 18:14:52 INFO | Writing results to /home/stefans/autotest/client/results/default
 18:14:52 INFO | Could not symlink init scripts (lack of permissions)
 18:14:52 INFO | Not logging /proc/slabinfo (lack of permissions)
 18:14:52 INFO | START	----	----	timestamp=1362413692	localtime=Mar 04 18:14:52	
 18:14:52 INFO | 	START	heater.minimal	heater.minimal	timestamp=1362413692	localtime=Mar 04 18:14:52	
 18:14:52 INFO | Not logging /proc/slabinfo (lack of permissions)
 18:14:52 INFO | Removing clone directory if it exists...
 18:14:52 INFO | Getting repo...
 18:14:52 INFO | Fetching git [REP 'git://git.yoctoproject.org/poky' BRANCH 'master'] -> /home/stefans/autotest/client/tmp/heater/src/poky
 18:15:08 INFO | git commit ID is 93ec7b4d1550e07caec86e2998d0f94a01c7e785 (tag 1.4_M4.rc1-142-g93ec7b4)
 18:15:08 INFO | The repo dir used is: /home/stefans/autotest/client/tmp/heater/src/poky
 18:15:08 INFO | The build dir used is: /home/stefans/autotest/client/tmp/heater/src/build
 18:15:08 INFO | Removing build directory if it exists...
 18:15:08 INFO | Config file is:
 BB_NUMBER_THREADS = "8"
 PARALLEL_MAKE = "-j 8"
 MACHINE ??= "qemux86-64"
 DISTRO ?= "poky"
 PACKAGE_CLASSES ?= "package_rpm"
 EXTRA_IMAGE_FEATURES = "debug-tweaks"
 USER_CLASSES ?= "buildstats image-mklibs image-prelink"
 PATCHRESOLVE = "noop"
 CONF_VERSION = "1"
 BB_DISKMON_DIRS = "\
             STOPTASKS,${TMPDIR},1G,100K \
             STOPTASKS,${DL_DIR},1G,100K \
             STOPTASKS,${SSTATE_DIR},1G,100K \
             ABORT,${TMPDIR},100M,1K \
             ABORT,${DL_DIR},100M,1K \
             ABORT,${SSTATE_DIR},100M,1K" 

 SSTATE_DIR ?= "/data/yocto/sstate-cache"
 DL_DIR ?= "/data/yocto/downloads"

 18:15:08 INFO | Command issued is: source ./oe-init-build-env /home/stefans/autotest/client/tmp/heater/src/build; export CLONEDIR=/home/stefans/autotest/client/tmp/heater/src/poky; bitbake core-image-minimal

How to write a new test aka control file

Stuff to know:

• heater.py is the actual code
• control files call heater with different args
• heater.py will take care of cloning poky and setting up local.conf and running a build.

Here is a example:

[stefans@firebird ~]$ cd ~/autotest/client/tmp/site_tests/heater/
[stefans@firebird heater]$ cat control.325-emgd 
AUTHOR = "Stefan Stanacar"
NAME = "325 - Build emgd-driver-bin"
TIME = "MEDIUM"
TEST_CATEGORY = "Functional"
TEST_CLASS = "General"
TEST_TYPE = "client"

DOC = """
Set LICENSE_FLAGS_WHITELIST, add meta-intel layer and build emgd-driver-bin for crownbay
"""

appendconf ="""LICENSE_FLAGS_WHITELIST = "license_emgd-driver-bin_1.14"
MACHINE = "crownbay"
"""


if job.run_test('heater', tag='emgd01', gitsrc="git://git.yoctoproject.org/meta-intel", meta=True):
    job.run_test('heater', tag='emgd02', cmd='bitbake emgd-driver-bin', appendconf=appendconf, addlayers="meta-intel meta-intel/meta-crownbay")