Build testing using Autotest
DRAFT
What is Autotest ?
Autotest is a framework for fully automated testing. It is designed primarily to test the Linux kernel, though it is useful for many other purposes such as qualifying new hardware, virtualization testing and other general user space program testing under linux platforms. If you want to know more about Autotest itself see the Autotest wiki
Because of the increasing need to automate our build tests the Yocto QA team decided to use Autotest.
Currently there are 12 test cases that we run using Autotest.
Before using them and/or writing new tests you must read some Autotest documentation to understand how some things work:
https://github.com/autotest/autotest/wiki/ClientQuickStart
https://github.com/autotest/autotest/wiki/ControlRequirements
https://github.com/autotest/autotest/wiki/ControlHowto
https://github.com/autotest/autotest/wiki/AutotestApi
https://github.com/autotest/autotest/wiki/ResultsSpecification
How to run the tests using the standalone autotest client on your machine
clone the autotest repo $ git clone https://github.com/autotest/autotest $ cd autotest/client/ get our tests $ git clone -b stefans/tests git://git.yoctoproject.org/poky-contrib tmp/site_tests
Ok, so now you should have autotest installed and our tests in ~/autotest/client/tmp/site_tests.
You'll see a bunch of control.* files and a heater.py if you "ls -l ~/autotest/client/tmp/site_tests"
That's the actual test code (so we only wrote one test which we reuse in different ways).
Let's see the available tests
$ cd ~/autotest/client $ ./autotest list Remotely fetched tests (/home/stefans/autotest/client/tmp/site_tests) [Control] [Description] heater/control.172-sstate-sato-satosdk 172 - Build core-image-sato and core-image-sato-sdk heater/control.292-lib64lsbsdk 292 - Build lib64-core-image-lsb-sdk heater/control.minimal Build core-image-minimal with a clean builddir heater/control.280-lib32connman 280 - Build core-image-sato with multilib and lib32-connman-gnome installed heater/control.169-remake 169 - Build remake and remake-native heater/control.290-lib64sato 290 - Build lib64-core-image-sato heater/control.291-lib64satosdk 291 - Build lib64-core-image-sato-sdk heater/control.311-buildapp 322 - Bitbake build-appliance-image heater/control.options This is just a dumb control file used for global options heater/control.296-perf 296 - Build perf recipe heater/control.281x-minx32 almost 281 - Build core-image-minimal with x32 tune heater/control.325-emgd 325 - Build emgd-driver-bin let's say we want to build core-image-minimal (aka bitbake core-image-minimal) $ ./autotest --verbose run heater/control.minimal or another one : bitbake core-image-minimal with x32 tune $ ./autotest --verbose run heater/control.281x-minx32
You'll start to see a lot of output from Autotest and the tests themselves (remove --verbose for less noise but you'll loose bitbake's output too (it's logged anyway))
Here some output from ./autotest run heater/control.minimal
[stefans@firebird client]$ ./autotest run heater/control.minimal
18:14:52 INFO | Writing results to /home/stefans/autotest/client/results/default
18:14:52 INFO | Could not symlink init scripts (lack of permissions)
18:14:52 INFO | Not logging /proc/slabinfo (lack of permissions)
18:14:52 INFO | START ---- ---- timestamp=1362413692 localtime=Mar 04 18:14:52
18:14:52 INFO | START heater.minimal heater.minimal timestamp=1362413692 localtime=Mar 04 18:14:52
18:14:52 INFO | Not logging /proc/slabinfo (lack of permissions)
18:14:52 INFO | Removing clone directory if it exists...
18:14:52 INFO | Getting repo...
18:14:52 INFO | Fetching git [REP 'git://git.yoctoproject.org/poky' BRANCH 'master'] -> /home/stefans/autotest/client/tmp/heater/src/poky
18:15:08 INFO | git commit ID is 93ec7b4d1550e07caec86e2998d0f94a01c7e785 (tag 1.4_M4.rc1-142-g93ec7b4)
18:15:08 INFO | The repo dir used is: /home/stefans/autotest/client/tmp/heater/src/poky
18:15:08 INFO | The build dir used is: /home/stefans/autotest/client/tmp/heater/src/build
18:15:08 INFO | Removing build directory if it exists...
18:15:08 INFO | Config file is:
BB_NUMBER_THREADS = "8"
PARALLEL_MAKE = "-j 8"
MACHINE ??= "qemux86-64"
DISTRO ?= "poky"
PACKAGE_CLASSES ?= "package_rpm"
EXTRA_IMAGE_FEATURES = "debug-tweaks"
USER_CLASSES ?= "buildstats image-mklibs image-prelink"
PATCHRESOLVE = "noop"
CONF_VERSION = "1"
BB_DISKMON_DIRS = "\
STOPTASKS,${TMPDIR},1G,100K \
STOPTASKS,${DL_DIR},1G,100K \
STOPTASKS,${SSTATE_DIR},1G,100K \
ABORT,${TMPDIR},100M,1K \
ABORT,${DL_DIR},100M,1K \
ABORT,${SSTATE_DIR},100M,1K"
SSTATE_DIR ?= "/data/yocto/sstate-cache"
DL_DIR ?= "/data/yocto/downloads"
18:15:08 INFO | Command issued is: source ./oe-init-build-env /home/stefans/autotest/client/tmp/heater/src/build; export CLONEDIR=/home/stefans/autotest/client/tmp/heater/src/poky; bitbake core-image-minimal
How to write a new test aka control file
Stuff to know:
- heater.py is the actual code
- control files call heater with different args
- heater.py will take care of cloning poky and setting up local.conf and running a build.
Here is a example:
[stefans@firebird ~]$ cd ~/autotest/client/tmp/site_tests/heater/
[stefans@firebird heater]$ cat control.325-emgd
AUTHOR = "Stefan Stanacar"
NAME = "325 - Build emgd-driver-bin"
TIME = "MEDIUM"
TEST_CATEGORY = "Functional"
TEST_CLASS = "General"
TEST_TYPE = "client"
DOC = """
Set LICENSE_FLAGS_WHITELIST, add meta-intel layer and build emgd-driver-bin for crownbay
"""
appendconf ="""LICENSE_FLAGS_WHITELIST = "license_emgd-driver-bin_1.14"
MACHINE = "crownbay"
"""
if job.run_test('heater', tag='emgd01', gitsrc="git://git.yoctoproject.org/meta-intel", meta=True):
job.run_test('heater', tag='emgd02', cmd='bitbake emgd-driver-bin', appendconf=appendconf, addlayers="meta-intel meta-intel/meta-crownbay")
