Build testing using Autotest

From Yocto Project
Revision as of 17:08, 4 March 2013 by Stefans (talk | contribs)
Jump to navigationJump to search

DRAFT DRAFT DRAFT

What is Autotest ?

Autotest is a framework for fully automated testing. It is designed primarily to test the Linux kernel, though it is useful for many other purposes such as qualifying new hardware, virtualization testing and other general user space program testing under linux platforms.

If you want to know more about Autotest itself see the Autotest wiki

Because of the increasing need to automate our build tests the Yocto QA team decided to use Autotest. Currently there are 12 test cases that we run using Autotest.

Before using them and/or writing new tests you must read some Autotest documentation to understand how some things work:

https://github.com/autotest/autotest/wiki/ClientQuickStart

https://github.com/autotest/autotest/wiki/ControlRequirements

https://github.com/autotest/autotest/wiki/ControlHowto

https://github.com/autotest/autotest/wiki/AutotestApi

https://github.com/autotest/autotest/wiki/ResultsSpecification


How to run the tests using the standalone autotest client on your machine

clone the autotest repo
$ git clone https://github.com/autotest/autotest 
$ cd autotest/client/
get our tests
$ git clone -b stefans/tests git://git.yoctoproject.org/poky-contrib tmp/site_tests

Ok, so now you should have autotest installed and our tests in ~/autotest/client/tmp/site_tests.

You'll see a bunch of control.* files and a heater.py if you "ls -l ~/autotest/client/tmp/site_tests"

That's the actual test code (so we only wrote one test which we reuse in different ways).


Let's see the available tests 

$ cd ~/autotest/client
$ ./autotest list
Remotely fetched tests (/home/stefans/autotest/client/tmp/site_tests)
[Control]                                          [Description]
heater/control.172-sstate-sato-satosdk             172 - Build core-image-sato and core-image-sato-sdk
heater/control.292-lib64lsbsdk                     292 - Build lib64-core-image-lsb-sdk
heater/control.minimal                             Build core-image-minimal with a clean builddir
heater/control.280-lib32connman                    280 - Build core-image-sato with multilib and lib32-connman-gnome installed
heater/control.169-remake                          169 - Build remake and remake-native
heater/control.290-lib64sato                       290 - Build lib64-core-image-sato
heater/control.291-lib64satosdk                    291 - Build lib64-core-image-sato-sdk
heater/control.311-buildapp                        322 - Bitbake build-appliance-image
heater/control.options                             This is just a dumb control file used for global options
heater/control.296-perf                            296 - Build perf recipe
heater/control.281x-minx32                         almost 281 - Build core-image-minimal with x32 tune
heater/control.325-emgd                            325 - Build emgd-driver-bin

let's say we want to build core-image-minimal (aka bitbake core-image-minimal)
$ ./autotest --verbose run heater/control.minimal
or another one : bitbake core-image-minimal with x32 tune
$ ./autotest --verbose run heater/control.281x-minx32


You'll start to see a lot of output from Autotest and the tests themselves (remove --verbose for less noise but you'll loose bitbake's output too (it's logged anyway))

Here some output from ./autotest run heater/control.minimal 

[stefans@firebird client]$ ./autotest run heater/control.minimal
18:14:52 INFO | Writing results to /home/stefans/autotest/client/results/default
18:14:52 INFO | Could not symlink init scripts (lack of permissions)
18:14:52 INFO | Not logging /proc/slabinfo (lack of permissions)
18:14:52 INFO | START	----	----	timestamp=1362413692	localtime=Mar 04 18:14:52	
18:14:52 INFO | 	START	heater.minimal	heater.minimal	timestamp=1362413692	localtime=Mar 04 18:14:52	
18:14:52 INFO | Not logging /proc/slabinfo (lack of permissions)
18:14:52 INFO | Removing clone directory if it exists...
18:14:52 INFO | Getting repo...
18:14:52 INFO | Fetching git [REP 'git://git.yoctoproject.org/poky' BRANCH 'master'] -> /home/stefans/autotest/client/tmp/heater/src/poky
18:15:08 INFO | git commit ID is 93ec7b4d1550e07caec86e2998d0f94a01c7e785 (tag 1.4_M4.rc1-142-g93ec7b4)
18:15:08 INFO | The repo dir used is: /home/stefans/autotest/client/tmp/heater/src/poky
18:15:08 INFO | The build dir used is: /home/stefans/autotest/client/tmp/heater/src/build
18:15:08 INFO | Removing build directory if it exists...
18:15:08 INFO | Config file is:
BB_NUMBER_THREADS = "8"
PARALLEL_MAKE = "-j 8"
MACHINE ??= "qemux86-64"
DISTRO ?= "poky"
PACKAGE_CLASSES ?= "package_rpm"
EXTRA_IMAGE_FEATURES = "debug-tweaks"
USER_CLASSES ?= "buildstats image-mklibs image-prelink"
PATCHRESOLVE = "noop"
CONF_VERSION = "1"
BB_DISKMON_DIRS = "\
            STOPTASKS,${TMPDIR},1G,100K \
            STOPTASKS,${DL_DIR},1G,100K \
            STOPTASKS,${SSTATE_DIR},1G,100K \
            ABORT,${TMPDIR},100M,1K \
            ABORT,${DL_DIR},100M,1K \
            ABORT,${SSTATE_DIR},100M,1K" 

SSTATE_DIR ?= "/data/yocto/sstate-cache"
DL_DIR ?= "/data/yocto/downloads"

18:15:08 INFO | Command issued is: source ./oe-init-build-env /home/stefans/autotest/client/tmp/heater/src/build; export CLONEDIR=/home/stefans/autotest/client/tmp/heater/src/poky; bitbake core-image-minimal
18:46:52 INFO | Command returned zero exit status... That's GOOD!
18:46:52 INFO | Not logging /proc/slabinfo (lack of permissions)
18:46:52 INFO | Not logging /var/log/messages (lack of permissions)
18:46:52 INFO | 		GOOD	heater.minimal	heater.minimal	timestamp=1362415612	localtime=Mar 04 18:46:52	completed successfully
18:46:52 INFO | 	END GOOD	heater.minimal	heater.minimal	timestamp=1362415612	localtime=Mar 04 18:46:52	
18:46:52 INFO | END GOOD	----	----	timestamp=1362415612	localtime=Mar 04 18:46:52

How to write a new test aka control file

What's a control file?

The key defining component of a job is its control file; this file defines all aspects of a jobs life cycle. The control file is a python script which directly drives execution of the tests in question.

You are automatically supplied with a job object which drives the job and supplies services to the control file. A control file can be as simple as: job.run_test('test')

Control files should always start with control.XXXXX. XXXXX is up to you, but we use something like control.<testopia-case-id>-<short descriptive name>

Stuff to know

  • heater.py is the actual code
  • control files call heater with different args
  • heater.py will take care of cloning poky and setting up local.conf and running a build.

In the future there will be more classes which can be used for different tests but for the most basic ones what we have now should suffice.

Example

Let's say we want to automate this test case https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=325 In this case we would write something like this: 

[stefans@firebird ~]$ cd ~/autotest/client/tmp/site_tests/heater/
[stefans@firebird heater]$ cat control.325-emgd 
AUTHOR = "Stefan Stanacar"
NAME = "325 - Build emgd-driver-bin"
TIME = "MEDIUM"
TEST_CATEGORY = "Functional"
TEST_CLASS = "General"
TEST_TYPE = "client"

DOC = """
Set LICENSE_FLAGS_WHITELIST, add meta-intel layer and build emgd-driver-bin for crownbay
"""

appendconf ="""LICENSE_FLAGS_WHITELIST = "license_emgd-driver-bin_1.14"
MACHINE = "crownbay"
"""


if job.run_test('heater', tag='emgd01', gitsrc="git://git.yoctoproject.org/meta-intel", meta=True):
    job.run_test('heater', tag='emgd02', cmd='bitbake emgd-driver-bin', appendconf=appendconf, addlayers="meta-intel meta-intel/meta-crownbay")

Notes:

What does

Retrieved from "https://wiki.yoctoproject.org/wiki/index.php?title=Build_testing_using_Autotest&oldid=9030"