Build testing using Autotest: Difference between revisions

From Yocto Project
Jump to navigationJump to search
No edit summary
No edit summary
Line 1: Line 1:
= DRAFT =
<font color="red"> ''' DRAFT DRAFT DRAFT ''' </font>


== What is Autotest ?  ==
= What is Autotest ?  =


Autotest is a framework for fully automated testing. It is designed primarily to test the Linux kernel, though it is useful for many other purposes such as qualifying new hardware, virtualization testing and other general user space program testing under linux platforms. If you want to know more about Autotest itself see the [https://github.com/autotest/autotest/wiki Autotest  wiki]
Autotest is a framework for fully automated testing. It is designed primarily to test the Linux kernel, though it is useful for many other purposes such as qualifying new hardware, virtualization testing and other general user space program testing under linux platforms. If you want to know more about Autotest itself see the [https://github.com/autotest/autotest/wiki Autotest  wiki]
Line 22: Line 22:




= Tests =


== How to run the tests using the standalone autotest client on your machine  ==
== How to run the tests using the standalone autotest client on your machine  ==
Line 40: Line 41:


Let's see the available tests
Let's see the available tests
<nowiki>$ cd ~/autotest/client
<pre style="white-space: pre-wrap;
white-space: -moz-pre-wrap;
word-wrap: break-word">
$ cd ~/autotest/client
$ ./autotest list
$ ./autotest list
Remotely fetched tests (/home/stefans/autotest/client/tmp/site_tests)
Remotely fetched tests (/home/stefans/autotest/client/tmp/site_tests)
Line 60: Line 64:
$ ./autotest --verbose run heater/control.minimal
$ ./autotest --verbose run heater/control.minimal
or another one : bitbake core-image-minimal with x32 tune
or another one : bitbake core-image-minimal with x32 tune
$ ./autotest --verbose run heater/control.281x-minx32</nowiki>
$ ./autotest --verbose run heater/control.281x-minx32
</pre>




Line 67: Line 72:
Here some output from ./autotest run heater/control.minimal
Here some output from ./autotest run heater/control.minimal


<pre style="white-space: pre-wrap;
white-space: -moz-pre-wrap;
word-wrap: break-word">
[stefans@firebird client]$ ./autotest run heater/control.minimal
18:14:52 INFO | Writing results to /home/stefans/autotest/client/results/default
18:14:52 INFO | Could not symlink init scripts (lack of permissions)
18:14:52 INFO | Not logging /proc/slabinfo (lack of permissions)
18:14:52 INFO | START ---- ---- timestamp=1362413692 localtime=Mar 04 18:14:52
18:14:52 INFO | START heater.minimal heater.minimal timestamp=1362413692 localtime=Mar 04 18:14:52
18:14:52 INFO | Not logging /proc/slabinfo (lack of permissions)
18:14:52 INFO | Removing clone directory if it exists...
18:14:52 INFO | Getting repo...
18:14:52 INFO | Fetching git [REP 'git://git.yoctoproject.org/poky' BRANCH 'master'] -> /home/stefans/autotest/client/tmp/heater/src/poky
18:15:08 INFO | git commit ID is 93ec7b4d1550e07caec86e2998d0f94a01c7e785 (tag 1.4_M4.rc1-142-g93ec7b4)
18:15:08 INFO | The repo dir used is: /home/stefans/autotest/client/tmp/heater/src/poky
18:15:08 INFO | The build dir used is: /home/stefans/autotest/client/tmp/heater/src/build
18:15:08 INFO | Removing build directory if it exists...
18:15:08 INFO | Config file is:
BB_NUMBER_THREADS = "8"
PARALLEL_MAKE = "-j 8"
MACHINE ??= "qemux86-64"
DISTRO ?= "poky"
PACKAGE_CLASSES ?= "package_rpm"
EXTRA_IMAGE_FEATURES = "debug-tweaks"
USER_CLASSES ?= "buildstats image-mklibs image-prelink"
PATCHRESOLVE = "noop"
CONF_VERSION = "1"
BB_DISKMON_DIRS = "\
            STOPTASKS,${TMPDIR},1G,100K \
            STOPTASKS,${DL_DIR},1G,100K \
            STOPTASKS,${SSTATE_DIR},1G,100K \
            ABORT,${TMPDIR},100M,1K \
            ABORT,${DL_DIR},100M,1K \
            ABORT,${SSTATE_DIR},100M,1K"


<nowiki>[stefans@firebird client]$ ./autotest run heater/control.minimal
SSTATE_DIR ?= "/data/yocto/sstate-cache"
18:14:52 INFO | Writing results to /home/stefans/autotest/client/results/default
DL_DIR ?= "/data/yocto/downloads"
18:14:52 INFO | Could not symlink init scripts (lack of permissions)
18:14:52 INFO | Not logging /proc/slabinfo (lack of permissions)
18:14:52 INFO | START ---- ---- timestamp=1362413692 localtime=Mar 04 18:14:52
18:14:52 INFO | START heater.minimal heater.minimal timestamp=1362413692 localtime=Mar 04 18:14:52
18:14:52 INFO | Not logging /proc/slabinfo (lack of permissions)
18:14:52 INFO | Removing clone directory if it exists...
18:14:52 INFO | Getting repo...
18:14:52 INFO | Fetching git [REP 'git://git.yoctoproject.org/poky' BRANCH 'master'] -> /home/stefans/autotest/client/tmp/heater/src/poky
18:15:08 INFO | git commit ID is 93ec7b4d1550e07caec86e2998d0f94a01c7e785 (tag 1.4_M4.rc1-142-g93ec7b4)
18:15:08 INFO | The repo dir used is: /home/stefans/autotest/client/tmp/heater/src/poky
18:15:08 INFO | The build dir used is: /home/stefans/autotest/client/tmp/heater/src/build
18:15:08 INFO | Removing build directory if it exists...
18:15:08 INFO | Config file is:
BB_NUMBER_THREADS = "8"
PARALLEL_MAKE = "-j 8"
MACHINE ??= "qemux86-64"
DISTRO ?= "poky"
PACKAGE_CLASSES ?= "package_rpm"
EXTRA_IMAGE_FEATURES = "debug-tweaks"
USER_CLASSES ?= "buildstats image-mklibs image-prelink"
PATCHRESOLVE = "noop"
CONF_VERSION = "1"
BB_DISKMON_DIRS = "\
            STOPTASKS,${TMPDIR},1G,100K \
            STOPTASKS,${DL_DIR},1G,100K \
            STOPTASKS,${SSTATE_DIR},1G,100K \
            ABORT,${TMPDIR},100M,1K \
            ABORT,${DL_DIR},100M,1K \
            ABORT,${SSTATE_DIR},100M,1K"  


SSTATE_DIR ?= "/data/yocto/sstate-cache"
18:15:08 INFO | Command issued is: source ./oe-init-build-env /home/stefans/autotest/client/tmp/heater/src/build; export CLONEDIR=/home/stefans/autotest/client/tmp/heater/src/poky; bitbake core-image-minimal
DL_DIR ?= "/data/yocto/downloads"
18:46:52 INFO | Command returned zero exit status... That's GOOD!
18:46:52 INFO | Not logging /proc/slabinfo (lack of permissions)
18:46:52 INFO | Not logging /var/log/messages (lack of permissions)
18:46:52 INFO | GOOD heater.minimal heater.minimal timestamp=1362415612 localtime=Mar 04 18:46:52 completed successfully
18:46:52 INFO | END GOOD heater.minimal heater.minimal timestamp=1362415612 localtime=Mar 04 18:46:52
18:46:52 INFO | END GOOD ---- ---- timestamp=1362415612 localtime=Mar 04 18:46:52
</pre>


18:15:08 INFO | Command issued is: source ./oe-init-build-env /home/stefans/autotest/client/tmp/heater/src/build; export CLONEDIR=/home/stefans/autotest/client/tmp/heater/src/poky; bitbake core-image-minimal</nowiki>
== How to write a new test aka control file ==


=== What's a control file? ===


The key defining component of a job is its control file; this file defines all aspects of a jobs life cycle. The control file is a python script which directly drives execution of the tests in question.


You are automatically supplied with a job object which drives the job and supplies services to the control file. A control file can be as simple as: job.run_test('test')


 
Control files should always start with control.XXXXX. XXXXX is up to you, but we use something like control.<testopia-case-id>-<short descriptive name>
== How to write a new test aka control file ==


Stuff to know:
Stuff to know:
Line 117: Line 135:
* heater.py will take care of cloning poky and setting up local.conf and running a build.
* heater.py will take care of cloning poky and setting up local.conf and running a build.


Here is a example:
In the future there will be more classes which can be used for different tests but for the most basic ones what we have now should suffice.
 
=== Example ===


<nowiki>[stefans@firebird ~]$ cd ~/autotest/client/tmp/site_tests/heater/
Let's say we want to automate this test case https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=325
In this case we would write something like this:
 
<pre style="white-space: pre-wrap;
white-space: -moz-pre-wrap;
word-wrap: break-word">
[stefans@firebird ~]$ cd ~/autotest/client/tmp/site_tests/heater/
[stefans@firebird heater]$ cat control.325-emgd  
[stefans@firebird heater]$ cat control.325-emgd  
AUTHOR = "Stefan Stanacar"
AUTHOR = "Stefan Stanacar"
Line 138: Line 164:


if job.run_test('heater', tag='emgd01', gitsrc="git://git.yoctoproject.org/meta-intel", meta=True):
if job.run_test('heater', tag='emgd01', gitsrc="git://git.yoctoproject.org/meta-intel", meta=True):
     job.run_test('heater', tag='emgd02', cmd='bitbake emgd-driver-bin', appendconf=appendconf, addlayers="meta-intel meta-intel/meta-crownbay")</nowiki>
     job.run_test('heater', tag='emgd02', cmd='bitbake emgd-driver-bin', appendconf=appendconf, addlayers="meta-intel meta-intel/meta-crownbay")
</pre>
 
Notes:
* AUTHOR, NAME, TIME, TEST* and DOC are mandatory in a control file (autotest requirement, see https://github.com/autotest/autotest/wiki/ControlRequirements )

Revision as of 16:56, 4 March 2013

DRAFT DRAFT DRAFT

What is Autotest ?

Autotest is a framework for fully automated testing. It is designed primarily to test the Linux kernel, though it is useful for many other purposes such as qualifying new hardware, virtualization testing and other general user space program testing under linux platforms. If you want to know more about Autotest itself see the Autotest wiki


Because of the increasing need to automate our build tests the Yocto QA team decided to use Autotest. Currently there are 12 test cases that we run using Autotest.

Before using them and/or writing new tests you must read some Autotest documentation to understand how some things work:

https://github.com/autotest/autotest/wiki/ClientQuickStart

https://github.com/autotest/autotest/wiki/ControlRequirements

https://github.com/autotest/autotest/wiki/ControlHowto

https://github.com/autotest/autotest/wiki/AutotestApi

https://github.com/autotest/autotest/wiki/ResultsSpecification


Tests

How to run the tests using the standalone autotest client on your machine

clone the autotest repo
$ git clone https://github.com/autotest/autotest 
$ cd autotest/client/
get our tests
$ git clone -b stefans/tests git://git.yoctoproject.org/poky-contrib tmp/site_tests

Ok, so now you should have autotest installed and our tests in ~/autotest/client/tmp/site_tests.

You'll see a bunch of control.* files and a heater.py if you "ls -l ~/autotest/client/tmp/site_tests"

That's the actual test code (so we only wrote one test which we reuse in different ways).


Let's see the available tests 

$ cd ~/autotest/client
$ ./autotest list
Remotely fetched tests (/home/stefans/autotest/client/tmp/site_tests)
[Control]                                          [Description]
heater/control.172-sstate-sato-satosdk             172 - Build core-image-sato and core-image-sato-sdk
heater/control.292-lib64lsbsdk                     292 - Build lib64-core-image-lsb-sdk
heater/control.minimal                             Build core-image-minimal with a clean builddir
heater/control.280-lib32connman                    280 - Build core-image-sato with multilib and lib32-connman-gnome installed
heater/control.169-remake                          169 - Build remake and remake-native
heater/control.290-lib64sato                       290 - Build lib64-core-image-sato
heater/control.291-lib64satosdk                    291 - Build lib64-core-image-sato-sdk
heater/control.311-buildapp                        322 - Bitbake build-appliance-image
heater/control.options                             This is just a dumb control file used for global options
heater/control.296-perf                            296 - Build perf recipe
heater/control.281x-minx32                         almost 281 - Build core-image-minimal with x32 tune
heater/control.325-emgd                            325 - Build emgd-driver-bin

let's say we want to build core-image-minimal (aka bitbake core-image-minimal)
$ ./autotest --verbose run heater/control.minimal
or another one : bitbake core-image-minimal with x32 tune
$ ./autotest --verbose run heater/control.281x-minx32


You'll start to see a lot of output from Autotest and the tests themselves (remove --verbose for less noise but you'll loose bitbake's output too (it's logged anyway))

Here some output from ./autotest run heater/control.minimal 

[stefans@firebird client]$ ./autotest run heater/control.minimal
18:14:52 INFO | Writing results to /home/stefans/autotest/client/results/default
18:14:52 INFO | Could not symlink init scripts (lack of permissions)
18:14:52 INFO | Not logging /proc/slabinfo (lack of permissions)
18:14:52 INFO | START	----	----	timestamp=1362413692	localtime=Mar 04 18:14:52	
18:14:52 INFO | 	START	heater.minimal	heater.minimal	timestamp=1362413692	localtime=Mar 04 18:14:52	
18:14:52 INFO | Not logging /proc/slabinfo (lack of permissions)
18:14:52 INFO | Removing clone directory if it exists...
18:14:52 INFO | Getting repo...
18:14:52 INFO | Fetching git [REP 'git://git.yoctoproject.org/poky' BRANCH 'master'] -> /home/stefans/autotest/client/tmp/heater/src/poky
18:15:08 INFO | git commit ID is 93ec7b4d1550e07caec86e2998d0f94a01c7e785 (tag 1.4_M4.rc1-142-g93ec7b4)
18:15:08 INFO | The repo dir used is: /home/stefans/autotest/client/tmp/heater/src/poky
18:15:08 INFO | The build dir used is: /home/stefans/autotest/client/tmp/heater/src/build
18:15:08 INFO | Removing build directory if it exists...
18:15:08 INFO | Config file is:
BB_NUMBER_THREADS = "8"
PARALLEL_MAKE = "-j 8"
MACHINE ??= "qemux86-64"
DISTRO ?= "poky"
PACKAGE_CLASSES ?= "package_rpm"
EXTRA_IMAGE_FEATURES = "debug-tweaks"
USER_CLASSES ?= "buildstats image-mklibs image-prelink"
PATCHRESOLVE = "noop"
CONF_VERSION = "1"
BB_DISKMON_DIRS = "\
            STOPTASKS,${TMPDIR},1G,100K \
            STOPTASKS,${DL_DIR},1G,100K \
            STOPTASKS,${SSTATE_DIR},1G,100K \
            ABORT,${TMPDIR},100M,1K \
            ABORT,${DL_DIR},100M,1K \
            ABORT,${SSTATE_DIR},100M,1K" 

SSTATE_DIR ?= "/data/yocto/sstate-cache"
DL_DIR ?= "/data/yocto/downloads"

18:15:08 INFO | Command issued is: source ./oe-init-build-env /home/stefans/autotest/client/tmp/heater/src/build; export CLONEDIR=/home/stefans/autotest/client/tmp/heater/src/poky; bitbake core-image-minimal
18:46:52 INFO | Command returned zero exit status... That's GOOD!
18:46:52 INFO | Not logging /proc/slabinfo (lack of permissions)
18:46:52 INFO | Not logging /var/log/messages (lack of permissions)
18:46:52 INFO | 		GOOD	heater.minimal	heater.minimal	timestamp=1362415612	localtime=Mar 04 18:46:52	completed successfully
18:46:52 INFO | 	END GOOD	heater.minimal	heater.minimal	timestamp=1362415612	localtime=Mar 04 18:46:52	
18:46:52 INFO | END GOOD	----	----	timestamp=1362415612	localtime=Mar 04 18:46:52

How to write a new test aka control file

What's a control file?

The key defining component of a job is its control file; this file defines all aspects of a jobs life cycle. The control file is a python script which directly drives execution of the tests in question.

You are automatically supplied with a job object which drives the job and supplies services to the control file. A control file can be as simple as: job.run_test('test')

Control files should always start with control.XXXXX. XXXXX is up to you, but we use something like control.<testopia-case-id>-<short descriptive name>

Stuff to know:

  • heater.py is the actual code
  • control files call heater with different args
  • heater.py will take care of cloning poky and setting up local.conf and running a build.

In the future there will be more classes which can be used for different tests but for the most basic ones what we have now should suffice.

Example

Let's say we want to automate this test case https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=325 In this case we would write something like this: 

[stefans@firebird ~]$ cd ~/autotest/client/tmp/site_tests/heater/
[stefans@firebird heater]$ cat control.325-emgd 
AUTHOR = "Stefan Stanacar"
NAME = "325 - Build emgd-driver-bin"
TIME = "MEDIUM"
TEST_CATEGORY = "Functional"
TEST_CLASS = "General"
TEST_TYPE = "client"

DOC = """
Set LICENSE_FLAGS_WHITELIST, add meta-intel layer and build emgd-driver-bin for crownbay
"""

appendconf ="""LICENSE_FLAGS_WHITELIST = "license_emgd-driver-bin_1.14"
MACHINE = "crownbay"
"""


if job.run_test('heater', tag='emgd01', gitsrc="git://git.yoctoproject.org/meta-intel", meta=True):
    job.run_test('heater', tag='emgd02', cmd='bitbake emgd-driver-bin', appendconf=appendconf, addlayers="meta-intel meta-intel/meta-crownbay")

Notes:

Retrieved from "https://wiki.yoctoproject.org/wiki/index.php?title=Build_testing_using_Autotest&oldid=9029"