Image tests: Difference between revisions

From Yocto Project
Jump to navigationJump to search
No edit summary
No edit summary
Line 20: Line 20:
** run as root the script in scripts/runqemu-gen-tapdev which should generate a list of tap devices (that's usually done in AutoBuilder-like setups)
** run as root the script in scripts/runqemu-gen-tapdev which should generate a list of tap devices (that's usually done in AutoBuilder-like setups)
* the DISPLAY variable needs to be set so that means you need to have an X server available (e.g start vncserver)
* the DISPLAY variable needs to be set so that means you need to have an X server available (e.g start vncserver)
 
* some of the tests (in particular smart tests) start a http server on a random high number port, used to serve files to the target. The smart module for example serves ${DEPLOY_DIR}/rpm to the target so we run smart channel commands. That means your host's firewall should accept incoming connection from 192.168.7.0/24 (the default class used for tap0 devices by runqemu)




Line 48: Line 48:
* each test module is loaded in the order found in TEST_SUITES (the full output of the commands ran over ssh is found in ${WORKDIR}/testimgage/ssh_target_log)
* each test module is loaded in the order found in TEST_SUITES (the full output of the commands ran over ssh is found in ${WORKDIR}/testimgage/ssh_target_log)
* if there are no fails, the task will end successfully. You can find the output from the unittest in the task log (in ${WORKDIR}/temp/log.do_testimage)
* if there are no fails, the task will end successfully. You can find the output from the unittest in the task log (in ${WORKDIR}/temp/log.do_testimage)
----


Example of log for an systemd-enabled image that has package-management features and ssh and  TEST_SUITES = "ping ssh rpm auto" in local.conf
Example of log for an systemd-enabled image that has package-management features and ssh and  TEST_SUITES = "ping ssh rpm auto" in local.conf
Line 103: Line 106:




As you can see some tests passed and some of them were skipped (because they weren't applicable for this image). And while I haven added systemd tests to TEST_SUITES the tests were run (because of auto). Let's see what happens if I use TEST_SUITES = "ping ssh gcc" instead (and this image doesn't have developement packages installed, the tools-sdk feature:
As you can see some tests passed and some of them were skipped (because they weren't applicable for this image). And while I haven added systemd tests to TEST_SUITES the tests were run (because of auto).  
 
 
----
 
Let's see what happens if I use TEST_SUITES = "ping ssh gcc" for a core-image-sato image (which doesn't have the tools-sdk feature):




Line 113: Line 121:
test_ping (oeqa.runtime.ping.PingTest) ... ok
test_ping (oeqa.runtime.ping.PingTest) ... ok
test_ssh (oeqa.runtime.ssh.SshTest) ... ok
test_ssh (oeqa.runtime.ssh.SshTest) ... ok
WARNING: Test gcc is required, not skipping
ERROR
test_gcc_compile (oeqa.runtime.gcc.GccCompileTest) ... FAIL
test_gpp_compile (oeqa.runtime.gcc.GccCompileTest) ... FAIL
test_make (oeqa.runtime.gcc.GccCompileTest) ... FAIL
 
======================================================================
FAIL: test_gcc_compile (oeqa.runtime.gcc.GccCompileTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/stefans/yocto/poky-work/meta/lib/oeqa/runtime/gcc.py", line 20, in test_gcc_compile
    self.assertEqual(status, 0, msg="gcc compile failed, output: %s" % output)
AssertionError: gcc compile failed, output: sh: gcc: command not found
 
======================================================================
FAIL: test_gpp_compile (oeqa.runtime.gcc.GccCompileTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/stefans/yocto/poky-work/meta/lib/oeqa/runtime/gcc.py", line 26, in test_gpp_compile
    self.assertEqual(status, 0, msg="g++ compile failed, output: %s" % output)
AssertionError: g++ compile failed, output: sh: g++: command not found


======================================================================
======================================================================
FAIL: test_make (oeqa.runtime.gcc.GccCompileTest)
ERROR: setUpModule (oeqa.runtime.gcc)
----------------------------------------------------------------------
----------------------------------------------------------------------
Traceback (most recent call last):
Traceback (most recent call last):
   File "/home/stefans/yocto/poky-work/meta/lib/oeqa/runtime/gcc.py", line 32, in test_make
   File "/home/stefans/z/poky/meta/lib/oeqa/runtime/gcc.py", line 8, in setUpModule
     self.assertEqual(status, 0, msg="running make failed, output %s" % output)
     skipModule("Image doesn't have tools-sdk in IMAGE_FEATURES")
AssertionError: running make failed, output sh: make: command not found
  File "/home/stefans/z/poky/meta/lib/oeqa/oetest.py", line 108, in skipModule
    "\nor the image really doesn't have the requred feature/package when it should." % (modname, reason))
Exception:
Test gcc wants to be skipped.
Reason is: Image doesn't have tools-sdk in IMAGE_FEATURES
Test was required in TEST_SUITES, so either the condition for skipping is wrong
or the image really doesn't have the requred feature/package when it should.


----------------------------------------------------------------------
----------------------------------------------------------------------
Ran 5 tests in 11.167s
Ran 2 tests in 6.255s


FAILED (failures=3)
FAILED (errors=1)
NOTE: Sending SIGTERM to runqemu
DEBUG: Python function do_testimage finished
DEBUG: Python function do_testimage finished
ERROR: Function failed: Some tests failed. You should check the task log and the ssh log. (ssh log is /home/stefans/yocto/builds/firefly/tmp/work/qemux86_64-poky-linux/core-image-base/1.0-r0/testimage/ssh_target_log.20130819120149
ERROR: Function failed: Some tests failed. You should check the task log and the ssh log. (ssh log is /home/stefans/z/poky/build/tmp/work/qemux86_64-poky-linux/core-image-sato/1.0-r0/testimage/ssh_target_log.20130827122341
</pre>
</pre>


* First, it tells us it loaded the module we required (ping, ssh and gcc) and that there are 5 tests (because the gcc module has 3 test methods)
* First, it tells us it loaded the module we required (ping, ssh and gcc) and that there are 5 tests (because the gcc module has 3 test methods)
* It gives a warning about the gcc being required and not skipping it (it does that because the test module itself checks for tools-sdk in IMAGE_FEATURES and wants to be skipped, as it's pointless to run the gcc compile test on an image that doesn't have gcc but because gcc is a required module - we added it to TEST_SUITES it can't be skipped).
* It starts running the tests
* the results are shown with their respective assertion errors and the output from the commands. Tn this case the error is simple to understand: gcc: command not found. These three test methods don't have a dependecy between themselves (that is if the first fails, skip the others, but that's not the case for all modules, were some test methods depend on others).
* the gcc module will error out giving us a traceback of why that happened. Because gcc was a required test, it wasn't skipped like earlier, instead it was marked as an error.


Looking at the ssh log we could see the exact commands ran:
<pre>
[Running]$ ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -l root 192.168.7.2 . /etc/profile; uname -a
Linux qemux86-64 3.8.13-yocto-standard #1 SMP PREEMPT Fri Aug 16 15:51:02 EEST 2013 x86_64 GNU/Linux
[SSH command returned]: 0
[Running SCP]$ scp -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no /home/stefans/yocto/poky-work/meta/lib/oeqa/runtime/files/test.c root@192.168.7.2:/tmp/test.c
Warning: Permanently added '192.168.7.2' (RSA) to the list of known hosts.


[Running SCP]$ scp -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no /home/stefans/yocto/poky-work/meta/lib/oeqa/runtime/files/testmakefile root@192.168.7.2:/tmp/testmakefile
----
Warning: Permanently added '192.168.7.2' (RSA) to the list of known hosts.


[Running]$ ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -l root 192.168.7.2 . /etc/profile; gcc /tmp/test.c -o /tmp/test -lm
sh: gcc: command not found
[SSH command returned]: 127
[Running]$ ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -l root 192.168.7.2 . /etc/profile; g++ /tmp/test.c -o /tmp/test -lm
sh: g++: command not found
[SSH command returned]: 127
[Running]$ ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -l root 192.168.7.2 . /etc/profile; cd /tmp; make -f testmakefile
sh: make: command not found
[SSH command returned]: 127
[Running]$ ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -l root 192.168.7.2 . /etc/profile; rm /tmp/test.c /tmp/test.o /tmp/test /tmp/testmakefile
rm: cannot remove '/tmp/test.o': No such file or directory
rm: cannot remove '/tmp/test': No such file or directory
[SSH command returned]: 1


</pre>
Example of a ssh log for a core-image-sato with TEST_SUITES = "ping ssh

Revision as of 12:35, 27 August 2013

About the the testimage class

The build system has the ability to run a series of automated tests for qemu images.

All the tests are actually commands run on the target system over ssh.

The tests themselves are written in Python, making use of the unittest module.

The class that enables this is testimage.bbclass (which handles loading the tests and starting the qemu image)


Enabling and running the tests

Requirements

You should be aware of the following:

  • runqemu script needs sudo access for setting up the tap interface, so you need to make sure it can do that non-interactively. That means you need to one of the following:
    • add NOPASSWD for your user either for ALL commands in /etc/sudoers, either just for runqemu-ifup (but you need to provide the full path and that can change if you have multiple poky clones)
    • manually configure a tap interface for your system
    • run as root the script in scripts/runqemu-gen-tapdev which should generate a list of tap devices (that's usually done in AutoBuilder-like setups)
  • the DISPLAY variable needs to be set so that means you need to have an X server available (e.g start vncserver)
  • some of the tests (in particular smart tests) start a http server on a random high number port, used to serve files to the target. The smart module for example serves ${DEPLOY_DIR}/rpm to the target so we run smart channel commands. That means your host's firewall should accept incoming connection from 192.168.7.0/24 (the default class used for tap0 devices by runqemu)


To use it add "testimage" to global inherit and call your target image with -c testimage, like this:

  • for example build a qemu core-image-sato: bitbake core-image-sato
  • add INHERIT += "testimage" in local.conf
  • then call "bitbake core-image-sato -c testimage". That will run a standard suite of tests.

The name of tests is the name of the python modules in meta/lib/oeqa/runtime.

You can change the tests run by appending or overrding the TEST_SUITES variable in local.conf. Each name in TEST_SUITES represents a required test for the image. That means that no skipping is allowed (even if the test isn't suitable for the image, e.g running the rpm tests on a images with no rpm). Appending "auto" to TEST_SUITES means that it will try to run all tests that are suitable for the image (each test decides that on it's own).

Note that the order in TEST_SUITES is important (it's the order modules run) and it influences tests dependencies. That means that tests that depend on other tests (e.g ssh depends on the ping test) should be added last. Each module can have multiple classes with multiple test methods (and Python unittest rules apply here).

Examples

Examples:

  • to run the default tests for core-image-sato you don't need to change TEST_SUITES. The default for core-image-sato is: "ping ssh connman df rpm smart xorg syslog dmesg"
  • to add your own test to the list of the defaults add: TEST_SUITES_append = " mytest"
  • to run a specific list of tests: TEST_SUITES = "ping ssh rpm" (remember order is important)

Once you call the testimage task (bitbake <my-image> -c testimage) a couple of things happen:

  • a copy of the rootfs is done in ${WORKDIR}/testimage
  • the image is booted under qemu using the standard runqemu script
  • there is a timeout of 500 seconds by default for the boot process to reach the login prompt (you can change the timeout by setting TEST_QEMUBOOT_TIMEOUT in local.conf)
  • once the boot process reached the login prompt the tests are run (you can find the full boot log in ${WORKDIR}/testimage/qemu_boot_log)
  • each test module is loaded in the order found in TEST_SUITES (the full output of the commands ran over ssh is found in ${WORKDIR}/testimgage/ssh_target_log)
  • if there are no fails, the task will end successfully. You can find the output from the unittest in the task log (in ${WORKDIR}/temp/log.do_testimage)


Example of log for an systemd-enabled image that has package-management features and ssh and TEST_SUITES = "ping ssh rpm auto" in local.conf 

$ cat tmp/work/qemux86_64-poky-linux/core-image-base/1.0-r0/temp/log.do_testimage
DEBUG: Executing python function do_testimage
NOTE: Created listening socket for qemu serial console on: 127.0.0.1:56358
NOTE: DISPLAY value: :0
NOTE: rootfs file: /home/stefans/yocto/builds/firefly/tmp/work/qemux86_64-poky-linux/core-image-base/1.0-r0/testimage/core-image-base-qemux86-64-testimage.ext3
NOTE: Qemu log file: /home/stefans/yocto/builds/firefly/tmp/work/qemux86_64-poky-linux/core-image-base/1.0-r0/testimage/qemu_boot_log.20130819115123
NOTE: SSH log file: /home/stefans/yocto/builds/firefly/tmp/work/qemux86_64-poky-linux/core-image-base/1.0-r0/testimage/ssh_target_log.20130819115123
NOTE: runqemu started, pid is 2979
NOTE: waiting at most 60 seconds for qemu pid
NOTE: qemu started - qemu procces pid is 3061
NOTE: IP found: 192.168.7.2
NOTE: Waiting at most 500 seconds for login banner
NOTE: Connection from 127.0.0.1:44406
NOTE: Reached login banner
NOTE: Test modules  ['oeqa.runtime.ping', 'oeqa.runtime.ssh', 'oeqa.runtime.rpm', 'oeqa.runtime.multilib', 'oeqa.runtime.smart', 'oeqa.runtime.dmesg', 'oeqa.runtime.df', 'oeqa.runtime.connman', 'oeqa.runtime.gcc', 'oeqa.runtime.xorg', 'oeqa.runtime.syslog', 'oeqa.runtime.systemd']
NOTE: Found 31 tests
test_ping (oeqa.runtime.ping.PingTest) ... ok
test_ssh (oeqa.runtime.ssh.SshTest) ... ok
test_rpm_help (oeqa.runtime.rpm.RpmHelpTest) ... ok
test_rpm_query (oeqa.runtime.rpm.RpmQueryTest) ... ok
skipped "multilib: this isn't a multilib:lib32 image"
test_smart_help (oeqa.runtime.smart.SmartHelpTest) ... ok
test_smart_info (oeqa.runtime.smart.SmartQueryTest) ... ok
test_smart_query (oeqa.runtime.smart.SmartQueryTest) ... ok
test_dmesg (oeqa.runtime.dmesg.DmesgTest) ... ok
test_df (oeqa.runtime.df.DfTest) ... ok
skipped 'connman: No connman package in image'
skipped "gcc: Image doesn't have tools-sdk in IMAGE_FEATURES"
skipped "xorg: target doesn't have x11 in IMAGE_FEATURES"
test_syslog_help (oeqa.runtime.syslog.SyslogTest) ... ok
test_syslog_running (oeqa.runtime.syslog.SyslogTest) ... ok
test_syslog_logger (oeqa.runtime.syslog.SyslogTestConfig) ... ok
test_syslog_restart (oeqa.runtime.syslog.SyslogTestConfig) ... ok
test_syslog_startup_config (oeqa.runtime.syslog.SyslogTestConfig) ... skipped 'Not appropiate for systemd image'
test_systemd_version (oeqa.runtime.systemd.SystemdBasicTest) ... ok
test_systemd_disable (oeqa.runtime.systemd.SystemdTests) ... ok
test_systemd_enable (oeqa.runtime.systemd.SystemdTests) ... ok
test_systemd_failed (oeqa.runtime.systemd.SystemdTests) ... ok
test_systemd_list (oeqa.runtime.systemd.SystemdTests) ... ok
test_systemd_service (oeqa.runtime.systemd.SystemdTests) ... ok
test_systemd_start (oeqa.runtime.systemd.SystemdTests) ... ok
test_systemd_stop (oeqa.runtime.systemd.SystemdTests) ... ok

----------------------------------------------------------------------
Ran 22 tests in 48.492s

OK (skipped=5)
NOTE: All required tests passed
DEBUG: Python function do_testimage finished


As you can see some tests passed and some of them were skipped (because they weren't applicable for this image). And while I haven added systemd tests to TEST_SUITES the tests were run (because of auto).


Let's see what happens if I use TEST_SUITES = "ping ssh gcc" for a core-image-sato image (which doesn't have the tools-sdk feature): 


--snip--
NOTE: Reached login banner
NOTE: Test modules  ['oeqa.runtime.ping', 'oeqa.runtime.ssh', 'oeqa.runtime.gcc']
NOTE: Found 5 tests
test_ping (oeqa.runtime.ping.PingTest) ... ok
test_ssh (oeqa.runtime.ssh.SshTest) ... ok
ERROR

======================================================================
ERROR: setUpModule (oeqa.runtime.gcc)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/stefans/z/poky/meta/lib/oeqa/runtime/gcc.py", line 8, in setUpModule
    skipModule("Image doesn't have tools-sdk in IMAGE_FEATURES")
  File "/home/stefans/z/poky/meta/lib/oeqa/oetest.py", line 108, in skipModule
    "\nor the image really doesn't have the requred feature/package when it should." % (modname, reason))
Exception: 
Test gcc wants to be skipped.
Reason is: Image doesn't have tools-sdk in IMAGE_FEATURES
Test was required in TEST_SUITES, so either the condition for skipping is wrong
or the image really doesn't have the requred feature/package when it should.

----------------------------------------------------------------------
Ran 2 tests in 6.255s

FAILED (errors=1)
NOTE: Sending SIGTERM to runqemu
DEBUG: Python function do_testimage finished
ERROR: Function failed: Some tests failed. You should check the task log and the ssh log. (ssh log is /home/stefans/z/poky/build/tmp/work/qemux86_64-poky-linux/core-image-sato/1.0-r0/testimage/ssh_target_log.20130827122341
  • First, it tells us it loaded the module we required (ping, ssh and gcc) and that there are 5 tests (because the gcc module has 3 test methods)
  • It starts running the tests
  • the gcc module will error out giving us a traceback of why that happened. Because gcc was a required test, it wasn't skipped like earlier, instead it was marked as an error.



Example of a ssh log for a core-image-sato with TEST_SUITES = "ping ssh

Retrieved from "https://wiki.yoctoproject.org/wiki/index.php?title=Image_tests&oldid=10951"