Oe-selftest

Introduction

Oe-selftest is a test framework used for testing the poky build system. Following are some key points in describing oe-selftest: - based on Python Unit Testing - designed to simulate poky external usage patterns - does not cover image run-time testing - implements a new layer that contains generic/specific metadata used only by tests

How to use

In order to use oe-selftest, you need the following(in this specific order): - the poky source (1.5 M2 or better) - source the build environment - add the meta-selftest layer added in conf/bblayers.conf

• How to run all(except hidden) tests:

From the source build directory, issue the command 'oe-selftest'.

• How to run specific tests

First you need to identify the test(s) that you want to run. You will need the following information: test module, test class, test method name. Look for the test modules inside /meta/lib/oeqa/selftest/ . After you have obtained the above information, issue from the command line: 'oe-selftest module1.Class1.test_1 module1.Class1.test_2 module2.Class2.test_3 ...'

• How to run hidden tests

Hidden tests are located in test modules whose names begin with an underscore '_' . Example: '_test_hidden_module.py'. These tests can be run by specifying them as arguments like the above. Example: 'oe-selftest '_test_hidden_module.Class_1.test_1'

Important file locations

The following files and locations are relevant to oe-selftest:

• the script itself: oe-selftest.py

is located in the /scripts directory under the main poky tree.

• the base.py module

is located in the meta/lib/oeqa/selftest directory and contains the base oeSelfTest class that extends unittest.TestCase and adds extra functionality

• the test modules

are located in the meta/lib/oeqa/selftest directory and contain either test classes with test methods or evolved versions of the oeSelfTest class with added specific functionality and being used by the aforementioned test classes.

• the 'utils' library

is located at meta/lib/oeqa/utils and contains modules with helper methods that are used and imported by the test modules.

Basic architecture information

When designing oe-selftest we wanted to be less intrusive as possible. Doing so we have implemented the following conventions, using the extended oeSelfTest class:

• appending to the build's global configuration:

In order to keep appending to existing global configuration files(local.conf, auto.conf, site.conf) to a minimum, when oe-selftest starts it adds 'include selftest.inc' to local.conf and removes it at the end. The tests manipulate the global configuration by manipulating selftest.inc using the built-in methods write_config(), append_config(), remove_config(). Also the file selftest.inc is deleted before each test; this way all tests are independent from a global configuration point of view.

• adding test files(like test classes) to the build:

In order not to modify the existing metadata structure, a new layer 'meta-selftest' has been created for testing purposes. This layer can be used to store files used for testing and thus remove the need to modify the structure of the existing metadata or the need to create them at runtime.

• appending to a recipe:

In order to keep from modifying existing recipes and related files(.inc, .bbappend) a select list of recipes has a corresponding <recipe_name>.bbappend in the meta-selftest layer that contains only 1 passable line: "include test_recipe.inc". If a test needs to append configuration to a specific recipe, it only needs to manipulate the recipe's test_recipe.inc using the built-in methods write_recipeinc(), append_recipeinc(), remove_recipeinc() and delete_recipeinc(). Note that before each test starts all test_recipe.inc files are removed from the meta-selftest layer in order to keep all tests independent from a recipe-level configuration point of view.

We also wanted to make cleanup and post-test actions simple so we added the following helper methods, using the extended oeSelfTest class:

• add_command_to_tearDown(command)

Adds <command> as a shell command to be run after the test ends, no matter how the test exists(error or fail).

• track_for_cleanup(path)

Adds <path> to a list of paths to be cleaned up(deleted) after the test ends, no matter how the test exists(error or fail).

Please be careful when using these methods as they have no safety checks.

You should be aware of the following

• oe-selftest uses the settings in conf/local.conf as a base for all test configurations. Any modifications made(like DISTRO) or not made here(like BB_NUMBER_THREADS) will impact all tests if the modifications are not overwritten specifically by a test.
• running all tests can take a long time!

How to contribute

You can send patches containing either improvements or tests to the openembedded mailing list: openembedded-core@lists.openembedded.org . Here is a link to a guide on doing so: http://www.openembedded.org/wiki/How_to_submit_a_patch_to_OpenEmbedded