Oe-selftest

Introduction

Oe-selftest is a test framework used for testing the OpenEmbedded build system. Following are some key points in describing oe-selftest:

• based on Python unittest
• designed to simulate normal usage patterns
• does not cover image run-time testing
• implements a new layer that contains generic/specific metadata used only by tests

How to use

In order to use oe-selftest, you need the following(in this specific order):

1. Download poky / openembedded-core (dora or newer branch)
2. source the build environment setup script
3. add the meta-selftest layer in conf/bblayers.conf
4. add SANITY_TESTED_DISTROS="" to local.conf

Some selftests require qemu support, which in turns sets up internal network to perform the tests. This is a privileged operation, so make sure the user you run the tests with has required privileges. Or for simplicity extend your /etc/sudoers to allow running following commands:

• poky/scripts/runqemu-ifup
• poky/scripts/runqemu-ifdown
• poky/scripts/runqemu-gen-tapdevs

Running all (except hidden) tests

To run all oe-selftest tests simply issue:

$ oe-selftest --run-all-tests

Listing tests

1. --list-modules: lists all available modules.
2. --list-classes: lists all available classes and the tests they contain.
3. --list-tests-by <name|class|module|id|tag>: will list the tests that match the searching criteria and provide additional details like: id, tag, name, class, module for each found test.

Listing used tags:

--list-tags

Listing all tests:

--list-tests

Running specific tests

First you need to identify the test(s) that you want to run. You will need the following information: test module, test class, test method name. Look for the test modules inside /meta/lib/oeqa/selftest/ . After you have obtained the above information, issue from the command line: 'oe-selftest module1.Class1.test_1 module1.Class1.test_2 module2.Class2.test_3 ...' .

There are 2 methods for running specific tests:

Note:

• Keep in mind this structure: module.class.test
• The order the tests are running is alphabetical.

1. --run-tests:

   -–run-tests moduleA moduleB

   will run all tests from this two modules

   –-run-tests moduleA.classA module.classB

   will run all tests from this two classes

   –-run-tests moduleA.classA.test_A module.classB.test_B

   will run these two tests

   You can mix and match modules, classes and test names:

   -–run-tests moduleA moduleB.classB moduleC.classC.testC

   will run all tests from moduleA, all tests from moduleB.classB and testC from moduleC.classC

2. -–run-tests-by <name|class|module|id|tag>
   

   name

   --run-tests-by name testA testB testC: simply specify test names and it will determine automatically where they are located.

   class

   --run-tests-by class classA classB classC: simply specify class names and it will determine automatically what tests they contain and where they are located.

   module

   --run-tests-by module moduleA moduleB: simply specify module names and it will determine automatically what tests they contain.

   id

   --run-tests-by id id1 id2 id3: simply specify test ids and it will determine automatically what tests the ids point to.

   tag

   --run-tests-by tag tag1 tag2: if tests are tagged with @tag(feature=<>) you can run them like this also.

For the previous running methods you can specify the –-machine option to tell oe-selftest to run tests with custom MACHINE:

• -–machine random: will run tests switching to a random MACHINE for each test.
• -–machine all: will run every test from the list against all available MACHINES.

Running hidden tests

Hidden tests are located in test modules whose names begin with an underscore '_' . Example: '_test_hidden_module.py'. These tests can be run by specifying them as arguments like the above. Example: 'oe-selftest _test_hidden_module.Class_1.test_1'

Important file locations

The following files and locations are relevant to oe-selftest:

• the script itself: oe-selftest

is located in the /scripts directory under the main OE-Core tree.

• the case.py module

is located in the meta/lib/oeqa/selftest directory and contains the base OESelftestTestCase class that extends unittest.TestCase (via OETestCase) and adds extra functionality.

• the test modules

are located in the meta/lib/oeqa/selftest/cases directory and contain either test classes with test methods or evolved versions of the oeSelfTest class with added specific functionality and being used by the aforementioned test classes.

• the 'utils' library

is located at meta/lib/oeqa/utils and contains modules with helper methods that are used and imported by the test modules.

Basic architecture information

When designing oe-selftest we wanted to be less intrusive as possible. Doing so we have implemented the following conventions, using the extended oeSelfTest class:

appending to the build's global configuration

In order to keep appending to existing global configuration files(local.conf, auto.conf, site.conf) to a minimum, when oe-selftest starts it adds 'include selftest.inc' to local.conf and removes it at the end. The tests manipulate the global configuration by manipulating selftest.inc using the built-in methods write_config(), append_config(), remove_config(). Also the file selftest.inc is deleted before each test; this way all tests are independent from a global configuration point of view.

adding test files(like test classes) to the build

In order not to modify the existing metadata structure, a new layer 'meta-selftest' has been created for testing purposes. This layer can be used to store files used for testing and thus remove the need to modify the structure of the existing metadata or the need to create them at runtime.

appending to a recipe

In order to avoid modifying existing recipes and related files (.inc, .bbappend) a select list of recipes has a corresponding <recipe_name>.bbappend in the meta-selftest layer that contains only 1 passable line: "include test_recipe.inc". If a test needs to append configuration to a specific recipe, it only needs to manipulate the recipe's test_recipe.inc using the built-in methods write_recipeinc(), append_recipeinc(), remove_recipeinc() and delete_recipeinc(). Note that before each test starts all test_recipe.inc files are removed from the meta-selftest layer in order to keep all tests independent from a recipe-level configuration point of view.

Cleaning up

We also wanted to make cleanup and post-test actions simple so we added the following helper methods, using the extended oeSelfTest class:

• add_command_to_tearDown(command)

Adds <command> as a shell command to be run after the test ends, no matter how the test exits (error or fail).

• track_for_cleanup(path)

Adds <path> to a list of paths to be cleaned up (deleted) after the test ends, no matter how the test exits (error or fail).

Please be careful when using these methods as they have no safety checks.

You should be aware of the following

• oe-selftest uses the settings in conf/local.conf as a base for all test configurations. Any modifications made (such as DISTRO) or not made here (e.g. BB_NUMBER_THREADS) will impact all tests if the modifications are not overwritten specifically by a test.
• running all tests can take a long time! If you're modifying a particular feature/script then you may want to run just the related test module(s) or case(s) to save time.

How to contribute

You can send patches containing either improvements or tests to the openembedded-core mailing list: openembedded-core@lists.openembedded.org . Here is a guide on submitting changes.