Oe-selftest: Difference between revisions

From Yocto Project
Jump to navigationJump to search
No edit summary
No edit summary
 
(15 intermediate revisions by 6 users not shown)
Line 1: Line 1:
=Introduction=
=Introduction=
Oe-selftest is a test framework used for testing the poky build system. Following are some key points in describing oe-selftest:
Oe-selftest is a test framework used for testing the OpenEmbedded build system. Following are some key points in describing oe-selftest:
- based on Python Unit Testing
* based on Python [http://docs.python.org/2/library/unittest.html unittest]
- designed to simulate poky external usage patterns  
* designed to simulate normal usage patterns  
- does not cover image run-time testing
* does not cover image run-time testing
- implements a new layer that contains generic/specific metadata used only by tests  
* implements a new layer that contains generic/specific metadata used only by tests


=How to use=
=How to use=


In order to use oe-selftest, you need the following(in this specific order):
In order to use oe-selftest, you need the following(in this specific order):
- the poky source (1.5 M2 or better)
# Download poky / openembedded-core (dora or newer branch)
- source the build environment
# source the build environment setup script
- add the meta-selftest layer added in conf/bblayers.conf
# add the meta-selftest layer in conf/bblayers.conf
# add SANITY_TESTED_DISTROS="" to local.conf


* How to run all(except hidden) tests:
Some selftests require qemu support, which in turns sets up internal network to perform the tests. This is a privileged operation, so make sure the user you run the tests with has required privileges. Or for simplicity extend your ''/etc/sudoers'' to allow running following commands:
From the source build directory, issue the command 'oe-selftest'.
* poky/scripts/runqemu-ifup
* poky/scripts/runqemu-ifdown
* poky/scripts/runqemu-gen-tapdevs


* How to run specific tests
== Running all (except hidden) tests ==
To run all oe-selftest tests simply issue:
$ oe-selftest --run-all-tests
 
==Listing tests==
#--list-modules: lists all available modules.
#--list-classes: lists all available classes and the tests they contain.
#--list-tests-by <name|class|module|id|tag>: will list the tests that match the searching criteria and provide additional details like: id, tag, name, class, module for each found test.
 
Listing used tags:
--list-tags
 
Listing all tests:
--list-tests
 
== Running specific tests ==
First you need to identify the test(s) that you want to run. You will need the following information: test module, test class, test method name.
First you need to identify the test(s) that you want to run. You will need the following information: test module, test class, test method name.
Look for the test modules inside /meta/lib/oeqa/selftest/ .
Look for the test modules inside /meta/lib/oeqa/selftest/ .
After you have obtained the above information, issue from the command line: 'oe-selftest module1.Class1.test_1 module1.Class1.test_2 module2.Class2.test_3 ...'
After you have obtained the above information, issue from the command line: 'oe-selftest module1.Class1.test_1 module1.Class1.test_2 module2.Class2.test_3 ...' .
 
 
There are 2 methods for running specific tests:
 
Note: 
*Keep in mind this structure: module.class.test
*The order the tests are running is alphabetical.
#--run-tests:
#;-–run-tests moduleA moduleB: will run all tests from this two modules
#;–-run-tests moduleA.classA module.classB: will run all tests from this two classes
#;–-run-tests moduleA.classA.test_A module.classB.test_B: will run these two tests
#:You can mix and match modules, classes and test names:
#;-–run-tests moduleA moduleB.classB moduleC.classC.testC: will run all tests from moduleA, all tests from moduleB.classB and testC from moduleC.classC
#-–run-tests-by  <name|class|module|id|tag> <br>
#;name: --run-tests-by name testA testB testC: simply specify test names and it will determine automatically where they are located.
#;class: --run-tests-by class classA classB classC: simply specify class names and it will determine automatically what tests they contain and  where they are located.
#;module: --run-tests-by module moduleA moduleB: simply specify module names and it will determine automatically what tests they contain.
#;id: --run-tests-by id id1 id2 id3: simply specify test ids and it will determine automatically what tests the ids point to.
#;tag: --run-tests-by tag tag1 tag2: if tests are tagged with @tag(feature=<>) you can run them like this also.
 
For the previous running methods you can specify the –-machine option to tell oe-selftest to run tests with custom MACHINE:
*-–machine random: will run tests switching to a random MACHINE for each test.
*-–machine all: will run every test from the list against all available MACHINES.
 


* How to run hidden tests
== Running hidden tests ==
Hidden tests are located in test modules whose names begin with an underscore '_' . Example: '_test_hidden_module.py'. These tests can be run by specifying them as arguments like the above.
Hidden tests are located in test modules whose names begin with an underscore '_' . Example: '_test_hidden_module.py'. These tests can be run by specifying them as arguments like the above.
Example: 'oe-selftest '_test_hidden_module.Class_1.test_1'
Example: 'oe-selftest _test_hidden_module.Class_1.test_1'


=Important file locations=
=Important file locations=
The following files and locations are relevant to oe-selftest:
The following files and locations are relevant to oe-selftest:
* the script itself: oe-selftest.py
* the script itself: oe-selftest
is located in the /scripts directory under the main poky tree.  
is located in the /scripts directory under the main OE-Core tree.  
* the base.py module
* the case.py module
is located in the meta/lib/oeqa/selftest directory and contains the base oeSelfTest class that extends unittest.TestCase and adds extra functionality
is located in the meta/lib/oeqa/selftest directory and contains the base OESelftestTestCase class that extends unittest.TestCase (via OETestCase) and adds extra functionality.
* the test modules
* the test modules
are located in the meta/lib/oeqa/selftest directory and contain either test classes with test methods or evolved versions of the oeSelfTest class with added specific functionality and being used by the aforementioned test classes.
are located in the meta/lib/oeqa/selftest/cases directory and contain either test classes with test methods or evolved versions of the oeSelfTest class with added specific functionality and being used by the aforementioned test classes.
* the 'utils' library
* the 'utils' library
is located at meta/lib/oeqa/utils and contains modules with helper methods that are used and imported by the test modules.
is located at meta/lib/oeqa/utils and contains modules with helper methods that are used and imported by the test modules.


=Basic architecture information=
=Basic architecture information=
When designing oe-selftest we wanted to be less intrusive as possible. Doing so we have implemented the following conventions, using the extended oeSelfTest class:
When designing oe-selftest we wanted to be less intrusive as possible. Doing so we have implemented the following conventions, using the extended oeSelfTest class:
* appending to the build's global configuration:
== appending to the build's global configuration ==
In order to keep appending to existing global configuration files(local.conf, auto.conf, site.conf) to a minimum, when oe-selftest starts it adds 'include selftest.inc' to local.conf and removes it at the end. The tests manipulate the global configuration by manipulating selftest.inc using the built-in methods write_config(), append_config(), remove_config().  Also the file selftest.inc is deleted before each test; this way all tests are independent from a global configuration point of view.
In order to keep appending to existing global configuration files(local.conf, auto.conf, site.conf) to a minimum, when oe-selftest starts it adds 'include selftest.inc' to local.conf and removes it at the end. The tests manipulate the global configuration by manipulating selftest.inc using the built-in methods write_config(), append_config(), remove_config().  Also the file selftest.inc is deleted before each test; this way all tests are independent from a global configuration point of view.
* adding test files(like test classes) to the build:
== adding test files(like test classes) to the build ==
In order not to modify the existing metadata structure, a new layer 'meta-selftest' has been created for testing purposes. This layer can be used to store files used for testing and thus remove the need to modify the structure of the existing metadata or the need to create them at runtime.
In order not to modify the existing metadata structure, a new layer 'meta-selftest' has been created for testing purposes. This layer can be used to store files used for testing and thus remove the need to modify the structure of the existing metadata or the need to create them at runtime.
* appending to a recipe:
== appending to a recipe ==
In order to keep from modifying existing recipes and related files(.inc, .bbappend) a select list of recipes has a corresponding <recipe_name>.bbappend in the meta-selftest layer that contains only 1 passable line: "include test_recipe.inc". If a test needs to append configuration to a specific recipe, it only needs to manipulate the recipe's test_recipe.inc using the built-in methods write_recipeinc(), append_recipeinc(), remove_recipeinc() and delete_recipeinc(). Note that before each test starts all test_recipe.inc files are removed from the meta-selftest layer in order to keep all tests independent from a recipe-level configuration point of view.
In order to avoid modifying existing recipes and related files (.inc, .bbappend) a select list of recipes has a corresponding <recipe_name>.bbappend in the meta-selftest layer that contains only 1 passable line: "include test_recipe.inc". If a test needs to append configuration to a specific recipe, it only needs to manipulate the recipe's test_recipe.inc using the built-in methods write_recipeinc(), append_recipeinc(), remove_recipeinc() and delete_recipeinc(). Note that before each test starts all test_recipe.inc files are removed from the meta-selftest layer in order to keep all tests independent from a recipe-level configuration point of view.
 
== Cleaning up ==


We also wanted to make cleanup and post-test actions simple so we added the following helper methods, using the extended oeSelfTest class:
We also wanted to make cleanup and post-test actions simple so we added the following helper methods, using the extended oeSelfTest class:
* add_command_to_tearDown(command)
* add_command_to_tearDown(command)
Adds <command> as a shell command to be run after the test ends, no matter how the test exists(error or fail).
Adds <command> as a shell command to be run after the test ends, no matter how the test exits (error or fail).
* track_for_cleanup(path)
* track_for_cleanup(path)
Adds <path> to a list of paths to be cleaned up(deleted) after the test ends, no matter how the test exists(error or fail).
Adds <path> to a list of paths to be cleaned up (deleted) after the test ends, no matter how the test exits (error or fail).


Please be careful when using these methods as they have no safety checks.
Please be careful when using these methods as they have no safety checks.


=You should be aware of the following=
=You should be aware of the following=
* oe-selftest uses the settings in conf/local.conf as a base for all test configurations. Any modifications made(like DISTRO) or not made here(like BB_NUMBER_THREADS) will impact all tests if the modifications are not overwritten specifically by a test.
* oe-selftest uses the settings in conf/local.conf as a base for all test configurations. Any modifications made (such as DISTRO) or not made here (e.g. BB_NUMBER_THREADS) will impact all tests if the modifications are not overwritten specifically by a test.
* running all tests can take a long time!
* running all tests can take a long time! If you're modifying a particular feature/script then you may want to run just the related test module(s) or case(s) to save time.


=How to contribute=
=How to contribute=
You can send patches containing either improvements or tests to the open-embedded mailing list: openembedded-core@lists.openembedded.org . Here is a link to a guide on doing so: http://www.openembedded.org/wiki/How_to_submit_a_patch_to_OpenEmbedded
You can send patches containing either improvements or tests to the openembedded-core mailing list: openembedded-core@lists.openembedded.org . Here is a [http://www.openembedded.org/wiki/How_to_submit_a_patch_to_OpenEmbedded guide on submitting changes].

Latest revision as of 06:48, 19 February 2024

Introduction

Oe-selftest is a test framework used for testing the OpenEmbedded build system. Following are some key points in describing oe-selftest:

  • based on Python unittest
  • designed to simulate normal usage patterns
  • does not cover image run-time testing
  • implements a new layer that contains generic/specific metadata used only by tests

How to use

In order to use oe-selftest, you need the following(in this specific order):

  1. Download poky / openembedded-core (dora or newer branch)
  2. source the build environment setup script
  3. add the meta-selftest layer in conf/bblayers.conf
  4. add SANITY_TESTED_DISTROS="" to local.conf

Some selftests require qemu support, which in turns sets up internal network to perform the tests. This is a privileged operation, so make sure the user you run the tests with has required privileges. Or for simplicity extend your /etc/sudoers to allow running following commands:

  • poky/scripts/runqemu-ifup
  • poky/scripts/runqemu-ifdown
  • poky/scripts/runqemu-gen-tapdevs

Running all (except hidden) tests

To run all oe-selftest tests simply issue:

$ oe-selftest --run-all-tests

Listing tests

  1. --list-modules: lists all available modules.
  2. --list-classes: lists all available classes and the tests they contain.
  3. --list-tests-by <name|class|module|id|tag>: will list the tests that match the searching criteria and provide additional details like: id, tag, name, class, module for each found test.

Listing used tags:

--list-tags

Listing all tests:

--list-tests

Running specific tests

First you need to identify the test(s) that you want to run. You will need the following information: test module, test class, test method name. Look for the test modules inside /meta/lib/oeqa/selftest/ . After you have obtained the above information, issue from the command line: 'oe-selftest module1.Class1.test_1 module1.Class1.test_2 module2.Class2.test_3 ...' .


There are 2 methods for running specific tests:

Note:

  • Keep in mind this structure: module.class.test
  • The order the tests are running is alphabetical.
  1. --run-tests:
    -–run-tests moduleA moduleB
    will run all tests from this two modules
    –-run-tests moduleA.classA module.classB
    will run all tests from this two classes
    –-run-tests moduleA.classA.test_A module.classB.test_B
    will run these two tests
    You can mix and match modules, classes and test names:
    -–run-tests moduleA moduleB.classB moduleC.classC.testC
    will run all tests from moduleA, all tests from moduleB.classB and testC from moduleC.classC
  2. -–run-tests-by <name|class|module|id|tag>
    name
    --run-tests-by name testA testB testC: simply specify test names and it will determine automatically where they are located.
    class
    --run-tests-by class classA classB classC: simply specify class names and it will determine automatically what tests they contain and where they are located.
    module
    --run-tests-by module moduleA moduleB: simply specify module names and it will determine automatically what tests they contain.
    id
    --run-tests-by id id1 id2 id3: simply specify test ids and it will determine automatically what tests the ids point to.
    tag
    --run-tests-by tag tag1 tag2: if tests are tagged with @tag(feature=<>) you can run them like this also.

For the previous running methods you can specify the –-machine option to tell oe-selftest to run tests with custom MACHINE:

  • -–machine random: will run tests switching to a random MACHINE for each test.
  • -–machine all: will run every test from the list against all available MACHINES.


Running hidden tests

Hidden tests are located in test modules whose names begin with an underscore '_' . Example: '_test_hidden_module.py'. These tests can be run by specifying them as arguments like the above. Example: 'oe-selftest _test_hidden_module.Class_1.test_1'

Important file locations

The following files and locations are relevant to oe-selftest:

  • the script itself: oe-selftest

is located in the /scripts directory under the main OE-Core tree.

  • the case.py module

is located in the meta/lib/oeqa/selftest directory and contains the base OESelftestTestCase class that extends unittest.TestCase (via OETestCase) and adds extra functionality.

  • the test modules

are located in the meta/lib/oeqa/selftest/cases directory and contain either test classes with test methods or evolved versions of the oeSelfTest class with added specific functionality and being used by the aforementioned test classes.

  • the 'utils' library

is located at meta/lib/oeqa/utils and contains modules with helper methods that are used and imported by the test modules.

Basic architecture information

When designing oe-selftest we wanted to be less intrusive as possible. Doing so we have implemented the following conventions, using the extended oeSelfTest class:

appending to the build's global configuration

In order to keep appending to existing global configuration files(local.conf, auto.conf, site.conf) to a minimum, when oe-selftest starts it adds 'include selftest.inc' to local.conf and removes it at the end. The tests manipulate the global configuration by manipulating selftest.inc using the built-in methods write_config(), append_config(), remove_config(). Also the file selftest.inc is deleted before each test; this way all tests are independent from a global configuration point of view.

adding test files(like test classes) to the build

In order not to modify the existing metadata structure, a new layer 'meta-selftest' has been created for testing purposes. This layer can be used to store files used for testing and thus remove the need to modify the structure of the existing metadata or the need to create them at runtime.

appending to a recipe

In order to avoid modifying existing recipes and related files (.inc, .bbappend) a select list of recipes has a corresponding <recipe_name>.bbappend in the meta-selftest layer that contains only 1 passable line: "include test_recipe.inc". If a test needs to append configuration to a specific recipe, it only needs to manipulate the recipe's test_recipe.inc using the built-in methods write_recipeinc(), append_recipeinc(), remove_recipeinc() and delete_recipeinc(). Note that before each test starts all test_recipe.inc files are removed from the meta-selftest layer in order to keep all tests independent from a recipe-level configuration point of view.

Cleaning up

We also wanted to make cleanup and post-test actions simple so we added the following helper methods, using the extended oeSelfTest class:

  • add_command_to_tearDown(command)

Adds <command> as a shell command to be run after the test ends, no matter how the test exits (error or fail).

  • track_for_cleanup(path)

Adds <path> to a list of paths to be cleaned up (deleted) after the test ends, no matter how the test exits (error or fail).

Please be careful when using these methods as they have no safety checks.

You should be aware of the following

  • oe-selftest uses the settings in conf/local.conf as a base for all test configurations. Any modifications made (such as DISTRO) or not made here (e.g. BB_NUMBER_THREADS) will impact all tests if the modifications are not overwritten specifically by a test.
  • running all tests can take a long time! If you're modifying a particular feature/script then you may want to run just the related test module(s) or case(s) to save time.

How to contribute

You can send patches containing either improvements or tests to the openembedded-core mailing list: openembedded-core@lists.openembedded.org . Here is a guide on submitting changes.