Oe-selftest: Difference between revisions
PaulEggleton (talk | contribs) (Formatting / grammar / etc. fixes) |
|||
| Line 1: | Line 1: | ||
=Introduction= | =Introduction= | ||
Oe-selftest is a test framework used for testing the | Oe-selftest is a test framework used for testing the OpenEmbedded build system. Following are some key points in describing oe-selftest: | ||
* based on Python | * based on Python [http://docs.python.org/2/library/unittest.html unittest] | ||
* designed to simulate | * designed to simulate normal usage patterns | ||
* does not cover image run-time testing | * does not cover image run-time testing | ||
* implements a new layer that contains generic/specific metadata used only by tests | * implements a new layer that contains generic/specific metadata used only by tests | ||
| Line 9: | Line 9: | ||
In order to use oe-selftest, you need the following(in this specific order): | In order to use oe-selftest, you need the following(in this specific order): | ||
# | # Download poky / openembedded-core (dora or newer branch) | ||
# source the build environment | # source the build environment setup script | ||
# add the meta-selftest layer in conf/bblayers.conf | # add the meta-selftest layer in conf/bblayers.conf | ||
== Running all (except hidden) tests == | |||
From the source build directory, issue the command 'oe-selftest'. | From the source build directory, issue the command 'oe-selftest'. | ||
== Running specific tests == | |||
First you need to identify the test(s) that you want to run. You will need the following information: test module, test class, test method name. | First you need to identify the test(s) that you want to run. You will need the following information: test module, test class, test method name. | ||
Look for the test modules inside /meta/lib/oeqa/selftest/ . | Look for the test modules inside /meta/lib/oeqa/selftest/ . | ||
After you have obtained the above information, issue from the command line: 'oe-selftest module1.Class1.test_1 module1.Class1.test_2 module2.Class2.test_3 ...' . | After you have obtained the above information, issue from the command line: 'oe-selftest module1.Class1.test_1 module1.Class1.test_2 module2.Class2.test_3 ...' . | ||
== Running hidden tests == | |||
Hidden tests are located in test modules whose names begin with an underscore '_' . Example: '_test_hidden_module.py'. These tests can be run by specifying them as arguments like the above. | Hidden tests are located in test modules whose names begin with an underscore '_' . Example: '_test_hidden_module.py'. These tests can be run by specifying them as arguments like the above. | ||
Example: 'oe-selftest _test_hidden_module.Class_1.test_1' | Example: 'oe-selftest _test_hidden_module.Class_1.test_1' | ||
| Line 28: | Line 27: | ||
=Important file locations= | =Important file locations= | ||
The following files and locations are relevant to oe-selftest: | The following files and locations are relevant to oe-selftest: | ||
* the script itself: oe-selftest | * the script itself: oe-selftest | ||
is located in the /scripts directory under the main | is located in the /scripts directory under the main OE-Core tree. | ||
* the base.py module | * the base.py module | ||
is located in the meta/lib/oeqa/selftest directory and contains the base oeSelfTest class that extends unittest.TestCase and adds extra functionality. | is located in the meta/lib/oeqa/selftest directory and contains the base oeSelfTest class that extends unittest.TestCase and adds extra functionality. | ||
| Line 38: | Line 37: | ||
=Basic architecture information= | =Basic architecture information= | ||
When designing oe-selftest we wanted to be less intrusive as possible. Doing so we have implemented the following conventions, using the extended oeSelfTest class: | When designing oe-selftest we wanted to be less intrusive as possible. Doing so we have implemented the following conventions, using the extended oeSelfTest class: | ||
== appending to the build's global configuration == | |||
In order to keep appending to existing global configuration files(local.conf, auto.conf, site.conf) to a minimum, when oe-selftest starts it adds 'include selftest.inc' to local.conf and removes it at the end. The tests manipulate the global configuration by manipulating selftest.inc using the built-in methods write_config(), append_config(), remove_config(). Also the file selftest.inc is deleted before each test; this way all tests are independent from a global configuration point of view. | In order to keep appending to existing global configuration files(local.conf, auto.conf, site.conf) to a minimum, when oe-selftest starts it adds 'include selftest.inc' to local.conf and removes it at the end. The tests manipulate the global configuration by manipulating selftest.inc using the built-in methods write_config(), append_config(), remove_config(). Also the file selftest.inc is deleted before each test; this way all tests are independent from a global configuration point of view. | ||
== adding test files(like test classes) to the build == | |||
In order not to modify the existing metadata structure, a new layer 'meta-selftest' has been created for testing purposes. This layer can be used to store files used for testing and thus remove the need to modify the structure of the existing metadata or the need to create them at runtime. | In order not to modify the existing metadata structure, a new layer 'meta-selftest' has been created for testing purposes. This layer can be used to store files used for testing and thus remove the need to modify the structure of the existing metadata or the need to create them at runtime. | ||
== appending to a recipe == | |||
In order to | In order to avoid modifying existing recipes and related files (.inc, .bbappend) a select list of recipes has a corresponding <recipe_name>.bbappend in the meta-selftest layer that contains only 1 passable line: "include test_recipe.inc". If a test needs to append configuration to a specific recipe, it only needs to manipulate the recipe's test_recipe.inc using the built-in methods write_recipeinc(), append_recipeinc(), remove_recipeinc() and delete_recipeinc(). Note that before each test starts all test_recipe.inc files are removed from the meta-selftest layer in order to keep all tests independent from a recipe-level configuration point of view. | ||
== Cleaning up == | |||
We also wanted to make cleanup and post-test actions simple so we added the following helper methods, using the extended oeSelfTest class: | We also wanted to make cleanup and post-test actions simple so we added the following helper methods, using the extended oeSelfTest class: | ||
* add_command_to_tearDown(command) | * add_command_to_tearDown(command) | ||
Adds <command> as a shell command to be run after the test ends, no matter how the test | Adds <command> as a shell command to be run after the test ends, no matter how the test exits (error or fail). | ||
* track_for_cleanup(path) | * track_for_cleanup(path) | ||
Adds <path> to a list of paths to be cleaned up(deleted) after the test ends, no matter how the test | Adds <path> to a list of paths to be cleaned up (deleted) after the test ends, no matter how the test exits (error or fail). | ||
Please be careful when using these methods as they have no safety checks. | Please be careful when using these methods as they have no safety checks. | ||
=You should be aware of the following= | =You should be aware of the following= | ||
* oe-selftest uses the settings in conf/local.conf as a base for all test configurations. Any modifications made( | * oe-selftest uses the settings in conf/local.conf as a base for all test configurations. Any modifications made (such as DISTRO) or not made here (e.g. BB_NUMBER_THREADS) will impact all tests if the modifications are not overwritten specifically by a test. | ||
* running all tests can take a long time! | * running all tests can take a long time! | ||
=How to contribute= | =How to contribute= | ||
You can send patches containing either improvements or tests to the openembedded mailing list: openembedded-core@lists.openembedded.org . Here is a | You can send patches containing either improvements or tests to the openembedded-core mailing list: openembedded-core@lists.openembedded.org . Here is a [http://www.openembedded.org/wiki/How_to_submit_a_patch_to_OpenEmbedded guide on submitting changes]. | ||
Revision as of 16:34, 24 February 2015
Introduction
Oe-selftest is a test framework used for testing the OpenEmbedded build system. Following are some key points in describing oe-selftest:
- based on Python unittest
- designed to simulate normal usage patterns
- does not cover image run-time testing
- implements a new layer that contains generic/specific metadata used only by tests
How to use
In order to use oe-selftest, you need the following(in this specific order):
- Download poky / openembedded-core (dora or newer branch)
- source the build environment setup script
- add the meta-selftest layer in conf/bblayers.conf
From the source build directory, issue the command 'oe-selftest'.
Running specific tests
First you need to identify the test(s) that you want to run. You will need the following information: test module, test class, test method name. Look for the test modules inside /meta/lib/oeqa/selftest/ . After you have obtained the above information, issue from the command line: 'oe-selftest module1.Class1.test_1 module1.Class1.test_2 module2.Class2.test_3 ...' .
Hidden tests are located in test modules whose names begin with an underscore '_' . Example: '_test_hidden_module.py'. These tests can be run by specifying them as arguments like the above. Example: 'oe-selftest _test_hidden_module.Class_1.test_1'
Important file locations
The following files and locations are relevant to oe-selftest:
- the script itself: oe-selftest
is located in the /scripts directory under the main OE-Core tree.
- the base.py module
is located in the meta/lib/oeqa/selftest directory and contains the base oeSelfTest class that extends unittest.TestCase and adds extra functionality.
- the test modules
are located in the meta/lib/oeqa/selftest directory and contain either test classes with test methods or evolved versions of the oeSelfTest class with added specific functionality and being used by the aforementioned test classes.
- the 'utils' library
is located at meta/lib/oeqa/utils and contains modules with helper methods that are used and imported by the test modules.
Basic architecture information
When designing oe-selftest we wanted to be less intrusive as possible. Doing so we have implemented the following conventions, using the extended oeSelfTest class:
appending to the build's global configuration
In order to keep appending to existing global configuration files(local.conf, auto.conf, site.conf) to a minimum, when oe-selftest starts it adds 'include selftest.inc' to local.conf and removes it at the end. The tests manipulate the global configuration by manipulating selftest.inc using the built-in methods write_config(), append_config(), remove_config(). Also the file selftest.inc is deleted before each test; this way all tests are independent from a global configuration point of view.
adding test files(like test classes) to the build
In order not to modify the existing metadata structure, a new layer 'meta-selftest' has been created for testing purposes. This layer can be used to store files used for testing and thus remove the need to modify the structure of the existing metadata or the need to create them at runtime.
appending to a recipe
In order to avoid modifying existing recipes and related files (.inc, .bbappend) a select list of recipes has a corresponding <recipe_name>.bbappend in the meta-selftest layer that contains only 1 passable line: "include test_recipe.inc". If a test needs to append configuration to a specific recipe, it only needs to manipulate the recipe's test_recipe.inc using the built-in methods write_recipeinc(), append_recipeinc(), remove_recipeinc() and delete_recipeinc(). Note that before each test starts all test_recipe.inc files are removed from the meta-selftest layer in order to keep all tests independent from a recipe-level configuration point of view.
Cleaning up
We also wanted to make cleanup and post-test actions simple so we added the following helper methods, using the extended oeSelfTest class:
- add_command_to_tearDown(command)
Adds <command> as a shell command to be run after the test ends, no matter how the test exits (error or fail).
- track_for_cleanup(path)
Adds <path> to a list of paths to be cleaned up (deleted) after the test ends, no matter how the test exits (error or fail).
Please be careful when using these methods as they have no safety checks.
You should be aware of the following
- oe-selftest uses the settings in conf/local.conf as a base for all test configurations. Any modifications made (such as DISTRO) or not made here (e.g. BB_NUMBER_THREADS) will impact all tests if the modifications are not overwritten specifically by a test.
- running all tests can take a long time!
How to contribute
You can send patches containing either improvements or tests to the openembedded-core mailing list: openembedded-core@lists.openembedded.org . Here is a guide on submitting changes.
