AutoBuilder/Adding Automated Tests

Preparation

Adding automated tests to the AutoBuilder assumes that the testing is done in either of the following formats:

• OpenEmbedded Selftest format
• Bitbake Runtime tests format

The AutoBuilder interfaces with the test tool via a calling command. Such test calling command is used to start executing a given set of tests. The test set description can come in the way of a configuration (file, variable, argument, etc) and it ultimately explicitly lists which tests are to be run. It is an expectation of the test tool that it procures the execution of the list of tests that has been specified at the moment of calling the starting command.

Sample Test for Instructions

For the purpose of this guide, the following test calling command will be used:

$ oe-selftest -r bbtests.BitbakeTests.test_warnings_errors

The test tool used is in the OpenEmbedded Selftest format and the calling line selects a single test from the BBTests suite. The reason for this choice is merely simplicity for writing this guide.

You can see the code of this particular test here (referencing the Krogoth release):

http://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/meta/lib/oeqa/selftest/bbtests.py?id=krogoth-15.0.0#n58

There should be about 5 lines of code there, the test can be simply explained as: execute a bitbake build from a file that doesn't exist and verify that it shows an error. It's a negative test.

Sample Test Output

Getting familiar with the output of the test is a good idea. Preparing the environment for running such test can be seen at the Oe-selftest#How_to_use section.

Once the test is run at the command line, the following terminal output is presented:

$ oe-selftest -r bbtests.BitbakeTests.test_warnings_errors
2016-06-15 13:55:30,964 - selftest - INFO - Running bitbake -e to get BBPATH
2016-06-15 13:55:32,027 - selftest - INFO - Checking that everything is in order before running the tests
2016-06-15 13:55:33,073 - selftest - INFO - Running bitbake -p
2016-06-15 13:55:36,228 - selftest - INFO - Loading tests from: oeqa.selftest.bbtests.BitbakeTests.test_warnings_errors
2016-06-15 13:55:36,230 - selftest - INFO - Adding: "include selftest.inc" in local.conf
2016-06-15 13:55:36,230 - selftest - INFO - Adding: "include bblayers.inc" in bblayers.conf
2016-06-15 13:55:36 - test_warnings_errors (oeqa.selftest.bbtests.BitbakeTests) ... ok

----------------------------------------------------------------------
Ran 1 test in 0.782s

OK
2016-06-15 13:55:37,012 - selftest - INFO - Finished
2016-06-15 13:55:37,012 - selftest - INFO - Removing the include from local.conf
2016-06-15 13:55:37,012 - selftest - INFO - Removing the include from bblayers.conf

an AutoBuilder to Run the Test

An AutoBuilder will be needed to run the testing, for letting the AutoBuilder know what you want to test, a config-set is required. Once you create the config-set, an AutoBuilder is needed to test and run it when finished.