Oe-selftest - Revision history

Vyacheslav Yurkov at 06:48, 19 February 2024

2024-02-19T06:48:56Z

← Older revision		Revision as of 06:48, 19 February 2024
Line 13:		Line 13:
	# add the meta-selftest layer in conf/bblayers.conf		# add the meta-selftest layer in conf/bblayers.conf
	# add SANITY_TESTED_DISTROS="" to local.conf		# add SANITY_TESTED_DISTROS="" to local.conf

			Some selftests require qemu support, which in turns sets up internal network to perform the tests. This is a privileged operation, so make sure the user you run the tests with has required privileges. Or for simplicity extend your ''/etc/sudoers'' to allow running following commands:
			* poky/scripts/runqemu-ifup
			* poky/scripts/runqemu-ifdown
			* poky/scripts/runqemu-gen-tapdevs

	== Running all (except hidden) tests ==		== Running all (except hidden) tests ==

Yeoh, Ee Peng at 06:04, 19 March 2018

2018-03-19T06:04:55Z

← Older revision		Revision as of 06:04, 19 March 2018
Line 12:		Line 12:
	# source the build environment setup script		# source the build environment setup script
	# add the meta-selftest layer in conf/bblayers.conf		# add the meta-selftest layer in conf/bblayers.conf
			# add SANITY_TESTED_DISTROS="" to local.conf

	== Running all (except hidden) tests ==		== Running all (except hidden) tests ==

PaulEggleton: Update

2018-01-11T19:46:05Z

Update

← Older revision		Revision as of 19:46, 11 January 2018
Line 65:		Line 65:
	* the script itself: oe-selftest		* the script itself: oe-selftest
	is located in the /scripts directory under the main OE-Core tree.		is located in the /scripts directory under the main OE-Core tree.
	* the ~~base~~.py module		* the case.py module
	is located in the meta/lib/oeqa/selftest directory and contains the base ~~oeSelfTest~~ class that extends unittest.TestCase and adds extra functionality.		is located in the meta/lib/oeqa/selftest directory and contains the base OESelftestTestCase class that extends unittest.TestCase (via OETestCase) and adds extra functionality.
	* the test modules		* the test modules
	are located in the meta/lib/oeqa/selftest directory and contain either test classes with test methods or evolved versions of the oeSelfTest class with added specific functionality and being used by the aforementioned test classes.		are located in the meta/lib/oeqa/selftest/cases directory and contain either test classes with test methods or evolved versions of the oeSelfTest class with added specific functionality and being used by the aforementioned test classes.
	* the 'utils' library		* the 'utils' library
	is located at meta/lib/oeqa/utils and contains modules with helper methods that are used and imported by the test modules.		is located at meta/lib/oeqa/utils and contains modules with helper methods that are used and imported by the test modules.
Line 94:		Line 94:
	=You should be aware of the following=		=You should be aware of the following=
	* oe-selftest uses the settings in conf/local.conf as a base for all test configurations. Any modifications made (such as DISTRO) or not made here (e.g. BB_NUMBER_THREADS) will impact all tests if the modifications are not overwritten specifically by a test.		* oe-selftest uses the settings in conf/local.conf as a base for all test configurations. Any modifications made (such as DISTRO) or not made here (e.g. BB_NUMBER_THREADS) will impact all tests if the modifications are not overwritten specifically by a test.
	* running all tests can take a long time!		* running all tests can take a long time! If you're modifying a particular feature/script then you may want to run just the related test module(s) or case(s) to save time.

	=How to contribute=		=How to contribute=
	You can send patches containing either improvements or tests to the openembedded-core mailing list: openembedded-core@lists.openembedded.org . Here is a [http://www.openembedded.org/wiki/How_to_submit_a_patch_to_OpenEmbedded guide on submitting changes].		You can send patches containing either improvements or tests to the openembedded-core mailing list: openembedded-core@lists.openembedded.org . Here is a [http://www.openembedded.org/wiki/How_to_submit_a_patch_to_OpenEmbedded guide on submitting changes].

Jussi Kukkonen: use hyphens not dashes for command line

2016-05-17T12:52:42Z

use hyphens not dashes for command line

← Older revision		Revision as of 12:52, 17 May 2016
Line 18:		Line 18:

	==Listing tests==		==Listing tests==
	#-~~–list~~-modules: lists all available modules.		#--list-modules: lists all available modules.
	#-~~–list~~-classes: lists all available classes and the tests they contain.		#--list-classes: lists all available classes and the tests they contain.
	#-~~–list~~-tests-by <name\|class\|module\|id\|tag>: will list the tests that match the searching criteria and provide additional details like: id, tag, name, class, module for each found test.		#--list-tests-by <name\|class\|module\|id\|tag>: will list the tests that match the searching criteria and provide additional details like: id, tag, name, class, module for each found test.

	Listing used tags:		Listing used tags:

Francisco Pedraza: /* Running all (except hidden) tests */

2016-04-02T00:21:47Z

Running all (except hidden) tests

← Older revision		Revision as of 00:21, 2 April 2016
Line 15:		Line 15:
	== Running all (except hidden) tests ==		== Running all (except hidden) tests ==
	To run all oe-selftest tests simply issue:		To run all oe-selftest tests simply issue:
	$ oe-selftest ~~–run~~-all-tests		$ oe-selftest --run-all-tests


	==Listing tests==		==Listing tests==

Alexandru.c.georgescu: /* Listing tests */

2016-01-08T09:25:11Z

Listing tests

← Older revision		Revision as of 09:25, 8 January 2016
Line 25:		Line 25:
	Listing used tags:		Listing used tags:
	--list-tags		--list-tags

			Listing all tests:
			--list-tests

	== Running specific tests ==		== Running specific tests ==

Alexandru.c.georgescu at 12:32, 6 January 2016

2016-01-06T12:32:33Z

← Older revision		Revision as of 12:32, 6 January 2016
Line 16:		Line 16:
	To run all oe-selftest tests simply issue:		To run all oe-selftest tests simply issue:
	$ oe-selftest –run-all-tests		$ oe-selftest –run-all-tests


			==Listing tests==
			#-–list-modules: lists all available modules.
			#-–list-classes: lists all available classes and the tests they contain.
			#-–list-tests-by <name\|class\|module\|id\|tag>: will list the tests that match the searching criteria and provide additional details like: id, tag, name, class, module for each found test.

			Listing used tags:
			--list-tags

	== Running specific tests ==		== Running specific tests ==
Line 45:		Line 54:
	*-–machine all: will run every test from the list against all available MACHINES.		*-–machine all: will run every test from the list against all available MACHINES.

	~~=Listing tests=~~
	~~#-–list-modules: lists all available modules.~~
	~~#-–list-classes: lists all available classes and the tests they contain.~~
	~~#-–list-tests-by <name\|class\|module\|id\|tag>: will list the tests that match the searching criteria and provide additional details like: id, tag, name, class, module for each found test.~~

	== Running hidden tests ==		== Running hidden tests ==

Alexandru.c.georgescu: /* Running specific tests */

2016-01-06T12:30:43Z

Running specific tests

@@ Line 21: / Line 21: @@
 Look for the test modules inside /meta/lib/oeqa/selftest/ .
 After you have obtained the above information, issue from the command line: 'oe-selftest module1.Class1.test_1 module1.Class1.test_2 module2.Class2.test_3 ...' .
 == Running hidden tests ==

Alexandru.c.georgescu: /* Running all (except hidden) tests */

2016-01-06T12:21:30Z

Running all (except hidden) tests

← Older revision		Revision as of 12:21, 6 January 2016
Line 14:		Line 14:

	== Running all (except hidden) tests ==		== Running all (except hidden) tests ==
	~~From the source build directory,~~ issue ~~the command '~~oe-selftest'.		To run all oe-selftest tests simply issue:
			$ oe-selftest –run-all-tests

	== Running specific tests ==		== Running specific tests ==

PaulEggleton: Formatting / grammar / etc. fixes

2015-02-24T16:34:09Z

Formatting / grammar / etc. fixes

← Older revision		Revision as of 16:34, 24 February 2015
Line 1:		Line 1:
	=Introduction=		=Introduction=
	Oe-selftest is a test framework used for testing the ~~poky~~ build system. Following are some key points in describing oe-selftest:		Oe-selftest is a test framework used for testing the OpenEmbedded build system. Following are some key points in describing oe-selftest:
	* based on Python ~~Unit Testing~~		* based on Python [http://docs.python.org/2/library/unittest.html unittest]
	* designed to simulate ~~poky external~~ usage patterns		* designed to simulate normal usage patterns
	* does not cover image run-time testing		* does not cover image run-time testing
	* implements a new layer that contains generic/specific metadata used only by tests		* implements a new layer that contains generic/specific metadata used only by tests
Line 9:		Line 9:

	In order to use oe-selftest, you need the following(in this specific order):		In order to use oe-selftest, you need the following(in this specific order):
	# ~~the~~ poky ~~source~~ (~~1.5 M2~~ or ~~better~~)		# Download poky / openembedded-core (dora or newer branch)
	# source the build environment		# source the build environment setup script
	# add the meta-selftest layer in conf/bblayers.conf		# add the meta-selftest layer in conf/bblayers.conf

			== Running all (except hidden) tests ==
	* How to run all(except hidden) tests:
	From the source build directory, issue the command 'oe-selftest'.		From the source build directory, issue the command 'oe-selftest'.

	* How to run specific tests		== Running specific tests ==
	First you need to identify the test(s) that you want to run. You will need the following information: test module, test class, test method name.		First you need to identify the test(s) that you want to run. You will need the following information: test module, test class, test method name.
	Look for the test modules inside /meta/lib/oeqa/selftest/ .		Look for the test modules inside /meta/lib/oeqa/selftest/ .
	After you have obtained the above information, issue from the command line: 'oe-selftest module1.Class1.test_1 module1.Class1.test_2 module2.Class2.test_3 ...' .		After you have obtained the above information, issue from the command line: 'oe-selftest module1.Class1.test_1 module1.Class1.test_2 module2.Class2.test_3 ...' .

	* How to run hidden tests		== Running hidden tests ==
	Hidden tests are located in test modules whose names begin with an underscore '_' . Example: '_test_hidden_module.py'. These tests can be run by specifying them as arguments like the above.		Hidden tests are located in test modules whose names begin with an underscore '_' . Example: '_test_hidden_module.py'. These tests can be run by specifying them as arguments like the above.
	Example: 'oe-selftest _test_hidden_module.Class_1.test_1'		Example: 'oe-selftest _test_hidden_module.Class_1.test_1'
Line 28:		Line 27:
	=Important file locations=		=Important file locations=
	The following files and locations are relevant to oe-selftest:		The following files and locations are relevant to oe-selftest:
	* the script itself: oe-selftest~~.py~~		* the script itself: oe-selftest
	is located in the /scripts directory under the main ~~poky~~ tree.		is located in the /scripts directory under the main OE-Core tree.
	* the base.py module		* the base.py module
	is located in the meta/lib/oeqa/selftest directory and contains the base oeSelfTest class that extends unittest.TestCase and adds extra functionality.		is located in the meta/lib/oeqa/selftest directory and contains the base oeSelfTest class that extends unittest.TestCase and adds extra functionality.
Line 38:		Line 37:

	=Basic architecture information=		=Basic architecture information=

	When designing oe-selftest we wanted to be less intrusive as possible. Doing so we have implemented the following conventions, using the extended oeSelfTest class:		When designing oe-selftest we wanted to be less intrusive as possible. Doing so we have implemented the following conventions, using the extended oeSelfTest class:
	* appending to the build's global configuration:		== appending to the build's global configuration ==
	In order to keep appending to existing global configuration files(local.conf, auto.conf, site.conf) to a minimum, when oe-selftest starts it adds 'include selftest.inc' to local.conf and removes it at the end. The tests manipulate the global configuration by manipulating selftest.inc using the built-in methods write_config(), append_config(), remove_config(). Also the file selftest.inc is deleted before each test; this way all tests are independent from a global configuration point of view.		In order to keep appending to existing global configuration files(local.conf, auto.conf, site.conf) to a minimum, when oe-selftest starts it adds 'include selftest.inc' to local.conf and removes it at the end. The tests manipulate the global configuration by manipulating selftest.inc using the built-in methods write_config(), append_config(), remove_config(). Also the file selftest.inc is deleted before each test; this way all tests are independent from a global configuration point of view.
	* adding test files(like test classes) to the build:		== adding test files(like test classes) to the build ==
	In order not to modify the existing metadata structure, a new layer 'meta-selftest' has been created for testing purposes. This layer can be used to store files used for testing and thus remove the need to modify the structure of the existing metadata or the need to create them at runtime.		In order not to modify the existing metadata structure, a new layer 'meta-selftest' has been created for testing purposes. This layer can be used to store files used for testing and thus remove the need to modify the structure of the existing metadata or the need to create them at runtime.
	* appending to a recipe:		== appending to a recipe ==
	In order to ~~keep from~~ modifying existing recipes and related files(.inc, .bbappend) a select list of recipes has a corresponding <recipe_name>.bbappend in the meta-selftest layer that contains only 1 passable line: "include test_recipe.inc". If a test needs to append configuration to a specific recipe, it only needs to manipulate the recipe's test_recipe.inc using the built-in methods write_recipeinc(), append_recipeinc(), remove_recipeinc() and delete_recipeinc(). Note that before each test starts all test_recipe.inc files are removed from the meta-selftest layer in order to keep all tests independent from a recipe-level configuration point of view.		In order to avoid modifying existing recipes and related files (.inc, .bbappend) a select list of recipes has a corresponding <recipe_name>.bbappend in the meta-selftest layer that contains only 1 passable line: "include test_recipe.inc". If a test needs to append configuration to a specific recipe, it only needs to manipulate the recipe's test_recipe.inc using the built-in methods write_recipeinc(), append_recipeinc(), remove_recipeinc() and delete_recipeinc(). Note that before each test starts all test_recipe.inc files are removed from the meta-selftest layer in order to keep all tests independent from a recipe-level configuration point of view.

			== Cleaning up ==

	We also wanted to make cleanup and post-test actions simple so we added the following helper methods, using the extended oeSelfTest class:		We also wanted to make cleanup and post-test actions simple so we added the following helper methods, using the extended oeSelfTest class:
	* add_command_to_tearDown(command)		* add_command_to_tearDown(command)
	Adds <command> as a shell command to be run after the test ends, no matter how the test ~~exists~~(error or fail).		Adds <command> as a shell command to be run after the test ends, no matter how the test exits (error or fail).
	* track_for_cleanup(path)		* track_for_cleanup(path)
	Adds <path> to a list of paths to be cleaned up(deleted) after the test ends, no matter how the test ~~exists~~(error or fail).		Adds <path> to a list of paths to be cleaned up (deleted) after the test ends, no matter how the test exits (error or fail).

	Please be careful when using these methods as they have no safety checks.		Please be careful when using these methods as they have no safety checks.


	=You should be aware of the following=		=You should be aware of the following=
	* oe-selftest uses the settings in conf/local.conf as a base for all test configurations. Any modifications made(~~like~~ DISTRO) or not made here(~~like~~ BB_NUMBER_THREADS) will impact all tests if the modifications are not overwritten specifically by a test.		* oe-selftest uses the settings in conf/local.conf as a base for all test configurations. Any modifications made (such as DISTRO) or not made here (e.g. BB_NUMBER_THREADS) will impact all tests if the modifications are not overwritten specifically by a test.
	* running all tests can take a long time!		* running all tests can take a long time!

	=How to contribute=		=How to contribute=
	You can send patches containing either improvements or tests to the openembedded mailing list: openembedded-core@lists.openembedded.org . Here is a ~~link to a guide on doing so:~~ http://www.openembedded.org/wiki/How_to_submit_a_patch_to_OpenEmbedded .		You can send patches containing either improvements or tests to the openembedded-core mailing list: openembedded-core@lists.openembedded.org . Here is a [http://www.openembedded.org/wiki/How_to_submit_a_patch_to_OpenEmbedded guide on submitting changes].