Image tests - Revision history

Stefans at 15:15, 4 December 2013

2013-12-04T15:15:54Z

← Older revision		Revision as of 15:15, 4 December 2013
Line 25:		Line 25:
	=== Known bugs/limitations ===		=== Known bugs/limitations ===

	* for builds where rm_work was used most of the tests will error out because they rely on ${WORKDIR}/installed_pkgs.txt (see https://bugzilla.yoctoproject.org/show_bug.cgi?id=5072 )		* for builds where rm_work was used most of the tests will error out because they rely on ${WORKDIR}/installed_pkgs.txt (see https://bugzilla.yoctoproject.org/show_bug.cgi?id=5072 ) (this is fixed in master, 1.6_M1.rc1)

	=== Usage ===		=== Usage ===

Stefans at 11:20, 13 September 2013

2013-09-13T11:20:37Z

Stefans at 11:01, 13 September 2013

2013-09-13T11:01:32Z

Show changes

Stefans at 10:40, 10 September 2013

2013-09-10T10:40:19Z

← Older revision		Revision as of 10:40, 10 September 2013
Line 344:		Line 344:
	== Writing new tests ==		== Writing new tests ==

	All new test ~~modules~~ should go in meta/lib/oeqa/runtime ~~(or~~ <meta-layer>/lib/oeqa/runtime ~~if that~~ test ~~is meant for a certain layer only~~).		All new test files should go in meta/lib/oeqa/runtime. The file names themselves are the actual tests names we use, also called test modules.
	You ~~could~~ start by copying an existing module (syslog.py or gcc.py) are good examples and go from there.		A layer can add its own tests in <meta-layer>/lib/oeqa/runtime, provided it extends BBPATH as normal in its layer.conf (test module names shouldn't collide though with those in core).

			You should start by copying an existing module, e.g syslog.py or gcc.py are good examples, and go from there.

Stefans at 10:36, 10 September 2013

2013-09-10T10:36:50Z

Show changes

Stefans at 10:24, 10 September 2013

2013-09-10T10:24:03Z

← Older revision		Revision as of 10:24, 10 September 2013
Line 42:		Line 42:
	Note that the '''order in TEST_SUITES''' is important (it's the order modules run) and it influences tests dependencies. That means that tests that depend on other tests (e.g ssh depends on the ping test) should be added last (there is no re-ordering/dependency handling by the test class, it just respects the order). Each module can have multiple classes with multiple test methods (and Python unittest rules apply here).		Note that the '''order in TEST_SUITES''' is important (it's the order modules run) and it influences tests dependencies. That means that tests that depend on other tests (e.g ssh depends on the ping test) should be added last (there is no re-ordering/dependency handling by the test class, it just respects the order). Each module can have multiple classes with multiple test methods (and Python unittest rules apply here).

	~~=== Examples ===~~		In short:

	~~Examples:~~		* to run the default tests for core-image-sato you don't need to change TEST_SUITES (just call bitbake core-image-sato -c testimage like above)
	* to run the default tests for core-image-sato you don't need to change TEST_SUITES.
	* The default for core-image-sato is defined as: DEFAULT_TEST_SUITES_pn-core-image-sato = "ping ssh df connman syslog xorg scp vnc date rpm smart dmesg"		* The default for core-image-sato is defined as: DEFAULT_TEST_SUITES_pn-core-image-sato = "ping ssh df connman syslog xorg scp vnc date rpm smart dmesg"
	* to add your own test to the list of the defaults add: TEST_SUITES_append = " mytest"		* to add your own test to the list of the defaults add: TEST_SUITES_append = " mytest"
	* to run a specific list of tests: TEST_SUITES = "ping ssh rpm" (~~remember~~ order is important)		* to run a specific list of tests: TEST_SUITES = "ping ssh rpm" (order is important)

	== Good to know ==		== Good to know ==

Stefans at 10:21, 10 September 2013

2013-09-10T10:21:23Z

← Older revision		Revision as of 10:21, 10 September 2013
Line 42:		Line 42:
	Note that the '''order in TEST_SUITES''' is important (it's the order modules run) and it influences tests dependencies. That means that tests that depend on other tests (e.g ssh depends on the ping test) should be added last (there is no re-ordering/dependency handling by the test class, it just respects the order). Each module can have multiple classes with multiple test methods (and Python unittest rules apply here).		Note that the '''order in TEST_SUITES''' is important (it's the order modules run) and it influences tests dependencies. That means that tests that depend on other tests (e.g ssh depends on the ping test) should be added last (there is no re-ordering/dependency handling by the test class, it just respects the order). Each module can have multiple classes with multiple test methods (and Python unittest rules apply here).

	== Examples ==		=== Examples ===

	Examples:		Examples:
Line 49:		Line 49:
	* to add your own test to the list of the defaults add: TEST_SUITES_append = " mytest"		* to add your own test to the list of the defaults add: TEST_SUITES_append = " mytest"
	* to run a specific list of tests: TEST_SUITES = "ping ssh rpm" (remember order is important)		* to run a specific list of tests: TEST_SUITES = "ping ssh rpm" (remember order is important)

			== Good to know ==

	Once you call the testimage task (bitbake <my-image> -c testimage) a couple of things happen:		Once you call the testimage task (bitbake <my-image> -c testimage) a couple of things happen:

Stefans: /* Usage */

2013-09-10T10:17:10Z

Usage

← Older revision		Revision as of 10:17, 10 September 2013
Line 34:		Line 34:
	* then call "'''bitbake core-image-sato -c testimage'''". That will run a standard suite of tests.		* then call "'''bitbake core-image-sato -c testimage'''". That will run a standard suite of tests.

	All test files are currently in '''meta/lib/oeqa/runtime'''. The file names themselves are the actual tests names we use (also called test modules~~, because a~~ module can have multiple classes and test methods, usually grouped together by ~~functionality/~~area ~~tests,~~ e.g: tests for systemd go in meta/lib/oeqa/runtime/systemd.py)		All test files are currently in '''meta/lib/oeqa/runtime'''. The file names themselves are the actual tests names we use, also called test modules. A module can have multiple classes and test methods, usually grouped together by the area tested (e.g: tests for systemd go in meta/lib/oeqa/runtime/systemd.py).

	A layer can add its own tests in <meta-layer>/lib/oeqa/runtime, provided it extends BBPATH as normal in its layer.conf (test module names shouldn't collide though with those in core).		A layer can add its own tests in <meta-layer>/lib/oeqa/runtime, provided it extends BBPATH as normal in its layer.conf (test module names shouldn't collide though with those in core).

Stefans at 10:15, 10 September 2013

2013-09-10T10:15:39Z

← Older revision		Revision as of 10:15, 10 September 2013
Line 34:		Line 34:
	* then call "'''bitbake core-image-sato -c testimage'''". That will run a standard suite of tests.		* then call "'''bitbake core-image-sato -c testimage'''". That will run a standard suite of tests.

	All test ~~modules~~ are currently in meta/lib/oeqa/runtime.		All test files are currently in '''meta/lib/oeqa/runtime'''. The file names themselves are the actual tests names we use (also called test modules, because a module can have multiple classes and test methods, usually grouped together by functionality/area tests, e.g: tests for systemd go in meta/lib/oeqa/runtime/systemd.py)
	A layer can add its own tests in <meta-layer>/lib/oeqa/runtime, provided it extends BBPATH as normal in its layer.conf (test module names shouldn't collide though with those in core).		A layer can add its own tests in <meta-layer>/lib/oeqa/runtime, provided it extends BBPATH as normal in its layer.conf (test module names shouldn't collide though with those in core).

Stefans: /* Usage */

2013-09-10T10:11:40Z

Usage

← Older revision		Revision as of 10:11, 10 September 2013
Line 37:		Line 37:
	A layer can add its own tests in <meta-layer>/lib/oeqa/runtime, provided it extends BBPATH as normal in its layer.conf (test module names shouldn't collide though with those in core).		A layer can add its own tests in <meta-layer>/lib/oeqa/runtime, provided it extends BBPATH as normal in its layer.conf (test module names shouldn't collide though with those in core).

	You can change the tests run by '''appending or overrding the TEST_SUITES variable in local.conf'''. Each name in TEST_SUITES represents a required test for the image. That means that no skipping is allowed (even if the test isn't suitable for the image, e.g running the rpm tests on a ~~images with no~~ rpm). Appending "auto" to TEST_SUITES means that it will try to run all tests that are suitable for the image (each test decides that on it's own).		You can change the tests run by '''appending or overrding the TEST_SUITES variable in local.conf'''. Each name in TEST_SUITES represents a '''required test''' for the image. That means that no module skipping is allowed, even if the test isn't suitable for the image (e.g running the rpm tests on a image without rpm). Appending "auto" to TEST_SUITES means that it will try to run all tests that are suitable for the image (each test decides that on it's own).

	Note that the '''order in TEST_SUITES''' is important (it's the order modules run) and it influences tests dependencies. That means that tests that depend on other tests (e.g ssh depends on the ping test) should be added last (there is no re-ordering/dependency handling by the test class, it just respects the order). Each module can have multiple classes with multiple test methods (and Python unittest rules apply here).		Note that the '''order in TEST_SUITES''' is important (it's the order modules run) and it influences tests dependencies. That means that tests that depend on other tests (e.g ssh depends on the ping test) should be added last (there is no re-ordering/dependency handling by the test class, it just respects the order). Each module can have multiple classes with multiple test methods (and Python unittest rules apply here).

← Older revision		Revision as of 11:20, 13 September 2013
Line 344:		Line 344:
	== Writing new tests ==		== Writing new tests ==

	All new test files should go in meta/lib/oeqa/runtime. The file names themselves are the actual tests names we use, also called test modules.		All new test files should go in '''meta/lib/oeqa/runtime'''. The file names themselves are the actual tests names we use, also called test modules.
	A layer can add its own tests in <meta-layer>/lib/oeqa/runtime, provided it extends BBPATH as normal in its layer.conf (test module names shouldn't collide though with those in core).		A layer can add its own tests in '''<meta-layer>/lib/oeqa/runtime''', provided it extends BBPATH as normal in its layer.conf (test module names shouldn't collide though with those in core).

	Test modules are found in meta/lib/oeqa/runtime and they can use code from meta/lib/oeqa/utils which are helper classes for extra stuff (like starting an http server)		Test modules are found in '''meta/lib/oeqa/runtime''' and they can use code from '''meta/lib/oeqa/utils''' which are helper classes for extra stuff (like starting an http server)

	You should start by copying an existing module, e.g syslog.py or gcc.py are good examples, and go from there.		You should start by copying an existing module, e.g syslog.py or gcc.py are good examples, and go from there.

	You'll see that all test classes inherit oeRuntimeTest (found in meta/lib/oetest.py). This base class offers some helper attributes. Here's a short list:		You'll see that all test classes inherit '''oeRuntimeTest''' (found in '''meta/lib/oetest.py'''). This base class offers some helper attributes. Here's a short list:

	Class methods:		Class methods:
	* hasPackage(pkg): returns True if pkg is in the installed package list of the image (based on WORKDIR/installed_pkgs.txt which is generated at do.rootfs)		* hasPackage(pkg): returns True if pkg is in the installed package list of the image (based on WORKDIR/installed_pkgs.txt which is generated at do.rootfs)
	* hasFeature(feature): ~~return~~ True if feature is in IMAGE_FEATURES or DISTRO_FEATURES		* hasFeature(feature): returns True if feature is in IMAGE_FEATURES or DISTRO_FEATURES
	* restartTarget(params): restarts the qemu image optionally passing params to runqemu's qemuparams (e.g "-m 1024" for more memory)		* restartTarget(params): restarts the qemu image optionally passing params to runqemu's qemuparams (e.g "-m 1024" for more memory)

Line 360:		Line 361:
	* pscmd: equals "ps -ef" if procps is installed in the image else it's "ps" (busybox)		* pscmd: equals "ps -ef" if procps is installed in the image else it's "ps" (busybox)
	* tc: called text context, this gives access to other attributes:		* tc: called text context, this gives access to other attributes:
	** d: the bitbake data store ( so you can do stuff like oeRuntimeTest.tc.d.getVar("VIRTUAL-RUNTIME_init_manager")		** d: the bitbake data store ( so you can do stuff like '''oeRuntimeTest.tc.d.getVar("VIRTUAL-RUNTIME_init_manager")''')
	** testslist and testsrequired: used internally, tests shouldn't need them		** testslist and testsrequired: used internally, tests shouldn't need them
	** filesdir: absolute path to meta/lib/oeqa/runtime/files (which contains helper files for tests meant for copying on the target, like small .c files to get compiled)		** filesdir: absolute path to meta/lib/oeqa/runtime/files (which contains helper files for tests meant for copying on the target, like small .c files to get compiled)
	** qemu: accces to the QemuRunner object, the class that boots the image. Useful attributes:		** qemu: accces to the QemuRunner object, the class that boots the image. Useful attributes:
	*** ip: the machine's IP		*** ip: the machine's IP
	*** host_ip: host IP, only used in smart tests		*** host_ip: host IP, only used by smart tests
	*** other stuff not relevant for tests		*** other stuff not relevant for tests
	** target: SSHControl object, used for running commands on the image		** target: SSHControl object, used for running commands on the image
	*** host: same as qemu.ip, not really used tests		*** host: same as qemu.ip, used internally, not really used in tests
	*** timeout: global timeout for commands ran on the target for this instance (default: 300)		*** timeout: global timeout for commands ran on the target for this instance (default: 300).
	*** run(cmd, timeout=None): returns ~~the~~ status ~~and~~ output of cmd ~~which get~~'~~s run on the target~~. The optional timeout ~~arg specifies~~ for ~~how long should this command runs~~ (if None the default instance's timeout is used, if 0 it runs until the command returns)		*** '''run(cmd, timeout=None)''': The single most used method. Basically a wrapper for: 'ssh root@host "cmd"'. It returns a tuple: '''(status, output)''' which are what their names says: the return code of 'cmd' and whatever output that produces. The optional ''timeout'' argument represents the number of seconds it should wait for 'cmd' to return (if None the default instance's timeout is used which is 300 now, if 0 it runs forever or until the command returns).
	*** copy_to(localpath, remotepath): basically: 'scp localpath root@ip:remotepath'		*** copy_to(localpath, remotepath): basically: 'scp localpath root@ip:remotepath'
	*** copy_from(remotepath, localpath): ~~the reverse of the above~~		*** copy_from(remotepath, localpath): basically: 'scp root@host:remotepath localpath'

	Instance attributes:		Instance attributes:
Line 479:		Line 480:
	This also makes use of unittest's skip decorators and our own decorator skipUnlessPassed which uses test methods names for skipping - basically some kind of dependencies between them.		This also makes use of unittest's skip decorators and our own decorator skipUnlessPassed which uses test methods names for skipping - basically some kind of dependencies between them.
	skipUnlessPassed is misleading and there is gotcha here: it only works when for ordered tests (that's why the order in TEST_SUITES is important and the order/name of the test methods).		skipUnlessPassed is misleading and there is gotcha here: it only works when for ordered tests (that's why the order in TEST_SUITES is important and the order/name of the test methods).
	Why? Because of the way unittest counts passed tests. A passed test is one which isn't skipped, failed or error and this becomes a problem when the respective test ~~methods has~~ run yet (so trying to depend on some test that ~~get~~ run after your module won't work as expected).		Why? Because of the way unittest counts passed tests. A passed test is one which isn't skipped, failed or error and this becomes a problem when the respective test method hasn't run yet (so trying to depend on some test that gets run after your module won't work as expected). That is there is almost no distinction between a test which has passed and one which hasn't run yet. (see python's unittest sources in result.py)

			One more thing: be inventive with the shell commands you run and make them so you can rely on them and have one good return code for success. Sometimes you do need to parse output, see df.py and date.py for examples.