Build testing using Autotest - Revision history

Stefans at 16:35, 6 March 2013

2013-03-06T16:35:32Z

← Older revision		Revision as of 16:35, 6 March 2013
Line 329:		Line 329:

	=== Add clients ===		=== Add clients ===


			Setup password-less ssh connection from the server to the client machine (which can be a VM running on the same host as the autotest server)
			* As autotest user, on the server, create a RSA key
			ssh-keygen -t rsa
			* Then, still on the server, and as autotest, copy it to the host:
			ssh-copy-id root@<client machine>

			From the web interface go to http://localhost/afe/server/admin and add some hosts. Then they would show up when you create a job.

Stefans at 16:20, 6 March 2013

2013-03-06T16:20:09Z

← Older revision		Revision as of 16:20, 6 March 2013
Line 52:		Line 52:
	heater.py it's the actual test code (so we only wrote one test which we reuse in different ways by running different control files).		heater.py it's the actual test code (so we only wrote one test which we reuse in different ways by running different control files).

	Let's see the available tests		* Let's see the available tests:
	<pre style="white-space: pre-wrap;		<pre style="white-space: pre-wrap;
	white-space: -moz-pre-wrap;		white-space: -moz-pre-wrap;
Line 72:		Line 72:
	heater/control.281x-minx32 almost 281 - Build core-image-minimal with x32 tune		heater/control.281x-minx32 almost 281 - Build core-image-minimal with x32 tune
	heater/control.325-emgd 325 - Build emgd-driver-bin		heater/control.325-emgd 325 - Build emgd-driver-bin

	~~#let's say we want to build core-image-minimal (aka bitbake core-image-minimal)~~
	~~$ ./autotest --verbose run heater/control.minimal~~
	~~#or another one : bitbake core-image-minimal with x32 tune~~
	~~$ ./autotest --verbose run heater/control.281x-minx32~~
	</pre>		</pre>

	~~You'll start to see a lot of output from Autotest and the tests themselves (remove --verbose for less noise but you'll loose bitbake's output too (it's logged anyway))~~		* Here is what autotest shows for heater/control.minimal (add --verbose more noise and bitbake's output)

	Here is what autotest shows for heater/control.minimal:
	<pre style="white-space: pre-wrap;		<pre style="white-space: pre-wrap;
	white-space: -moz-pre-wrap;		white-space: -moz-pre-wrap;
Line 250:		Line 243:
	** updates the autotest software on each client before commencing work.		** updates the autotest software on each client before commencing work.

	=== Installing ===		=== Installing and configuring ===

	On https://github.com/autotest/autotest/wiki/SysAdmin you can find documentation on how to setup an autotest server infrastructure on your distro. They provide a script for Fedora/Ubuntu which automates the installation. However I recommend you follow the manual steps to get a better understanding of what's happening behind the scenes.		On https://github.com/autotest/autotest/wiki/SysAdmin you can find documentation on how to setup an autotest server infrastructure on your distro. They provide a script for Fedora/Ubuntu which automates the installation. However I recommend you follow the manual steps to get a better understanding of what's happening behind the scenes.
Line 319:		Line 312:
	# systemctl enable autotestd.service		# systemctl enable autotestd.service
	# systemctl start autotestd.service		# systemctl start autotestd.service
			</pre>

			That's it, now go to http://localhost/afe and you should see a web interface.

			=== Adding the tests ===

			<pre>
	# su - autotest		# su - autotest
	$ cd client		$ cd client
	~~- add tests and import them~~
	$ mv site_tests site_tests.orig		$ mv site_tests site_tests.orig
	$ git clone -b stefans/tests git://git.yoctoproject.org/poky-contrib site_tests		$ git clone -b stefans/tests git://git.yoctoproject.org/poky-contrib site_tests
	$ ~/utils/test_importer.py		$ ~/utils/test_importer.py
			</pre>

	Each time you want to update the tests you need to go to /usr/local/autotest/client/site_tests and do a git pull and then run /usr/local/autotest/utils/test_importer.py (so they show up in the web interface)		Each time you want to update the tests you need to go to /usr/local/autotest/client/site_tests and do a git pull and then run /usr/local/autotest/utils/test_importer.py (so they show up in the web interface)

	~~- How to add~~ clients:		=== Add clients ===

Stefans at 16:15, 6 March 2013

2013-03-06T16:15:34Z

← Older revision		Revision as of 16:15, 6 March 2013
Line 24:		Line 24:
	== How to run the tests using the standalone autotest client on your machine ==		== How to run the tests using the standalone autotest client on your machine ==
	'''Types of tests and paths'''		'''Types of tests and paths'''
	If you read the autotest docs you~~'ll~~ should know that there are more types of tests:
			If you have read the autotest docs you should know that there are more types of tests:
	* client-side control files, which can be		* client-side control files, which can be
	** manually deployed and run (what this section is about)		** manually deployed and run (what this section is about)

Stefans: /* How to run the tests using the standalone autotest client on your machine */

2013-03-06T15:31:37Z

How to run the tests using the standalone autotest client on your machine

← Older revision		Revision as of 15:31, 6 March 2013
Line 23:		Line 23:

	== How to run the tests using the standalone autotest client on your machine ==		== How to run the tests using the standalone autotest client on your machine ==
			'''Types of tests and paths'''
	If you read the autotest docs you'll should know that there are more types of tests:		If you read the autotest docs you'll should know that there are more types of tests:
	* client-side control files, which can be		* client-side control files, which can be
	** manually deployed and run (~~this~~ what this section is about)		** manually deployed and run (what this section is about)
	** deployed by autotest server and run automatically on client machines (next section)		** deployed by autotest server and run automatically on client machines (next section)
	* server-side control files (when jobs need to run stuff both on the server and multiple autotest clients) - we don't care about ~~tese~~.		* server-side control files (when jobs need to run stuff both on the server and multiple autotest clients) - we don't care about these.

			Have a look here https://github.com/autotest/autotest/wiki/DirectoryStructure to learn more about the directory structure of Autotest.
			----

	~~So let~~'~~s start:~~		'''Get Autotest'''

	* clone the autotest repo		* clone the autotest repo
Line 38:		Line 40:
	$ cd autotest/client/		$ cd autotest/client/
	</pre>		</pre>

	* get our tests (don't mind the path, just clone it to tmp/site_tests)		* get our tests (don't mind the path, just clone it to tmp/site_tests)
	<pre>		<pre>

Stefans at 14:08, 6 March 2013

2013-03-06T14:08:24Z

← Older revision		Revision as of 14:08, 6 March 2013
Line 25:		Line 25:

	If you read the autotest docs you'll should know that there are more types of tests:		If you read the autotest docs you'll should know that there are more types of tests:
	* client-side control files		* client-side control files, which can be
	~~Which~~ can be:
	** manually deployed and run (this what this section is about)		** manually deployed and run (this what this section is about)
	** deployed by autotest server and run automatically on client machines (next section)		** deployed by autotest server and run automatically on client machines (next section)
	* server-side control files (when jobs need to run stuff both on the server and multiple autotest clients) - we don't care about tese.		* server-side control files (when jobs need to run stuff both on the server and multiple autotest clients) - we don't care about tese.


	So let's start:		So let's start:

Stefans at 14:07, 6 March 2013

2013-03-06T14:07:37Z

← Older revision		Revision as of 14:07, 6 March 2013
Line 5:		Line 5:
	Autotest is a framework for fully automated testing. It is designed primarily to test the Linux kernel, though it can serve many other purposes such as qualifying new hardware, virtualization testing and other general user space program testing under linux platforms.		Autotest is a framework for fully automated testing. It is designed primarily to test the Linux kernel, though it can serve many other purposes such as qualifying new hardware, virtualization testing and other general user space program testing under linux platforms.

	If you want to know more about Autotest itself see the [https://github.com/autotest/autotest/wiki Autotest wiki]		If you want to know more about Autotest itself see the [https://github.com/autotest/autotest/wiki Autotest wiki]. [[https://fedorahosted.org/autoqa/ Fedora's AutoQA]] is one example of a project that uses autotest as their test automation framework.

	Because of the increasing need to automate our build tests the Yocto QA team decided to use Autotest.		Because of the increasing need to automate our build tests the Yocto QA team decided to use Autotest.
Line 23:		Line 23:

	== How to run the tests using the standalone autotest client on your machine ==		== How to run the tests using the standalone autotest client on your machine ==

			If you read the autotest docs you'll should know that there are more types of tests:
			* client-side control files
			Which can be:
			** manually deployed and run (this what this section is about)
			** deployed by autotest server and run automatically on client machines (next section)
			* server-side control files (when jobs need to run stuff both on the server and multiple autotest clients) - we don't care about tese.

			So let's start:

	* clone the autotest repo		* clone the autotest repo
Line 29:		Line 38:
	$ cd autotest/client/		$ cd autotest/client/
	</pre>		</pre>
	* get our tests		* get our tests (don't mind the path, just clone it to tmp/site_tests)
	<pre>		<pre>
	$ git clone -b stefans/tests git://git.yoctoproject.org/poky-contrib tmp/site_tests		$ git clone -b stefans/tests git://git.yoctoproject.org/poky-contrib tmp/site_tests
Line 35:		Line 44:
	Ok, so now you should have autotest installed and our tests in ~/autotest/client/tmp/site_tests.		Ok, so now you should have autotest installed and our tests in ~/autotest/client/tmp/site_tests.

	You'll see a bunch of control.* files and a heater.py if you "ls -l ~/autotest/client/tmp/site_tests"		You'll see a bunch of control.* files and a heater.py if you "ls -l ~/autotest/client/tmp/site_tests/heater"

	~~That~~'s the actual test code (so we only wrote one test which we reuse in different ways).		heater.py it's the actual test code (so we only wrote one test which we reuse in different ways by running different control files).

	Let's see the available tests		Let's see the available tests
Line 241:		Line 250:
	On https://github.com/autotest/autotest/wiki/SysAdmin you can find documentation on how to setup an autotest server infrastructure on your distro. They provide a script for Fedora/Ubuntu which automates the installation. However I recommend you follow the manual steps to get a better understanding of what's happening behind the scenes.		On https://github.com/autotest/autotest/wiki/SysAdmin you can find documentation on how to setup an autotest server infrastructure on your distro. They provide a script for Fedora/Ubuntu which automates the installation. However I recommend you follow the manual steps to get a better understanding of what's happening behind the scenes.

	Here is a step by step install on my Fedora 18 machine~~. The replicate~~ pretty close the docs (or script) from the autotest wiki with some small changes.		Here is a step by step install on my Fedora 18 machine, which replicates pretty close the docs (or script) from the autotest wiki with some small changes.

	<pre style="white-space: pre-wrap;		<pre style="white-space: pre-wrap;
Line 266:		Line 275:
	- set apache configs		- set apache configs
	# ln -s /usr/local/autotest/apache/conf /etc/httpd/autotest.d		# ln -s /usr/local/autotest/apache/conf /etc/httpd/autotest.d
	# ln -s /usr/local/autotest/apache/apache-conf /etc/httpd/conf.d/# z_autotest.conf		# ln -s /usr/local/autotest/apache/apache-conf /etc/httpd/conf.d/z_autotest.conf
	# ln -s /usr/local/autotest/apache/apache-web-conf /etc/httpd/conf.d/z_autotest-web.conf		# ln -s /usr/local/autotest/apache/apache-web-conf /etc/httpd/conf.d/z_autotest-web.conf

Line 272:		Line 281:
	# useradd -d /usr/local/autotest autotest		# useradd -d /usr/local/autotest autotest
	# passwd autotest		# passwd autotest
	# chown -R autotest:autotest autotest		# chown -R autotest:autotest /usr/local/autotest
	# su - autotest		# su - autotest

Line 278:		Line 287:
	- edit ~autotest/.gitconfig and add gitproxy config fragment if you need a proxy		- edit ~autotest/.gitconfig and add gitproxy config fragment if you need a proxy

			- set autotest db (replace MYSQL_ROOT_PASS with the pasword you set earlier and MYSQL_AUTOTEST_PASS wit a passor for the autotest db
	- set autotest db
	$ ./installation_support/autotest-database-turnkey -s --root-password MYSQL_ROOT_PASS -p MYSQL_AUTOTEST_PASS		$ ./installation_support/autotest-database-turnkey -s --root-password MYSQL_ROOT_PASS -p MYSQL_AUTOTEST_PASS
	- as autotest user		- as autotest user
Line 285:		Line 293:
	- compile frontend		- compile frontend
	$ ./utils/compile_gwt_clients.py -a		$ ./utils/compile_gwt_clients.py -a
	- ~~change~~ the ~~following:~~
			- Now you need to edit global_config.ini and set the password usef for autotest db (earlier MYSQL_AUTOTEST_PASS)
			- and set a path for autotest clients install (where autotest server will deploy it's code and tests on the machines/VMs)
			- Make sure that path it's on a large partition because that's where the builds take place
	$ vi global_config.ini		$ vi global_config.ini
			---
			password: please_set_this_password

			[scroll down]

	~~password: please_set_this_password~~
	~~...~~
	[AUTOSERV]		[AUTOSERV]
	# Autotest potential install paths		# Autotest potential install paths
	client_autodir_paths: /data/autotest		client_autodir_paths: /data/autotest

			----

			$ exit
	# systemctl enable httpd.service		# systemctl enable httpd.service
	# systemctl start httpd.service		# systemctl start httpd.service
Line 300:		Line 315:
	# systemctl enable autotestd.service		# systemctl enable autotestd.service
	# systemctl start autotestd.service		# systemctl start autotestd.service


	# su - autotest		# su - autotest
	$ cd client		$ cd client
	- add tests and import them		- add tests and import them
	$ git clone -b stefans/tests git://git.yoctoproject.org/poky-contrib ~~tmp/~~site_tests		$ mv site_tests site_tests.orig
			$ git clone -b stefans/tests git://git.yoctoproject.org/poky-contrib site_tests
	$ ~/utils/test_importer.py		$ ~/utils/test_importer.py

			Each time you want to update the tests you need to go to /usr/local/autotest/client/site_tests and do a git pull and then run /usr/local/autotest/utils/test_importer.py (so they show up in the web interface)

			- How to add clients:

Stefans at 17:16, 5 March 2013

2013-03-05T17:16:01Z

@@ Line 116: / Line 116: @@
 </pre>
-=== How to write a new test aka control file ===
+== How to write a new test aka control file ==
 ===== What's a control file? =====
@@ Line 209: / Line 209: @@
 .....
 TBD

Stefans: /* How to run the tests using the standalone autotest client on your machine */

2013-03-05T13:14:48Z

How to run the tests using the standalone autotest client on your machine

← Older revision		Revision as of 13:14, 5 March 2013
Line 31:		Line 31:
	* get our tests		* get our tests
	<pre>		<pre>
	$ git clone -b stefans/tests git://git.yoctoproject.org/poky-contrib tmp/site_tests~~</nowiki>~~		$ git clone -b stefans/tests git://git.yoctoproject.org/poky-contrib tmp/site_tests
	</pre>		</pre>
	Ok, so now you should have autotest installed and our tests in ~/autotest/client/tmp/site_tests.		Ok, so now you should have autotest installed and our tests in ~/autotest/client/tmp/site_tests.

Stefans at 13:11, 5 March 2013

2013-03-05T13:11:23Z

← Older revision		Revision as of 13:11, 5 March 2013
Line 161:		Line 161:
	Notes:		Notes:
	* AUTHOR, NAME, TIME, TEST_* and DOC are mandatory in a control file (autotest requirement, see https://github.com/autotest/autotest/wiki/ControlRequirements )		* AUTHOR, NAME, TIME, TEST_* and DOC are mandatory in a control file (autotest requirement, see https://github.com/autotest/autotest/wiki/ControlRequirements )
	* the rest is just python code (and in our case a call to run_test('heater') with some args		* the rest is just python code (and in our case a call to run_test('heater') with some args)

	What happens behind the scene on my machine (step by step):		What happens behind the scene on my machine (step by step):

Stefans at 13:07, 5 March 2013

2013-03-05T13:07:43Z

← Older revision		Revision as of 13:07, 5 March 2013
Line 10:		Line 10:
	Currently there are 12 test cases that we run using Autotest.		Currently there are 12 test cases that we run using Autotest.

	Before using ~~them and/~~or writing new tests you ~~'''~~must~~''' read some Autotest documentation to~~ understand how ~~some~~ things work:		Before using any of the test cases or writing new tests you must understand how things work, and in order to do that you should read the Autotest documentation:

	https://github.com/autotest/autotest/wiki/ClientQuickStart		https://github.com/autotest/autotest/wiki/ClientQuickStart
Line 24:		Line 24:
	== How to run the tests using the standalone autotest client on your machine ==		== How to run the tests using the standalone autotest client on your machine ==

	~~<nowiki>~~clone the autotest repo		* clone the autotest repo
			<pre>
	$ git clone https://github.com/autotest/autotest		$ git clone https://github.com/autotest/autotest
	$ cd autotest/client/		$ cd autotest/client/
	get our tests		</pre>
			* get our tests
			<pre>
	$ git clone -b stefans/tests git://git.yoctoproject.org/poky-contrib tmp/site_tests</nowiki>		$ git clone -b stefans/tests git://git.yoctoproject.org/poky-contrib tmp/site_tests</nowiki>
			</pre>
	Ok, so now you should have autotest installed and our tests in ~/autotest/client/tmp/site_tests.		Ok, so now you should have autotest installed and our tests in ~/autotest/client/tmp/site_tests.

Line 57:		Line 60:
	heater/control.325-emgd 325 - Build emgd-driver-bin		heater/control.325-emgd 325 - Build emgd-driver-bin

	let's say we want to build core-image-minimal (aka bitbake core-image-minimal)		#let's say we want to build core-image-minimal (aka bitbake core-image-minimal)
	$ ./autotest --verbose run heater/control.minimal		$ ./autotest --verbose run heater/control.minimal
	or another one : bitbake core-image-minimal with x32 tune		#or another one : bitbake core-image-minimal with x32 tune
	$ ./autotest --verbose run heater/control.281x-minx32		$ ./autotest --verbose run heater/control.281x-minx32
	</pre>		</pre>
Line 65:		Line 68:
	You'll start to see a lot of output from Autotest and the tests themselves (remove --verbose for less noise but you'll loose bitbake's output too (it's logged anyway))		You'll start to see a lot of output from Autotest and the tests themselves (remove --verbose for less noise but you'll loose bitbake's output too (it's logged anyway))

	Here~~'s some output from ./~~autotest ~~run~~ heater/control.minimal		Here is what autotest shows for heater/control.minimal:
	<pre style="white-space: pre-wrap;		<pre style="white-space: pre-wrap;
	white-space: -moz-pre-wrap;		white-space: -moz-pre-wrap;