Running Toaster Tests with Containers - Revision history

Bavery: Replaced content with "See TipsAndTricks/TestingToasterWithContainers"

2016-11-08T02:48:32Z

Replaced content with "See TipsAndTricks/TestingToasterWithContainers"

← Older revision		Revision as of 02:48, 8 November 2016
Line 1:		Line 1:
	~~= How to test Toaster with Containers, Two Ways =~~		See [[TipsAndTricks/TestingToasterWithContainers]]
	It had been awhile since I ran the Selenium browser tests on one of my build systems and, once again, I found that something (?) I had upgraded was interfering with the tests. So, this page documents how to use the existing containers to run the tests .
	~~== Running the toaster built-in Selenium tests ==~~
	~~Note: Currently this only supports the firefox selenium container.~~

	~~Steps~~
	* Run the Selenium container:
	** <code>docker run -it --rm=true -p 5900:5900 -p 4444:4444 --name=selenium selenium/standalone-firefox-debug:2.53.0</code>
	*** 5900 is the default vnc port. If you are runing a vnc server on your machine map a different port e.g. -p 6900:5900 . This first number is the port mapping for your host and can be anything you want that isn't in use. The second 5900 is the vnc server port <strong>inside</strong> the container and cannot be changed.
	*** 4444 is the default selenium sever port. This provides a port for the Django tests to connect to on the host.
	** This container can be left running for multiple runs of the Django tests. Once you Ctrl-C out of it, it will be removed.
	* Run the tests
	** <code>TOASTER_TESTS_BROWSER=http://127.0.0.1:4444/wd/hub TOASTER_TESTS_URL=http://172.17.0.1:8000 ./bitbake/lib/toaster/manage.py test --liveserver=172.17.0.1:8000 tests.browser</code>
	*** TOASTER_TESTS_BROWSER -This provides the host side address for the Django tests to connect to. If you needed to change the first 4444 in the container step. Make sure you match it here. do not forget the wd/hub or you will get a 403 forbidden error.
	*** TOASTER_TESTS_URL - This is the web server the selenium instance inside the continer connects to to perform the tests. The 172.17.0.1 is explained below.
	*** --liveserver=xxx tells Django to run the test server on a particular interface and port. the default is to run it on localhost:8081 which is not reachable by the container so we need to specify. For more information on liveserver see https://docs.djangoproject.com/en/1.10/topics/testing/tools/
	**** These 2addresses MUST match and they must be reachable from inside and outside the container. The magic 172.17.0.1 is the default docker0 bridge on linux. Find it with
	~~<code> ip -4 addr show dev docker0~~
	~~4: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default~~
	~~inet 172.17.0.1/16 scope global docker0~~
	~~valid_lft forever preferred_lft forever~~
	~~</code>~~
	* if you would like to see the tests run, you can connect to the vnc server as so:
	** xtightvncviewer 127.0.0.1:5900
	** you need to wait for the test container to come up before this can connect.
	** if you changed the host side vnc port make sure you sue the same number here.
	** <strong> The password is "secret"</strong>. Chosen by the selenium folks...
	~~== Running the CROPS toaster container tests ==~~
	The Toaster container located at https://github.com/crops/toaster-container has a smoke test it runs as part of building a new container. This smoketest is run on travis and new docker images get pushed to Dockerhub iff the test passes. This test is slightly different from the above Django test framework tests in that
	* creates a bitbake environment
	* starts up toaster in that environment
	* pulls in information from the layer index
	* creates a new project
	* builds quilt-native

	~~It is a more limited test but it is end to end, so to speak. You get access to the test by cloning the toaster container repo.~~
	~~<code>~~
	~~git clone https://github.com/crops/toaster-container.git~~
	~~</code>~~
	~~Here's how you would run it~~
	~~<code>~~
	~~IMAGE=crops/toaster-master POKYDIR=/big/src/intel/myBugs/poky POKYBRANCH=COW VNCPORT=5900 ./tests/runtests.sh~~
	~~</code>~~
	~~The variables:~~
	* IMAGE=crops/toaster-master - This is the toaster container to use to run the toaster instance being tested. Use one that matches the poky you are testing e.g. toaster-master for master, toaster-krogoth for krogoth, etc.
	* POKYDIR=/home/me/poky - This is where the checkout of poky that you want to test lives on your host system. It will be bind mounted into the container.
	* POKYBRANCH=local - The type of project that gets created to test. on master, it would be a one of master,morty, or local
	* VNCPORT=5900 - If you specify this, you can watch the test run by connecting to the host on port 5900 with a vnc viewer. e.g. <code>xtightvncviewer 127.0.0.1:5900</code>
	** VNCPORT=5900 - This binds the vnc server to 0.0.0.0 and it can be accessed from any machine on the network.
	** VNCPORT=127.0.0.1:5900 - This binds it to localhost and it can be accessed from the machine running the tests only.
	** <strong> The password is "secret"</strong>. Chosen by the selenium folks...
	* runtests.sh - the script that handles the containers and runs the smoketest.py Located in the toaster container repo.

	~~A successful test looks like:~~
	~~<code>~~
	~~Running tests...~~
	~~smoketests PASSED!~~
	~~checkartifacts PASSED!~~
	~~ALL TESTS PASSED!~~
	~~</code>~~

	~~An unsuccessful test looks like~~
	~~<code>~~
	~~Running tests...~~
	~~Traceback (most recent call last):~~
	~~File "/big/src/intel/myBugs/toaster-container/tests/smoketests.py", line 76, in <module>~~
	~~raise Exception(msg)~~
	~~Exception: Error: Branch COW not in options: "local yocto project", "yocto project 2.2 "morty""~~

	~~Attempting to save screenshot to /big/src/intel/myBugs/toaster-container/screenshot.png~~
	~~FAILURE:~~ See ~~logs in /tmp/tmp.QzRu10hIfutoastertest~~
	~~</code>~~
	~~and logs can be found in the refenced~~ /~~tmp directory.~~

Bavery at 23:21, 5 November 2016

2016-11-05T23:21:55Z

← Older revision		Revision as of 23:21, 5 November 2016
Line 25:		Line 25:
	** you need to wait for the test container to come up before this can connect.		** you need to wait for the test container to come up before this can connect.
	** if you changed the host side vnc port make sure you sue the same number here.		** if you changed the host side vnc port make sure you sue the same number here.
			** <strong> The password is "secret"</strong>. Chosen by the selenium folks...
	== Running the CROPS toaster container tests ==		== Running the CROPS toaster container tests ==
	The Toaster container located at https://github.com/crops/toaster-container has a smoke test it runs as part of building a new container. This smoketest is run on travis and new docker images get pushed to Dockerhub iff the test passes. This test is slightly different from the above Django test framework tests in that		The Toaster container located at https://github.com/crops/toaster-container has a smoke test it runs as part of building a new container. This smoketest is run on travis and new docker images get pushed to Dockerhub iff the test passes. This test is slightly different from the above Django test framework tests in that
Line 46:		Line 47:
	* POKYBRANCH=local - The type of project that gets created to test. on master, it would be a one of master,morty, or local		* POKYBRANCH=local - The type of project that gets created to test. on master, it would be a one of master,morty, or local
	* VNCPORT=5900 - If you specify this, you can watch the test run by connecting to the host on port 5900 with a vnc viewer. e.g. <code>xtightvncviewer 127.0.0.1:5900</code>		* VNCPORT=5900 - If you specify this, you can watch the test run by connecting to the host on port 5900 with a vnc viewer. e.g. <code>xtightvncviewer 127.0.0.1:5900</code>
			** VNCPORT=5900 - This binds the vnc server to 0.0.0.0 and it can be accessed from any machine on the network.
			** VNCPORT=127.0.0.1:5900 - This binds it to localhost and it can be accessed from the machine running the tests only.
			** <strong> The password is "secret"</strong>. Chosen by the selenium folks...
	* runtests.sh - the script that handles the containers and runs the smoketest.py Located in the toaster container repo.		* runtests.sh - the script that handles the containers and runs the smoketest.py Located in the toaster container repo.

Bavery: /* Running the CROPS toaster container tests */

2016-11-05T23:15:18Z

Running the CROPS toaster container tests

← Older revision		Revision as of 23:15, 5 November 2016
Line 35:		Line 35:
	It is a more limited test but it is end to end, so to speak. You get access to the test by cloning the toaster container repo.		It is a more limited test but it is end to end, so to speak. You get access to the test by cloning the toaster container repo.
	<code>		<code>
	git ~~clonehttps~~://github.com/crops/toaster-container.git		git clone https://github.com/crops/toaster-container.git
	</code>		</code>
	Here's how you would run it		Here's how you would run it

Bavery at 23:15, 5 November 2016

2016-11-05T23:15:02Z

← Older revision		Revision as of 23:15, 5 November 2016
Line 50:		Line 50:
	A successful test looks like:		A successful test looks like:
	<code>		<code>
			Running tests...
			smoketests PASSED!
			checkartifacts PASSED!
			ALL TESTS PASSED!
	</code>		</code>

Line 61:		Line 65:

	Attempting to save screenshot to /big/src/intel/myBugs/toaster-container/screenshot.png		Attempting to save screenshot to /big/src/intel/myBugs/toaster-container/screenshot.png
	~~network-ada556e7-cbd6-47d4-81e3-a75bb5bda5ee~~
	FAILURE: See logs in /tmp/tmp.QzRu10hIfutoastertest		FAILURE: See logs in /tmp/tmp.QzRu10hIfutoastertest
	</code>		</code>
	and logs can be found in the refenced /tmp directory.		and logs can be found in the refenced /tmp directory.

Bavery at 23:06, 5 November 2016

2016-11-05T23:06:47Z

← Older revision		Revision as of 23:06, 5 November 2016
Line 25:		Line 25:
	** you need to wait for the test container to come up before this can connect.		** you need to wait for the test container to come up before this can connect.
	** if you changed the host side vnc port make sure you sue the same number here.		** if you changed the host side vnc port make sure you sue the same number here.
			== Running the CROPS toaster container tests ==
			The Toaster container located at https://github.com/crops/toaster-container has a smoke test it runs as part of building a new container. This smoketest is run on travis and new docker images get pushed to Dockerhub iff the test passes. This test is slightly different from the above Django test framework tests in that
			* creates a bitbake environment
			* starts up toaster in that environment
			* pulls in information from the layer index
			* creates a new project
			* builds quilt-native

			It is a more limited test but it is end to end, so to speak. You get access to the test by cloning the toaster container repo.
			<code>
			git clonehttps://github.com/crops/toaster-container.git
			</code>
			Here's how you would run it
			<code>
			IMAGE=crops/toaster-master POKYDIR=/big/src/intel/myBugs/poky POKYBRANCH=COW VNCPORT=5900 ./tests/runtests.sh
			</code>
			The variables:
			* IMAGE=crops/toaster-master - This is the toaster container to use to run the toaster instance being tested. Use one that matches the poky you are testing e.g. toaster-master for master, toaster-krogoth for krogoth, etc.
			* POKYDIR=/home/me/poky - This is where the checkout of poky that you want to test lives on your host system. It will be bind mounted into the container.
			* POKYBRANCH=local - The type of project that gets created to test. on master, it would be a one of master,morty, or local
			* VNCPORT=5900 - If you specify this, you can watch the test run by connecting to the host on port 5900 with a vnc viewer. e.g. <code>xtightvncviewer 127.0.0.1:5900</code>
			* runtests.sh - the script that handles the containers and runs the smoketest.py Located in the toaster container repo.

			A successful test looks like:
			<code>
			</code>

			An unsuccessful test looks like
			<code>
			Running tests...
			Traceback (most recent call last):
			File "/big/src/intel/myBugs/toaster-container/tests/smoketests.py", line 76, in <module>
			raise Exception(msg)
			Exception: Error: Branch COW not in options: "local yocto project", "yocto project 2.2 "morty""

			Attempting to save screenshot to /big/src/intel/myBugs/toaster-container/screenshot.png
			network-ada556e7-cbd6-47d4-81e3-a75bb5bda5ee
			FAILURE: See logs in /tmp/tmp.QzRu10hIfutoastertest
			</code>
			and logs can be found in the refenced /tmp directory.

Bavery at 20:32, 5 November 2016

2016-11-05T20:32:44Z

← Older revision		Revision as of 20:32, 5 November 2016
Line 6:		Line 6:
	Steps		Steps
	* Run the Selenium container:		* Run the Selenium container:
	** docker run -it --rm=true -p 5900:5900 -p 4444:4444 --name=selenium selenium/standalone-firefox-debug:2.53.0		** <code>docker run -it --rm=true -p 5900:5900 -p 4444:4444 --name=selenium selenium/standalone-firefox-debug:2.53.0</code>
	*** 5900 is the default vnc port. If you are runing a vnc server on your machine map a different port e.g. -p 6900:5900 . This first number is the port mapping for your host and can be anything you want that isn't in use. The second 5900 is the vnc server port <strong>inside</strong> the container and cannot be changed.		*** 5900 is the default vnc port. If you are runing a vnc server on your machine map a different port e.g. -p 6900:5900 . This first number is the port mapping for your host and can be anything you want that isn't in use. The second 5900 is the vnc server port <strong>inside</strong> the container and cannot be changed.
	*** 4444 is the default selenium sever port. This provides a port for the Django tests to connect to on the host.		*** 4444 is the default selenium sever port. This provides a port for the Django tests to connect to on the host.
	** This container can be left running for multiple runs of the Django tests. Once you Ctrl-C out of it, it will be removed.		** This container can be left running for multiple runs of the Django tests. Once you Ctrl-C out of it, it will be removed.
	* Run the tests		* Run the tests
	** TOASTER_TESTS_BROWSER=http://127.0.0.1:4444/wd/hub TOASTER_TESTS_URL=http://172.17.0.1:8000 ./bitbake/lib/toaster/manage.py test --liveserver=172.17.0.1:8000 tests.browser		** <code>TOASTER_TESTS_BROWSER=http://127.0.0.1:4444/wd/hub TOASTER_TESTS_URL=http://172.17.0.1:8000 ./bitbake/lib/toaster/manage.py test --liveserver=172.17.0.1:8000 tests.browser</code>
	*** TOASTER_TESTS_BROWSER -This provides the host side address for the Django tests to connect to. If you needed to change the first 4444 in the container step. Make sure you match it here. do not forget the wd/hub or you will get a 403 forbidden error.		*** TOASTER_TESTS_BROWSER -This provides the host side address for the Django tests to connect to. If you needed to change the first 4444 in the container step. Make sure you match it here. do not forget the wd/hub or you will get a 403 forbidden error.
	*** TOASTER_TESTS_URL - This is the web server the selenium instance inside the continer connects to to perform the tests. The 172.17.0.1 is explained below.		*** TOASTER_TESTS_URL - This is the web server the selenium instance inside the continer connects to to perform the tests. The 172.17.0.1 is explained below.

Bavery at 20:32, 5 November 2016

2016-11-05T20:32:03Z

Bavery at 20:31, 5 November 2016

2016-11-05T20:31:28Z

← Older revision		Revision as of 20:31, 5 November 2016
Line 5:		Line 5:

	Steps		Steps
	*		* Run the Selenium container:
			** docker run -it --rm=true -p 5900:5900 -p 4444:4444 --name=selenium selenium/standalone-firefox-debug:2.53.0
			*** 5900 is the default vnc port. If you are runing a vnc server on your machine map a different port e.g. -p 6900:5900 . This first number is the port mapping for your host and can be anything you want that isn't in use. The second 5900 is the vnc server port <strong>inside</strong> the container and cannot be changed.
			*** 4444 is the default selenium sever port. This provides a port for the Django tests to connect to on the host.
			** This container can be left running for multiple runs of the Django tests. Once you Ctrl-C out of it, it will be removed.
			* Run the tests
			** TOASTER_TESTS_BROWSER=http://127.0.0.1:4444/wd/hub TOASTER_TESTS_URL=http://172.17.0.1:8000 ./bitbake/lib/toaster/manage.py test --liveserver=172.17.0.1:8000 tests.browser
			*** TOASTER_TESTS_BROWSER -This provides the host side address for the Django tests to connect to. If you needed to change the first 4444 in the container step. Make sure you match it here. do not forget the wd/hub or you will get a 403 forbidden error.
			*** TOASTER_TESTS_URL - This is the web server the selenium instance inside the continer connects to to perform the tests. The 172.17.0.1 is explained below.
			*** --liveserver=xxx tells Django to run the test server on a particular interface and port. the default is to run it on localhost:8081 which is not reachable by the container so we need to specify. For more information on liveserver see https://docs.djangoproject.com/en/1.10/topics/testing/tools/
			**** These 2addresses MUST match and they must be reachable from inside and outside the container. The magic 172.17.0.1 is the default docker0 bridge on linux. Find it with
			<code> ip -4 addr show dev docker0
			4: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
			inet 172.17.0.1/16 scope global docker0
			valid_lft forever preferred_lft forever
			</code>
			* if you would like to see the tests run, you can connect to the vnc server as so:
			** xtightvncviewer 127.0.0.1:5900
			** you need to wait for the test container to come up before this can connect.
			** if you changed the host side vnc port make sure you sue the same number here.

Bavery: Created page with "= How to test Toaster with Containers, Two Ways = It had been awhile since I ran the Selenium browser tests on one of my build systems and, once again, I found that something ..."

2016-11-05T20:12:19Z

Created page with "= How to test Toaster with Containers, Two Ways = It had been awhile since I ran the Selenium browser tests on one of my build systems and, once again, I found that something ..."

New page

= How to test Toaster with Containers, Two Ways =
It had been awhile since I ran the Selenium browser tests on one of my build systems and, once again, I found that something (?) I had upgraded was interfering with the tests. So, this page documents how to use the existing containers to run the tests .
== Running the toaster built-in Selenium tests ==
Note: Currently this only supports the firefox selenium container.

Steps
*