|
|
| Line 1: |
Line 1: |
| = How to test Toaster with Containers, Two Ways =
| | See [[TipsAndTricks/TestingToasterWithContainers]] |
| It had been awhile since I ran the Selenium browser tests on one of my build systems and, once again, I found that something (?) I had upgraded was interfering with the tests. So, this page documents how to use the existing containers to run the tests .
| |
| == Running the toaster built-in Selenium tests ==
| |
| Note: Currently this only supports the firefox selenium container.
| |
| | |
| Steps
| |
| * Run the Selenium container:
| |
| ** <code>docker run -it --rm=true -p 5900:5900 -p 4444:4444 --name=selenium selenium/standalone-firefox-debug:2.53.0</code>
| |
| *** 5900 is the default vnc port. If you are runing a vnc server on your machine map a different port e.g. -p 6900:5900 . This first number is the port mapping for your host and can be anything you want that isn't in use. The second 5900 is the vnc server port <strong>inside</strong> the container and cannot be changed.
| |
| *** 4444 is the default selenium sever port. This provides a port for the Django tests to connect to on the host.
| |
| ** This container can be left running for multiple runs of the Django tests. Once you Ctrl-C out of it, it will be removed.
| |
| * Run the tests
| |
| ** <code>TOASTER_TESTS_BROWSER=http://127.0.0.1:4444/wd/hub TOASTER_TESTS_URL=http://172.17.0.1:8000 ./bitbake/lib/toaster/manage.py test --liveserver=172.17.0.1:8000 tests.browser</code>
| |
| *** TOASTER_TESTS_BROWSER -This provides the host side address for the Django tests to connect to. If you needed to change the first 4444 in the container step. Make sure you match it here. do not forget the wd/hub or you will get a 403 forbidden error.
| |
| *** TOASTER_TESTS_URL - This is the web server the selenium instance inside the continer connects to to perform the tests. The 172.17.0.1 is explained below.
| |
| *** --liveserver=xxx tells Django to run the test server on a particular interface and port. the default is to run it on localhost:8081 which is not reachable by the container so we need to specify. For more information on liveserver see https://docs.djangoproject.com/en/1.10/topics/testing/tools/
| |
| **** These 2addresses MUST match and they must be reachable from inside and outside the container. The magic 172.17.0.1 is the default docker0 bridge on linux. Find it with
| |
| <code> ip -4 addr show dev docker0
| |
| 4: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
| |
| inet 172.17.0.1/16 scope global docker0
| |
| valid_lft forever preferred_lft forever
| |
| </code>
| |
| * if you would like to see the tests run, you can connect to the vnc server as so:
| |
| ** xtightvncviewer 127.0.0.1:5900
| |
| ** you need to wait for the test container to come up before this can connect.
| |
| ** if you changed the host side vnc port make sure you sue the same number here.
| |
| ** <strong> The password is "secret"</strong>. Chosen by the selenium folks...
| |
| == Running the CROPS toaster container tests ==
| |
| The Toaster container located at https://github.com/crops/toaster-container has a smoke test it runs as part of building a new container. This smoketest is run on travis and new docker images get pushed to Dockerhub iff the test passes. This test is slightly different from the above Django test framework tests in that
| |
| * creates a bitbake environment
| |
| * starts up toaster in that environment
| |
| * pulls in information from the layer index
| |
| * creates a new project
| |
| * builds quilt-native
| |
| | |
| It is a more limited test but it is end to end, so to speak. You get access to the test by cloning the toaster container repo.
| |
| <code>
| |
| git clone https://github.com/crops/toaster-container.git
| |
| </code>
| |
| Here's how you would run it
| |
| <code>
| |
| IMAGE=crops/toaster-master POKYDIR=/big/src/intel/myBugs/poky POKYBRANCH=COW VNCPORT=5900 ./tests/runtests.sh
| |
| </code>
| |
| The variables:
| |
| * IMAGE=crops/toaster-master - This is the toaster container to use to run the toaster instance being tested. Use one that matches the poky you are testing e.g. toaster-master for master, toaster-krogoth for krogoth, etc.
| |
| * POKYDIR=/home/me/poky - This is where the checkout of poky that you want to test lives on your host system. It will be bind mounted into the container.
| |
| * POKYBRANCH=local - The type of project that gets created to test. on master, it would be a one of master,morty, or local
| |
| * VNCPORT=5900 - If you specify this, you can watch the test run by connecting to the host on port 5900 with a vnc viewer. e.g. <code>xtightvncviewer 127.0.0.1:5900</code>
| |
| ** VNCPORT=5900 - This binds the vnc server to 0.0.0.0 and it can be accessed from any machine on the network.
| |
| ** VNCPORT=127.0.0.1:5900 - This binds it to localhost and it can be accessed from the machine running the tests only.
| |
| ** <strong> The password is "secret"</strong>. Chosen by the selenium folks...
| |
| * runtests.sh - the script that handles the containers and runs the smoketest.py Located in the toaster container repo.
| |
| | |
| A successful test looks like:
| |
| <code>
| |
| Running tests...
| |
| smoketests PASSED!
| |
| checkartifacts PASSED!
| |
| ALL TESTS PASSED!
| |
| </code>
| |
| | |
| An unsuccessful test looks like
| |
| <code>
| |
| Running tests...
| |
| Traceback (most recent call last):
| |
| File "/big/src/intel/myBugs/toaster-container/tests/smoketests.py", line 76, in <module>
| |
| raise Exception(msg)
| |
| Exception: Error: Branch COW not in options: "local yocto project", "yocto project 2.2 "morty""
| |
| | |
| Attempting to save screenshot to /big/src/intel/myBugs/toaster-container/screenshot.png
| |
| FAILURE: See logs in /tmp/tmp.QzRu10hIfutoastertest
| |
| </code>
| |
| and logs can be found in the refenced /tmp directory.
| |
