TipsAndTricks/TestingToasterWithContainers

How to test Toaster with Containers, Two Ways

It had been awhile since I ran the Selenium browser tests on one of my build systems and, once again, I found that something (?) I had upgraded was interfering with the tests. So, this page documents how to use the existing containers to run the tests .

Running the toaster built-in Selenium tests

Note: Currently this only supports the firefox selenium container.

Steps

• Run the Selenium container:
  • docker run -it --rm=true -p 5900:5900 -p 4444:4444 --name=selenium selenium/standalone-firefox-debug:2.53.0
    • 5900 is the default vnc port. If you are runing a vnc server on your machine map a different port e.g. -p 6900:5900 . This first number is the port mapping for your host and can be anything you want that isn't in use. The second 5900 is the vnc server port inside the container and cannot be changed.
    • 4444 is the default selenium sever port. This provides a port for the Django tests to connect to on the host.
  • This container can be left running for multiple runs of the Django tests. Once you Ctrl-C out of it, it will be removed.
• Run the tests
  • TOASTER_TESTS_BROWSER=remote TOASTER_REMOTE_HUB=http://127.0.0.1:4444/wd/hub ./bitbake/lib/toaster/manage.py test --liveserver=172.17.0.1:8001 tests.browser
    • TOASTER_TESTS_BROWSER - remote is now an option , like firefox or chrome
    • TOASTER_REMOTE_HUB - This provides the host side address for the Django tests to connect to. If you needed to change the first 4444 in the container step. Make sure you match it here. do not forget the wd/hub or you will get a 403 forbidden error.
    • --liveserver=xxx tells Django to run the test server on a particular interface and port. The default is to run it on localhost:8081 which is not reachable by the container so we need to specify an address that is reachable by both the host and the container. For more information on liveserver see https://docs.djangoproject.com/en/1.10/topics/testing/tools/
      • The magic 172.17.0.1 is the default docker0 bridge on linux. Find it with

 ip -4 addr show dev docker0
4: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default 
  inet 172.17.0.1/16 scope global docker0
  valid_lft forever preferred_lft forever

• if you would like to see the tests run, you can connect to the vnc server as so:
  • xtightvncviewer 127.0.0.1:5900
  • you need to wait for the test container to come up before this can connect.
  • if you changed the host side vnc port make sure you sue the same number here.
  • The password is "secret". Chosen by the selenium folks...

Running the CROPS toaster container tests

The Toaster container located at https://github.com/crops/toaster-container has a smoke test it runs as part of building a new container. This smoketest is run on travis and new docker images get pushed to Dockerhub iff the test passes. This test is slightly different from the above Django test framework tests in that

• creates a bitbake environment
• starts up toaster in that environment
• pulls in information from the layer index
• creates a new project
• builds quilt-native

It is a more limited test but it is end to end, so to speak. You get access to the test by cloning the toaster container repo.

 
git clone https://github.com/crops/toaster-container.git

Here's how you would run it

IMAGE=crops/toaster-master POKYDIR=/big/src/intel/myBugs/poky POKYBRANCH=COW VNCPORT=5900 ./tests/runtests.sh

The variables:

• IMAGE=crops/toaster-master - This is the toaster container to use to run the toaster instance being tested. Use one that matches the poky you are testing e.g. toaster-master for master, toaster-krogoth for krogoth, etc.
• POKYDIR=/home/me/poky - This is where the checkout of poky that you want to test lives on your host system. It will be bind mounted into the container.
• POKYBRANCH=local - The type of project that gets created to test. on master, it would be a one of master,morty, or local
• VNCPORT=5900 - If you specify this, you can watch the test run by connecting to the host on port 5900 with a vnc viewer. e.g. xtightvncviewer 127.0.0.1:5900
  • VNCPORT=5900 - This binds the vnc server to 0.0.0.0 and it can be accessed from any machine on the network.
  • VNCPORT=127.0.0.1:5900 - This binds it to localhost and it can be accessed from the machine running the tests only.
  • The password is "secret". Chosen by the selenium folks...
• runtests.sh - the script that handles the containers and runs the smoketest.py Located in the toaster container repo.

A successful test looks like:

Running tests...
smoketests PASSED!
checkartifacts PASSED!
ALL TESTS PASSED!

An unsuccessful test looks like

Running tests...
Traceback (most recent call last):
 File "/big/src/intel/myBugs/toaster-container/tests/smoketests.py", line 76, in <module>
   raise Exception(msg)
Exception: Error: Branch COW not in options: "local yocto project", "yocto project 2.2 "morty""

Attempting to save screenshot to /big/src/intel/myBugs/toaster-container/screenshot.png
FAILURE: See logs in /tmp/tmp.QzRu10hIfutoastertest

and logs can be found in the referenced /tmp directory.

Using the CROPS toaster container test to do a git bisect

• Pending Review.