Testing Toaster

 This page is outdated. Please refer to Toaster testing plan page for testing Toaster approach.

Toaster has 4 testing suites, targeted at verifying different parts of the system. This document briefly describes each testing suite.

Django Unit Tests

Toaster is primarely a Django application. As such, it makes use of the Django test system that runs unit tests on mock data to ensure pieces of functionality work as expected, and prevent regressions.

To run the django unit tests, invoke them as with any Django test suite

   ./bitbake/lib/toaster/manage.py test

To add unit tests, simply add needed tests to the _tests.py_ file in the module you're editing.

Toaster Test System (TTS)

The TTS is a suite aimed at running integration and acceptance tests. Currently, it verifying the correct syntax and form of the Python code, tries to start Toaster in both modes, and runs HTML5 validation on HTML-returning pages.

The tests are designed to be triggered through a continuous integration system.

To start manually it, run the tts test runner:

   ./bitbake/lib/toaster/contrib/tts/runner.py

The oe-selftest toaster tests

These tests are verifying the correctness of the data gathered by the build process - it tests the bitbake/lib/bb/ui/toasterui.py and bitbake/lib/bb/ui/buildinfohelper.py files and the interface to the bitbake server process.

In order for these tests to run properly, the following steps need to be taken before running, on a clean toaster setup:

• Create new project using local poky release
• Create 1 core-image-minimal image
• Create 1 core-image-sato image to check building from cache
• Break a recipe file (by modifying a link, for example)
• Create a new image using broken recipe to check error detection is working

To run these tests you need to source an environment by issuing:

   source oe-init-build-env

then modify the bblayers.conf file to add the meta-selftest layer, create a symbolic link to the toaster.sqlite file:

   ln -s ../toaster.sqlite toaster.sqlite

and then run the command:

   oe-selftest --run-tests _toaster

The selenium automated UI tests

The automated UI tests are targeted at validating and identifying regressions in the UI pages. These tests are based on the Selenium framework to drive interactions in the interface.

The selenium tests require a couple of components:

• scrot (a screen capture utility) and pip - a utility for installing python modules. The following is how you install it for ubuntu:
  • sudo apt-get install scrot python-pip python-virtualenv
  • virtualenv t-selenium
  • source ./t-selenium/bin/activate
  • pip install selenium
• Since this is a web application, the UI can also be tested from a mac. To install the requirements there do:
  • sudo easy_install pip
  • sudo pip install virtualenv
  • virtualenv t-selenium
  • source ./t-selenium/bin/activate
• also install the latest filefox extension from selenium located at: http://release.seleniumhq.org/selenium-ide/ and install all the extensions.

The selenium tests are located in:

   ./ bitbake/lib/toaster/contrib/tts/toasteruitest

You run them with:

  ./bitbake/lib/toaster/contrib/tts/toasteruitest/run_toastertests.py

The above command takes the list of the tests to run from the toaster_test.cfg file. The tests themselves are located in toaster_automation_test.py. The toaster_test.cfg file has a couple of important variables in it separated by which os you are running the tests on. These varables control where the toaster instance is located (defaults to http://127.0.0.1:8000) what browser to use (firefox is safest for selenium though there is a chrome driver available as well), what tests to run and what log level to save data at.

In order for these tests to run properly, the following steps need to be taken before running, on a clean toaster setup:

• Create new project named "selenium-project" using local poky release
• Create 1 core-image-minimal image
• Break a recipe file (by modifying a link, for example)
• Create a new image using broken recipe to check error detection is working
• Fix the recipe broken in previous step
• Create 1 core-image-sato image

In order to run all current tests, you need to modify the configuration file "toaster_test.cfg" with the following line:

   test_cases = [901, 902, 903, 904, 906, 910, 911, 912, 913, 914, 915, 916, 923, 924, 940, 941, 942, 943, 944, 945, 946, 947, 948, 949, 950, 951, 955, 956]