Testing Toaster

Toaster has 4 testing suites, targeted at verifying different parts of the system. This document briefly describes each testing suite.

Django Unit Tests

Toaster is primarely a Django application. As such, it makes use of the Django test system that runs unit tests on mock data to ensure pieces of functionality work as expected, and prevent regressions.

To run the django unit tests, invoke them as with any Django test suite

   ./bitbake/lib/toaster/manage.py test

To add unit tests, simply add needed tests to the _tests.py_ file in the module you're editing.

Toaster Test System (TTS)

The TTS is a suite aimed at running integration and acceptance tests. Currently, it verifying the correct syntax and form of the Python code, tries to start Toaster in both modes, and runs HTML5 validation on HTML-returning pages.

The tests are designed to be triggered through a continuous integration system.

To start manually it, run the tts test runner:

   ./bitbake/lib/toaster/contrib/tts/runner.py

The oe-selftest toaster tests

These tests are verifying the correctness of the data gathered by the build process - it tests the bitbake/lib/bb/ui/toasterui.py and bitbake/lib/bb/ui/buildinfohelper.py files and the interface to the bitbake server process.

In order for these tests to run properly, the following steps need to be taken before running, on a clean toaster setup:

• Create new project using local poky release
• Create 1 core-image-minimal image
• Create 1 core-image-sato image to check building from cache
• Break a recipe file (by modifying a link, for example)
• Create a new image using broken recipe to check error detection is working

To run these tests you need to source an environment by issuing:

   source oe-init-build-env

then modify the bblayers.conf file to add the meta-selftest layer and then run the command:

   oe-selftest --run-tests _toaster

The selenium automated UI tests

The automated UI tests are targeted at validating and identifying regressions in the UI pages. These tests are based on the Selenium framework to drive interactions in the interface.

To run the selenium tests, please use:

   ./ bitbake/lib/toaster/contrib/tts/toasteruitest

  ./bitbake/lib/toaster/contrib/tts/toasteruitest/run_toastertests.py

In order for these tests to run properly, the following steps need to be taken before running, on a clean toaster setup:

• Create new project named "selenium-project" using local poky release
• Create 1 core-image-minimal image
• Break a recipe file (by modifying a link, for example)
• Create a new image using broken recipe to check error detection is working
• Fix the recipe broken in previous step
• Create 1 core-image-sato image

In order to run all current tests, you need to modify the configuration file "toaster_test.cfg" with the following line:

   test_cases = [901, 902, 903, 904, 906, 910, 911, 912, 913, 914, 915, 916, 923, 924, 940, 941, 942, 943, 944, 945, 946, 947, 948, 949, 950, 951, 955, 956]