Toaster testing plan: Difference between revisions

From Yocto Project
Jump to navigationJump to search
Line 73: Line 73:
**AR Cristina + Mihai:
**AR Cristina + Mihai:
*please validate and add info from Testing Toaster wiki in this section(Test Strategy): https://wiki.yoctoproject.org/wiki/Testing_Toaster
*please validate and add info from Testing Toaster wiki in this section(Test Strategy): https://wiki.yoctoproject.org/wiki/Testing_Toaster
== Test automation ==
Toaster has 3 testing suites, targeted at verifying different parts of the system.
=== Django Unit Tests ===
Toaster is primarely a Django application. As such, it makes use of the Django test system that runs unit tests on mock data to ensure pieces of functionality work as expected, and prevent regressions.
To run the django unit tests, invoke them as with any Django test suite
./bitbake/lib/toaster/manage.py test
To add unit tests, simply add needed tests to the _tests.py_ file in the module you're editing.
=== Toaster Test System (TTS) ===
The TTS is a suite aimed at running integration and acceptance tests. Currently, it verifying the correct syntax and form of the Python code, tries to start Toaster in both modes, and runs HTML5 validation on HTML-returning pages.
The tests are designed to be triggered through a continuous integration system.
To start manually it, run the tts test runner:
./bitbake/lib/toaster/contrib/tts/runner.py
=== The oe-selftest toaster tests ===
These tests are verifying the correctness of the data gathered by the build process - it tests the bitbake/lib/bb/ui/toasterui.py and bitbake/lib/bb/ui/buildinfohelper.py files and the interface to the bitbake server process.
In order for these tests to run properly, the following steps need to be taken before running, on a clean toaster setup:
*Create new project using local poky release
*Create 1 core-image-minimal image
*Create 1 core-image-sato image to check building from cache
*Break a recipe file (by modifying a link, for example)
*Create a new image using broken recipe to check error detection is working
To run these tests you need to source an environment by issuing:
  source oe-init-build-env
then modify the bblayers.conf file to add the meta-selftest layer, create a symbolic link to the toaster.sqlite file:
  ln -s ../toaster.sqlite toaster.sqlite
and then run the command:
  oe-selftest --run-tests _toaster
=== The selenium automated UI tests ===
The automated UI tests are targeted at validating and identifying regressions in the UI pages. These tests are based on the Selenium framework to drive interactions in the interface.
The selenium tests require a couple of components:
*scrot (a screen capture utility) and pip - a utility for installing python modules. The following is how you install it for ubuntu:
sudo apt-get install scrot python-pip python-virtualenv
virtualenv t-selenium
source ./t-selenium/bin/activate
pip install selenium
*Since this is a web application, the UI can also be tested from a mac. To install the requirements there do:
sudo easy_install pip
sudo pip install virtualenv
virtualenv t-selenium
source ./t-selenium/bin/activate
*also install the latest filefox extension from selenium located at: http://release.seleniumhq.org/selenium-ide/ and install all the extensions.
The selenium tests are located in:
./ bitbake/lib/toaster/contrib/tts/toasteruitest
You run them with:
./bitbake/lib/toaster/contrib/tts/toasteruitest/run_toastertests.py
The above command takes the list of the tests to run from the toaster_test.cfg file. The tests themselves are located in toaster_automation_test.py. The toaster_test.cfg file has a couple of important variables in it separated by which os you are running the tests on. These varables control where the toaster instance is located (defaults to http://127.0.0.1:8000) what browser to use (firefox is safest for selenium though there is a chrome driver available as well), what tests to run and what log level to save data at.
In order for these tests to run properly, the following steps need to be taken before running, on a clean toaster setup:
*Create new project named "selenium-project" using local poky release
*Create 1 core-image-minimal image
*Break a recipe file (by modifying a link, for example)
*Create a new image using broken recipe to check error detection is working
*Fix the recipe broken in previous step
*Create 1 core-image-sato image
In order to run all current tests, you need to modify the configuration file "toaster_test.cfg" with the following line:
test_cases = [901, 902, 903, 904, 906, 910, 911, 912, 913, 914, 915, 916, 923, 924, 940, 941, 942, 943, 944, 945, 946, 947, 948, 949, 950, 951, 955, 956]


= Test Approach =
= Test Approach =

Revision as of 13:20, 4 December 2015

This article is the test plan for Toaster.

Objectives

Verify all Toaster components to be fully functional.

Components to be verified:

  • UI interface
  • Backend interaction with bitbake for build purposes
  • Backend interaction with database for storing and accessing build information


The testing objective involves only positive testing for existing features on Toaster. Perform exploratory testing focusing on newer features

  • this can sometimes generate new test cases.

Verifying the bugs and features.

Team members

Scope

Testing scope has been agreed upon as local installation only, using sqlite database on Fedora and Ubuntu operating systems on Firefox browser.

New features are to be tested as they are finished with test cases written for them once features are stabilized.

UI features being tested:

  • Create new project
  • Modify project settings and variables
  • Add/Remove layers to project
  • Building images
  • Build details
  • All builds report table

Backend features:

  • Toaster installation and first time configuration
  • Data presence
  • Data consistency
  • Build data

Test Strategy

There are several test approaches for Toaster Testing, such as:

  • Perform test cases agreed upon the development during periodic full passes according to the Schedule
  • Write new test cases based on developer request or new features requires it.
    • Maintain current test cases and update them accordingly
  • Perform exploratory testing on existing functionalities.
    • YP 2.1 Testopia template here.

Major feature groups and their testing approach

FrontEnd - UI Basic functionalities

Toaster UI features are tested such as links, buttons, tables, pages basic functionalities.

Testing needs :


BackEnd functionalities

Covers data integrity, data transfer between Bitbake and Toaster, Start/Stop, building images, configuration, etc

Testing needs:


Test automation

Toaster has 3 testing suites, targeted at verifying different parts of the system.

Django Unit Tests

Toaster is primarely a Django application. As such, it makes use of the Django test system that runs unit tests on mock data to ensure pieces of functionality work as expected, and prevent regressions.

To run the django unit tests, invoke them as with any Django test suite

./bitbake/lib/toaster/manage.py test

To add unit tests, simply add needed tests to the _tests.py_ file in the module you're editing.

Toaster Test System (TTS)

The TTS is a suite aimed at running integration and acceptance tests. Currently, it verifying the correct syntax and form of the Python code, tries to start Toaster in both modes, and runs HTML5 validation on HTML-returning pages. The tests are designed to be triggered through a continuous integration system.

To start manually it, run the tts test runner:

./bitbake/lib/toaster/contrib/tts/runner.py


The oe-selftest toaster tests

These tests are verifying the correctness of the data gathered by the build process - it tests the bitbake/lib/bb/ui/toasterui.py and bitbake/lib/bb/ui/buildinfohelper.py files and the interface to the bitbake server process.

In order for these tests to run properly, the following steps need to be taken before running, on a clean toaster setup:

  • Create new project using local poky release
  • Create 1 core-image-minimal image
  • Create 1 core-image-sato image to check building from cache
  • Break a recipe file (by modifying a link, for example)
  • Create a new image using broken recipe to check error detection is working


To run these tests you need to source an environment by issuing:

 source oe-init-build-env

then modify the bblayers.conf file to add the meta-selftest layer, create a symbolic link to the toaster.sqlite file:

 ln -s ../toaster.sqlite toaster.sqlite

and then run the command:

 oe-selftest --run-tests _toaster


The selenium automated UI tests

The automated UI tests are targeted at validating and identifying regressions in the UI pages. These tests are based on the Selenium framework to drive interactions in the interface.

The selenium tests require a couple of components:

  • scrot (a screen capture utility) and pip - a utility for installing python modules. The following is how you install it for ubuntu:
sudo apt-get install scrot python-pip python-virtualenv
virtualenv t-selenium
source ./t-selenium/bin/activate
pip install selenium
  • Since this is a web application, the UI can also be tested from a mac. To install the requirements there do:
sudo easy_install pip
sudo pip install virtualenv
virtualenv t-selenium
source ./t-selenium/bin/activate

The selenium tests are located in:

./ bitbake/lib/toaster/contrib/tts/toasteruitest

You run them with:

./bitbake/lib/toaster/contrib/tts/toasteruitest/run_toastertests.py

The above command takes the list of the tests to run from the toaster_test.cfg file. The tests themselves are located in toaster_automation_test.py. The toaster_test.cfg file has a couple of important variables in it separated by which os you are running the tests on. These varables control where the toaster instance is located (defaults to http://127.0.0.1:8000) what browser to use (firefox is safest for selenium though there is a chrome driver available as well), what tests to run and what log level to save data at.

In order for these tests to run properly, the following steps need to be taken before running, on a clean toaster setup:

  • Create new project named "selenium-project" using local poky release
  • Create 1 core-image-minimal image
  • Break a recipe file (by modifying a link, for example)
  • Create a new image using broken recipe to check error detection is working
  • Fix the recipe broken in previous step
  • Create 1 core-image-sato image

In order to run all current tests, you need to modify the configuration file "toaster_test.cfg" with the following line:

test_cases = [901, 902, 903, 904, 906, 910, 911, 912, 913, 914, 915, 916, 923, 924, 940, 941, 942, 943, 944, 945, 946, 947, 948, 949, 950, 951, 955, 956]

Test Approach

Overall approach to testing. Specify major activities, techniques and tools used to test groups of features

Regression Testing

Performance and stress testing

Test Automation

Test environment

Deliverables

Test schedule

Deliverables

  • Features to be tested

Resources roles & responsibilities

Release Criteria/ Exit Criteria

Dependencies

  • Personal Dependencies
  • Software Dependencies
  • Hardware Dependencies

Risks

Schedule

Technical

Requirements

Tools

Documentation