Toaster testing plan

This article is the test plan for Toaster.

Objectives

Verify all Toaster components to be fully functional.

Components to be verified:

• UI interface
• Backend interaction with bitbake for build purposes
• Backend interaction with database for storing and accessing build information

The testing objective involves only positive testing for existing features on Toaster. Perform exploratory testing focusing on newer features

• this can sometimes generate new test cases.

Verifying the bugs and features.

Scope

Testing scope has been agreed upon as local installation only, using sqlite database on Fedora and Ubuntu operating systems on Firefox browser.

New features are to be tested as they are finished with test cases written for them once features are stabilized.

UI features being tested:

• Create new project
• Modify project settings and variables
• Add/Remove layers to project
• Building images
• Build details
• All builds report table

Backend features:

• Toaster installation and first time configuration
• Data presence
• Data consistency
• Build data

 

Test Strategy

There are several test approaches for Toaster Testing, such as:

• Perform test cases agreed upon the development during periodic full passes according to the Schedule
• Write new test cases based on developer request or new features requires it.
  • Maintain current test cases and update them accordingly
• Perform exploratory testing on existing functionalities.
  • YP 2.1 Testopia template here.

Major feature groups and their testing approach

FrontEnd - UI Basic functionalities

Toaster UI features are tested such as links, buttons, tables, pages basic functionalities.

Testing needs :

• Built only in python 2.7
• Latest version of Selenium as python module
• Latest version of Firefox
• In order to test Selenium automated based tests (https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4027 ), the following steps need to be taken before running, on a clean toaster setup
  • ckeck the steps and requirements for “The Selenium automated UI tests” from https://wiki.yoctoproject.org/wiki/Testing_Toaster
• For UI manual testing follow the steps from https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4029 TCs
  • currently working on their automation using Selenium

BackEnd functionalities

Covers data integrity, data transfer between Bitbake and Toaster, Start/Stop, building images, configuration, etc

Testing needs:

• Built in python 2.7
• Direct sql requests to local sqlite instance
• Requires sqlite3 client installed (for manual debugging/investigation)
• BackEnd automated tests are verifying the correctness of the data gathered by the build process as presented in the Backend template: https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4028
  • In order for these tests to run properly, check the the steps for “The oe-selftest toaster tests” from https://wiki.yoctoproject.org/wiki/Testing_Toaster
• For BackEnd managed mode manual testing follow the steps from the following template https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4030
  • currently working on their automation
  • AR Cristina + Mihai:
• please validate and add info from Testing Toaster wiki in this section(Test Strategy): https://wiki.yoctoproject.org/wiki/Testing_Toaster

Test automation

Toaster has 3 testing suites, targeted at verifying different parts of the system.

Django Unit Tests

Toaster is primarely a Django application. As such, it makes use of the Django test system that runs unit tests on mock data to ensure pieces of functionality work as expected, and prevent regressions.

To run the django unit tests, invoke them as with any Django test suite

./bitbake/lib/toaster/manage.py test

To add unit tests, simply add needed tests to the _tests.py_ file in the module you're editing.

Toaster Test System (TTS)

The TTS is a suite aimed at running integration and acceptance tests. Currently, it verifying the correct syntax and form of the Python code, tries to start Toaster in both modes, and runs HTML5 validation on HTML-returning pages. The tests are designed to be triggered through a continuous integration system.

To start manually it, run the tts test runner:

./bitbake/lib/toaster/contrib/tts/runner.py

The oe-selftest toaster tests

These tests are verifying the correctness of the data gathered by the build process - it tests the bitbake/lib/bb/ui/toasterui.py and bitbake/lib/bb/ui/buildinfohelper.py files and the interface to the bitbake server process.

In order for these tests to run properly, the following steps need to be taken before running, on a clean toaster setup:

• Create new project using local poky release
• Create 1 core-image-minimal image
• Create 1 core-image-sato image to check building from cache
• Break a recipe file (by modifying a link, for example)
• Create a new image using broken recipe to check error detection is working

To run these tests you need to source an environment by issuing:

 source oe-init-build-env

then modify the bblayers.conf file to add the meta-selftest layer, create a symbolic link to the toaster.sqlite file:

 ln -s ../toaster.sqlite toaster.sqlite

and then run the command:

 oe-selftest --run-tests _toaster

The selenium automated UI tests

The automated UI tests are targeted at validating and identifying regressions in the UI pages. These tests are based on the Selenium framework to drive interactions in the interface.

The selenium tests require a couple of components:

• scrot (a screen capture utility) and pip - a utility for installing python modules. The following is how you install it for ubuntu:

sudo apt-get install scrot python-pip python-virtualenv
virtualenv t-selenium
source ./t-selenium/bin/activate
pip install selenium

• Since this is a web application, the UI can also be tested from a mac. To install the requirements there do:

sudo easy_install pip
sudo pip install virtualenv
virtualenv t-selenium
source ./t-selenium/bin/activate

• also install the latest filefox extension from selenium located at: http://release.seleniumhq.org/selenium-ide/ and install all the extensions.

The selenium tests are located in:

./ bitbake/lib/toaster/contrib/tts/toasteruitest

You run them with:

./bitbake/lib/toaster/contrib/tts/toasteruitest/run_toastertests.py

The above command takes the list of the tests to run from the toaster_test.cfg file. The tests themselves are located in toaster_automation_test.py. The toaster_test.cfg file has a couple of important variables in it separated by which os you are running the tests on. These varables control where the toaster instance is located (defaults to http://127.0.0.1:8000) what browser to use (firefox is safest for selenium though there is a chrome driver available as well), what tests to run and what log level to save data at.

In order for these tests to run properly, the following steps need to be taken before running, on a clean toaster setup:

• Create new project named "selenium-project" using local poky release
• Create 1 core-image-minimal image
• Break a recipe file (by modifying a link, for example)
• Create a new image using broken recipe to check error detection is working
• Fix the recipe broken in previous step
• Create 1 core-image-sato image

In order to run all current tests, you need to modify the configuration file "toaster_test.cfg" with the following line:

test_cases = [901, 902, 903, 904, 906, 910, 911, 912, 913, 914, 915, 916, 923, 924, 940, 941, 942, 943, 944, 945, 946, 947, 948, 949, 950, 951, 955, 956]

Test Approach

Sanity testing

• Not covered at this moment
• TBD following DEV discussions

Performance and Stress

• Testing two concurrent builds
• TC…

Load testing

• Not covered
• TBD

== System Integration Testing

• Not covered (maybe covered by DEV w/ Jenkins)

Regression

• All Full pass tests for
  • BackEnd
  • FrontEnd

Maintaining the test cases

Toaster test cases are officially stored in Testopia, the Yocto Project test case management system. Testopia is a test case management platform and an extension for Bugzilla developed by the Mozilla Foundation that helps organize test cases using a tight relation with elements in Bugzilla. More details on Testopia, can be found on the YP Wiki at https://wiki.yoctoproject.org/wiki/Testopia.

A quick start guide for Testopia is found on the Testopia wiki at https://wiki.yoctoproject.org/wiki/Testopia#What_do_I_need_to_start_using_Testopia.3F.

As currently Testopia is structured, all Toaster test cases are stored under Classifications/Build System & Metadata/ Toaster component within Testopia. All the test cases are organized in Test Run Templates as following:

• Toaster Backend - Weekly: https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4028
• Toaster Backend - Managed Mode: https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4030
• Toaster UI: https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4027
• Toaster UI - Managed Mode: https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4029
• Toaster Exploratory - https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4043

Submitting Bugs

Being part of the Yocto Project, Toaster follows the same Yocto Project guidelines and principles. The guidelines can be found at https://wiki.yoctoproject.org/wiki/Community_Guidelines. Toaster bugs are no different and are tracked into Bugzilla, the official Yocto Project bug tracker. Learn more about our process for reporting bugs.

Requirements

HW Requirements

A host system with a minimum of 50 Gbytes of free disk space that is running a supported Linux distribution

Software Requirements

• Django 1.6
• South 0.8.4
• argparse 1.2.1
• wsgiref 0.1.2
• beautifulsoup4 4.4.0 or greater
• A release of the Yocto Project: http://git.yoctoproject.org/cgit/cgit.cgi/poky/
• YP Quick Start requirements
  • Git 1.7.8 or greater
  • tar 1.24 or greater
  • Python 2.7.3

Environment Requirements

• Supported Browsers – no official browser selected yet, usually Firefox is used
• Supported operating systems as per SANITY_TESTED_DISTROS listed in meta-yocto/conf/distro/poky.conf
  • YP 2.0 Release:
  • Ubuntu-14.04
  • Ubuntu-14.10
  • Ubuntu-15.04
  • Fedora-21
  • CentOS-6.*
  • CentOS-7.*
  • Debian-7.*
  • Debian-8.*
  • openSUSE-project-13.2

• Latest version of Selenium installed as python module
• YP QS (Maybe, depending where Toaster server is installed)
• SQLITE 3 Client

Deliverables

Deliverables

• Features to be tested

Dependencies

• Personal Dependencies
• Software Dependencies
• Hardware Dependencies