Toaster testing plan

From Yocto Project
Jump to navigationJump to search

This article is the test plan for Toaster.

Objectives

Verify all Toaster components to be fully functional.

Components to be verified:

  • UI interface
  • Backend interaction with bitbake for build purposes
  • Backend interaction with database for storing and accessing build information

Test Areas

Toaster consists of two big components, as follows:

Backend

Functionality tests

  • REST API verification – create Django tests to detect API calls returning incomplete (fields having null values) or wrong data for a certain set of entries from each table - our goal is to have 90% data collected. The tests can be found here;
  • Calculation of the data collection rate - the ratio between the number of the variables having null values ​​and the total number of variables collected (the data collection rate does not include wrong values); Run: ./fail_rate.py path_to_toaster.sqlite_file. Output: table_name field_name value_that_never_changes and for each table a percentage: (the number of occurences of all the fields that never change their values)/(total number of entries for that table);
  • Verify that all links in the simple UI are available;
  • Verify the quality of the data collected through the simple UI;
  • Verify the quality of the data collected by verifying most of the values from the database by connecting directly to the database - our immediate goal is to have 90% correct data collected;

Usability tests

Frontend

Functionality tests

  • Manual testing in the first stage;
  • Automate testing using Selenium, in the second stage;

Compatibility tests

  • Verify the behavior of the GUI on different browsers and operating systems; TBD

Usability tests

Performance tests

  • Stress testing (e.g. display appropriate error messages when the system is under stress);

Test Cycle

Test execution cycle
#Weekly Test #Full Pass Test #Release Test
Build type Weekly Yes Yes
Release Yes Yes Yes
#Test Areas #Backend Yes Yes Yes
#Frontend Yes Yes Yes
Target machine qemuarm Yes Yes
qemumips Yes
qemuppc Yes
qemux86 Yes Yes Yes
qemux86-64 Yes
Target image core-image-minimal Yes
core-image-sato-sdk Yes Yes

Weekly Test

  • Scope
    • Images built weekly and released through the distribution team.
  • Objective
    • Functionality test on most areas with minimum sets of tests;
    • Regression test with high probability to find bugs.

Full Pass Test

  • Scope
    • Images built as candidates for milestone or final release;
    • Passed #Weekly Test
  • Objective
    • Ensure functionality of Toaster component.

Release Test

  • Objective
    • All scheduled features are covered, or rescheduled;
    • All relevant bugs are fixed and verified.