Toaster testing plan: Difference between revisions
From Yocto Project
Jump to navigationJump to search
No edit summary |
|||
(44 intermediate revisions by 2 users not shown) | |||
Line 1: | Line 1: | ||
[[Category:Toaster]] | |||
This article is the test plan for [https://wiki.yoctoproject.org/wiki/WebHob Toaster]. | This article is the test plan for [https://wiki.yoctoproject.org/wiki/WebHob Toaster]. | ||
= Objectives = | = Objectives = | ||
The test process | The test process focuses on: | ||
* validating the data collected from the build process; | |||
* verifying the correct functioning of the Toaster GUI; TBD | |||
= Test Areas = | = Test Areas = | ||
Toaster consists of two big components, as follows: | |||
== Backend == | |||
==== Functionality tests ==== | ==== Functionality tests ==== | ||
* REST | * [[REST_API_Contracts | REST API]] verification – create Django tests to detect API calls returning incomplete (fields having null values) or wrong data for a certain set of entries from each table - our goal is to have 90% data collected. The tests can be found [http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/commit/?h=andreeap/toaster-tests&id=7cfd179be5db2a5530d60093fd09d0240138c2fa here]; | ||
* Calculation of the data collection rate; | * Calculation of the data collection rate - the ratio between the number of the variables having null values and the total number of variables collected (the data collection rate does not include wrong values); Run: ./fail_rate.py path_to_toaster.sqlite_file. Output: table_name field_name value_that_never_changes and for each table a percentage: (the number of occurences of all the fields that never change their values)/(total number of entries for that table); | ||
* Verify | * Verify that all links in the simple UI are available; | ||
* Verify the quality of the data collected through the simple UI; | * Verify the quality of the data collected through the simple UI; | ||
* Verify the quality of the data collected by verifying most of the values from the database by connecting directly to the database - our immediate goal is to have 90% correct data collected; | |||
==== Usability tests ==== | ==== Usability tests ==== | ||
* Verify the easy usage of | * Verify the easy usage of Toaster ([https://wiki.yoctoproject.org/wiki/WebHob#Installation_and_Running easy to install and start/stop the toaster server]) | ||
== Frontend == | |||
=== Functionality tests === | ==== Functionality tests ==== | ||
* Manual testing in the first stage; | * Manual testing in the first stage; | ||
* Automate testing using [http://www.seleniumhq.org/ Selenium], in the second stage; | * Automate testing using [http://www.seleniumhq.org/ Selenium], in the second stage; | ||
=== Compatibility tests === | ==== Compatibility tests ==== | ||
* Verify the behavior of the GUI on different browsers and operating systems; | * Verify the behavior of the GUI on different browsers and operating systems; TBD | ||
=== Usability tests === | ==== Usability tests ==== | ||
* Verify if the GUI design is as described here: | * Verify if the GUI design is as described here: http://yoctoproject.org/webhob; | ||
* Friendly graphical user interface; | * Friendly graphical user interface; | ||
=== Performance tests === | ==== Performance tests ==== | ||
* Stress testing (e.g. display appropriate error messages when the system is under stress); | * Stress testing (e.g. display appropriate error messages when the system is under stress); | ||
= Test Cycle = | |||
{|class="wikitable" style="text-align: center;" | |||
| || || colspan="3" | Test execution cycle | |||
|- | |||
| || || [[#Weekly Test]] || [[#Full Pass Test]] || [[#Release Test]] | |||
|- | |||
| rowspan="2" | Build type || Weekly || Yes || Yes || | |||
|- | |||
| Release || Yes || Yes || Yes | |||
|- | |||
| rowspan="2" | [[#Test Areas]] || [[#Backend]] || Yes || Yes || Yes | |||
|- | |||
| [[#Frontend]] || Yes || Yes || Yes | |||
|- | |||
| rowspan="5" | Target machine || qemuarm || || Yes || Yes | |||
|- | |||
| qemumips|| || || Yes | |||
|- | |||
| qemuppc|| || || Yes | |||
|- | |||
| qemux86|| Yes || Yes || Yes | |||
|- | |||
| qemux86-64 || || || Yes | |||
|- | |||
| rowspan="3" | Target image|| core-image-minimal|| Yes || || | |||
|- | |||
| core-image-sato-sdk|| || Yes || Yes | |||
|} | |||
== Weekly Test == | |||
*; Scope: | |||
** Images built weekly and released through the distribution team. | |||
*; Objective: | |||
** Functionality test on most areas with minimum sets of tests; | |||
** Regression test with high probability to find bugs. | |||
== Full Pass Test == | |||
*; Scope: | |||
** Images built as candidates for milestone or final release; | |||
** Passed [[#Weekly Test]] | |||
*; Objective: | |||
** Ensure functionality of Toaster component. | |||
== Release Test == | |||
*; Scope: | |||
** Release candidates that pass [[#Full Pass Test]] | |||
*; Objective: | |||
** All scheduled features are covered, or rescheduled; | |||
** All relevant bugs are fixed and verified. |
Revision as of 15:47, 16 May 2014
This article is the test plan for Toaster.
Objectives
The test process focuses on:
- validating the data collected from the build process;
- verifying the correct functioning of the Toaster GUI; TBD
Test Areas
Toaster consists of two big components, as follows:
Backend
Functionality tests
- REST API verification – create Django tests to detect API calls returning incomplete (fields having null values) or wrong data for a certain set of entries from each table - our goal is to have 90% data collected. The tests can be found here;
- Calculation of the data collection rate - the ratio between the number of the variables having null values and the total number of variables collected (the data collection rate does not include wrong values); Run: ./fail_rate.py path_to_toaster.sqlite_file. Output: table_name field_name value_that_never_changes and for each table a percentage: (the number of occurences of all the fields that never change their values)/(total number of entries for that table);
- Verify that all links in the simple UI are available;
- Verify the quality of the data collected through the simple UI;
- Verify the quality of the data collected by verifying most of the values from the database by connecting directly to the database - our immediate goal is to have 90% correct data collected;
Usability tests
- Verify the easy usage of Toaster (easy to install and start/stop the toaster server)
Frontend
Functionality tests
- Manual testing in the first stage;
- Automate testing using Selenium, in the second stage;
Compatibility tests
- Verify the behavior of the GUI on different browsers and operating systems; TBD
Usability tests
- Verify if the GUI design is as described here: http://yoctoproject.org/webhob;
- Friendly graphical user interface;
Performance tests
- Stress testing (e.g. display appropriate error messages when the system is under stress);
Test Cycle
Test execution cycle | ||||
#Weekly Test | #Full Pass Test | #Release Test | ||
Build type | Weekly | Yes | Yes | |
Release | Yes | Yes | Yes | |
#Test Areas | #Backend | Yes | Yes | Yes |
#Frontend | Yes | Yes | Yes | |
Target machine | qemuarm | Yes | Yes | |
qemumips | Yes | |||
qemuppc | Yes | |||
qemux86 | Yes | Yes | Yes | |
qemux86-64 | Yes | |||
Target image | core-image-minimal | Yes | ||
core-image-sato-sdk | Yes | Yes |
Weekly Test
- Scope
- Images built weekly and released through the distribution team.
- Objective
- Functionality test on most areas with minimum sets of tests;
- Regression test with high probability to find bugs.
Full Pass Test
- Scope
- Images built as candidates for milestone or final release;
- Passed #Weekly Test
- Objective
- Ensure functionality of Toaster component.
Release Test
- Scope
- Release candidates that pass #Full Pass Test
- Objective
- All scheduled features are covered, or rescheduled;
- All relevant bugs are fixed and verified.