Toaster testing plan: Difference between revisions
From Yocto Project
Jump to navigationJump to search
No edit summary |
No edit summary |
||
(20 intermediate revisions by one other user not shown) | |||
Line 1: | Line 1: | ||
[[Category:Toaster]] | |||
This article is the test plan for [https://wiki.yoctoproject.org/wiki/WebHob Toaster]. | This article is the test plan for [https://wiki.yoctoproject.org/wiki/WebHob Toaster]. | ||
Line 5: | Line 5: | ||
The test process focuses on: | The test process focuses on: | ||
* validating the data collected from the build process | * validating the data collected from the build process; | ||
* verifying the correct functioning of the Toaster GUI | * verifying the correct functioning of the Toaster GUI; TBD | ||
= Test Areas = | = Test Areas = | ||
Line 14: | Line 14: | ||
==== Functionality tests ==== | ==== Functionality tests ==== | ||
* [[REST_API_Contracts | REST API]] verification – create Django tests to detect API calls returning | * [[REST_API_Contracts | REST API]] verification – create Django tests to detect API calls returning incomplete (fields having null values) or wrong data for a certain set of entries from each table - our goal is to have 90% data collected. The tests can be found [http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/commit/?h=andreeap/toaster-tests&id=7cfd179be5db2a5530d60093fd09d0240138c2fa here]; | ||
* Calculation of the data collection rate - the ratio | * Calculation of the data collection rate - the ratio between the number of the variables having null values and the total number of variables collected (the data collection rate does not include wrong values); Run: ./fail_rate.py path_to_toaster.sqlite_file. Output: table_name field_name value_that_never_changes and for each table a percentage: (the number of occurences of all the fields that never change their values)/(total number of entries for that table); | ||
* Verify that all links in the simple UI are available; | * Verify that all links in the simple UI are available; | ||
* Verify the quality of the data collected through the simple UI; | * Verify the quality of the data collected through the simple UI; | ||
* Verify the quality of the data collected by verifying most of the values from the database by connecting directly to the database - our immediate goal is to have 90% correct data collected; | |||
==== Usability tests ==== | ==== Usability tests ==== | ||
* Verify the easy usage of Toaster ([https://wiki.yoctoproject.org/wiki/WebHob#Installation_and_Running easy to install and stop | * Verify the easy usage of Toaster ([https://wiki.yoctoproject.org/wiki/WebHob#Installation_and_Running easy to install and start/stop the toaster server]) | ||
== Frontend == | == Frontend == | ||
Line 29: | Line 30: | ||
==== Compatibility tests ==== | ==== Compatibility tests ==== | ||
* Verify the behavior of the GUI on different browsers and operating systems; | * Verify the behavior of the GUI on different browsers and operating systems; TBD | ||
==== Usability tests ==== | ==== Usability tests ==== | ||
Line 40: | Line 41: | ||
= Test Cycle = | = Test Cycle = | ||
{|class="wikitable" style="text-align: center;" | {|class="wikitable" style="text-align: center;" | ||
| || || colspan=" | | || || colspan="3" | Test execution cycle | ||
|- | |- | ||
| || || [[#Weekly Test]] || [[#Full Pass Test]] | | || || [[#Weekly Test]] || [[#Full Pass Test]] || [[#Release Test]] | ||
|- | |- | ||
| Build type || Weekly || Yes || Yes | | rowspan="2" | Build type || Weekly || Yes || Yes || | ||
|- | |- | ||
| | | Release || Yes || Yes || Yes | ||
|- | |- | ||
| [[# | | rowspan="2" | [[#Test Areas]] || [[#Backend]] || Yes || Yes || Yes | ||
|- | |- | ||
| | | [[#Frontend]] || Yes || Yes || Yes | ||
|- | |- | ||
| | | rowspan="5" | Target machine || qemuarm || || Yes || Yes | ||
|- | |- | ||
| | | qemumips|| || || Yes | ||
|- | |- | ||
| | | qemuppc|| || || Yes | ||
|- | |- | ||
| | | qemux86|| Yes || Yes || Yes | ||
|- | |- | ||
| core-image-sato|| || Yes | | qemux86-64 || || || Yes | ||
|- | |||
| rowspan="3" | Target image|| core-image-minimal|| Yes || || | |||
|- | |||
| core-image-sato-sdk|| || Yes || Yes | |||
|} | |} | ||
Line 78: | Line 83: | ||
*; Objective: | *; Objective: | ||
** Ensure functionality of Toaster component. | ** Ensure functionality of Toaster component. | ||
== Release Test == | |||
*; Scope: | |||
** Release candidates that pass [[#Full Pass Test]] | |||
*; Objective: | |||
** All scheduled features are covered, or rescheduled; | |||
** All relevant bugs are fixed and verified. |
Revision as of 15:47, 16 May 2014
This article is the test plan for Toaster.
Objectives
The test process focuses on:
- validating the data collected from the build process;
- verifying the correct functioning of the Toaster GUI; TBD
Test Areas
Toaster consists of two big components, as follows:
Backend
Functionality tests
- REST API verification – create Django tests to detect API calls returning incomplete (fields having null values) or wrong data for a certain set of entries from each table - our goal is to have 90% data collected. The tests can be found here;
- Calculation of the data collection rate - the ratio between the number of the variables having null values and the total number of variables collected (the data collection rate does not include wrong values); Run: ./fail_rate.py path_to_toaster.sqlite_file. Output: table_name field_name value_that_never_changes and for each table a percentage: (the number of occurences of all the fields that never change their values)/(total number of entries for that table);
- Verify that all links in the simple UI are available;
- Verify the quality of the data collected through the simple UI;
- Verify the quality of the data collected by verifying most of the values from the database by connecting directly to the database - our immediate goal is to have 90% correct data collected;
Usability tests
- Verify the easy usage of Toaster (easy to install and start/stop the toaster server)
Frontend
Functionality tests
- Manual testing in the first stage;
- Automate testing using Selenium, in the second stage;
Compatibility tests
- Verify the behavior of the GUI on different browsers and operating systems; TBD
Usability tests
- Verify if the GUI design is as described here: http://yoctoproject.org/webhob;
- Friendly graphical user interface;
Performance tests
- Stress testing (e.g. display appropriate error messages when the system is under stress);
Test Cycle
Test execution cycle | ||||
#Weekly Test | #Full Pass Test | #Release Test | ||
Build type | Weekly | Yes | Yes | |
Release | Yes | Yes | Yes | |
#Test Areas | #Backend | Yes | Yes | Yes |
#Frontend | Yes | Yes | Yes | |
Target machine | qemuarm | Yes | Yes | |
qemumips | Yes | |||
qemuppc | Yes | |||
qemux86 | Yes | Yes | Yes | |
qemux86-64 | Yes | |||
Target image | core-image-minimal | Yes | ||
core-image-sato-sdk | Yes | Yes |
Weekly Test
- Scope
- Images built weekly and released through the distribution team.
- Objective
- Functionality test on most areas with minimum sets of tests;
- Regression test with high probability to find bugs.
Full Pass Test
- Scope
- Images built as candidates for milestone or final release;
- Passed #Weekly Test
- Objective
- Ensure functionality of Toaster component.
Release Test
- Scope
- Release candidates that pass #Full Pass Test
- Objective
- All scheduled features are covered, or rescheduled;
- All relevant bugs are fixed and verified.