Toaster testing plan: Difference between revisions

From Yocto Project
Jump to navigationJump to search
No edit summary
(28 intermediate revisions by one other user not shown)
Line 1: Line 1:
= Introduction =
[[Category:Toaster]]
This article is the test plan for [https://wiki.yoctoproject.org/wiki/WebHob Toaster].
This article is the test plan for [https://wiki.yoctoproject.org/wiki/WebHob Toaster].


Line 5: Line 5:
The test process focuses on:
The test process focuses on:
   
   
* validating the data collected from the build process
* validating the data collected from the build process;
* verifying the correct functioning of the Toaster GUI.
* verifying the correct functioning of the Toaster GUI; TBD


= Test Areas =
= Test Areas =
Toaster consists of two big components, as follows:
Toaster consists of two big components, as follows:


=== 1. Backend ===
== Backend ==


==== Functionality tests ====
==== Functionality tests ====
*  [[REST_API_Contracts | REST API]]  verification – create Django tests to detect API calls returning no data;  
*  [[REST_API_Contracts | REST API]]  verification – create Django tests to detect API calls returning incomplete (fields having null values) or wrong data for a certain set of entries from each table - our goal is to have 90% data collected. The tests can be found [http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/commit/?h=andreeap/toaster-tests&id=7cfd179be5db2a5530d60093fd09d0240138c2fa here];
*  Calculation of the data collection rate - the ratio of the number of the variables having null values ​​and the total number of variables collected;
*  Calculation of the data collection rate - the ratio between the number of the variables having null values ​​and the total number of variables collected (the data collection rate does not include wrong values); Run: ./fail_rate.py path_to_toaster.sqlite_file. Output: table_name field_name value_that_never_changes and for each table a percentage: (the number of occurences of all the fields that never change their values)/(total number of entries for that table);
*  Verify that all links in the simple UI are available;
*  Verify that all links in the simple UI are available;
*  Verify the quality of the data collected through the simple UI;
*  Verify the quality of the data collected through the simple UI;
*  Verify the quality of the data collected by verifying most of the values from the database by connecting directly to the database - our immediate goal is to have 90% correct data collected;


==== Usability tests ====
==== Usability tests ====
*  Verify the easy usage of Toaster ([https://wiki.yoctoproject.org/wiki/WebHob#Installation_and_Running easy to install and stop/start the toaster server])
*  Verify the easy usage of Toaster ([https://wiki.yoctoproject.org/wiki/WebHob#Installation_and_Running easy to install and start/stop the toaster server])


=== 2. Frontend ===
== Frontend ==


==== Functionality tests ====
==== Functionality tests ====
Line 29: Line 30:


==== Compatibility tests ====
==== Compatibility tests ====
*  Verify the behavior of the GUI on different browsers and operating systems;
*  Verify the behavior of the GUI on different browsers and operating systems; TBD


==== Usability tests ====  
==== Usability tests ====  
Line 38: Line 39:
*  Stress testing (e.g. display appropriate error messages when the system is under stress);
*  Stress testing (e.g. display appropriate error messages when the system is under stress);


 
= Test Cycle =
{|class="wikitable" style="text-align: center;"
{|class="wikitable" style="text-align: center;"
| ||
| || || colspan="3" | Test execution cycle
!colspan="4"|Test execution cycle
|-
|-
| ||  
| || || [[#Weekly Test]] || [[#Full Pass Test]] || [[#Release Test]]
! [[#Weekly Test]] || [[#Full Pass Test]]
|-
|-
! rowspan="2" |Build type  
| rowspan="2" | Build type || Weekly  || Yes || Yes ||
|-
|-
! Weekly
| Release || Yes || Yes || Yes
| yes ||
|-
|-
! rowspan="3" |Build type
| rowspan="2" | [[#Test Areas]] || [[#Backend]] || Yes || Yes || Yes
! [[#Backend]]
| yes || yes
|-
|-
! [[#Frontend]]
| [[#Frontend]] || Yes || Yes || Yes
| yes ||  
|-
|-
! rowspan="5" |Target machine
| rowspan="5" | Target machine || qemuarm || || Yes || Yes
|-
|-
! qemuarm
| qemumips|| || || Yes
| yes || yes
|-
|-
! qemumips
| qemuppc|| || || Yes
| || yes
|-
|-
! qemuppc
| qemux86|| Yes || Yes || Yes
| || yes
|-
|-
! qemux86
| qemux86-64  ||  || || Yes
| yes || yes
|-
|-
| rowspan="3" | Target image|| core-image-minimal|| Yes || ||
|-
| core-image-sato-sdk|| || Yes || Yes
|}
== Weekly Test ==
*; Scope:
** Images built weekly and released through the distribution team.
*; Objective:
** Functionality test on most areas with minimum sets of tests;
** Regression test with high probability to find bugs.
== Full Pass Test ==
*; Scope:
** Images built as candidates for milestone or final release;
** Passed [[#Weekly Test]]
*; Objective:
** Ensure functionality of Toaster component.
== Release Test ==
*; Scope:
** Release candidates that pass [[#Full Pass Test]]


==== perf 2 ====
*; Objective:
{|class="wikitable" style="text-align: center;"
** All scheduled features are covered, or rescheduled;
| || || colspan="2" | Test execution cycle
** All relevant bugs are fixed and verified.
|-
| || || Weekly Test || Full Pass Test
|-
| Build type || Weekly  || Yes || Yes
|-
| rowspan="2" | Build type || Backend || Yes || Yes
|-
| Frontend  || Yes || Yes

Revision as of 15:47, 16 May 2014

This article is the test plan for Toaster.

Objectives

The test process focuses on:

  • validating the data collected from the build process;
  • verifying the correct functioning of the Toaster GUI; TBD

Test Areas

Toaster consists of two big components, as follows:

Backend

Functionality tests

  • REST API verification – create Django tests to detect API calls returning incomplete (fields having null values) or wrong data for a certain set of entries from each table - our goal is to have 90% data collected. The tests can be found here;
  • Calculation of the data collection rate - the ratio between the number of the variables having null values ​​and the total number of variables collected (the data collection rate does not include wrong values); Run: ./fail_rate.py path_to_toaster.sqlite_file. Output: table_name field_name value_that_never_changes and for each table a percentage: (the number of occurences of all the fields that never change their values)/(total number of entries for that table);
  • Verify that all links in the simple UI are available;
  • Verify the quality of the data collected through the simple UI;
  • Verify the quality of the data collected by verifying most of the values from the database by connecting directly to the database - our immediate goal is to have 90% correct data collected;

Usability tests

Frontend

Functionality tests

  • Manual testing in the first stage;
  • Automate testing using Selenium, in the second stage;

Compatibility tests

  • Verify the behavior of the GUI on different browsers and operating systems; TBD

Usability tests

Performance tests

  • Stress testing (e.g. display appropriate error messages when the system is under stress);

Test Cycle

Test execution cycle
#Weekly Test #Full Pass Test #Release Test
Build type Weekly Yes Yes
Release Yes Yes Yes
#Test Areas #Backend Yes Yes Yes
#Frontend Yes Yes Yes
Target machine qemuarm Yes Yes
qemumips Yes
qemuppc Yes
qemux86 Yes Yes Yes
qemux86-64 Yes
Target image core-image-minimal Yes
core-image-sato-sdk Yes Yes

Weekly Test

  • Scope
    • Images built weekly and released through the distribution team.
  • Objective
    • Functionality test on most areas with minimum sets of tests;
    • Regression test with high probability to find bugs.

Full Pass Test

  • Scope
    • Images built as candidates for milestone or final release;
    • Passed #Weekly Test
  • Objective
    • Ensure functionality of Toaster component.

Release Test

  • Objective
    • All scheduled features are covered, or rescheduled;
    • All relevant bugs are fixed and verified.