Toaster testing plan
From Yocto Project
Revision as of 08:47, 16 May 2014 by Belenbarrospena
This article is the test plan for Toaster.
The test process focuses on:
- validating the data collected from the build process;
- verifying the correct functioning of the Toaster GUI; TBD
Toaster consists of two big components, as follows:
- REST API verification – create Django tests to detect API calls returning incomplete (fields having null values) or wrong data for a certain set of entries from each table - our goal is to have 90% data collected. The tests can be found here;
- Calculation of the data collection rate - the ratio between the number of the variables having null values and the total number of variables collected (the data collection rate does not include wrong values); Run: ./fail_rate.py path_to_toaster.sqlite_file. Output: table_name field_name value_that_never_changes and for each table a percentage: (the number of occurences of all the fields that never change their values)/(total number of entries for that table);
- Verify that all links in the simple UI are available;
- Verify the quality of the data collected through the simple UI;
- Verify the quality of the data collected by verifying most of the values from the database by connecting directly to the database - our immediate goal is to have 90% correct data collected;
- Verify the easy usage of Toaster (easy to install and start/stop the toaster server)
- Manual testing in the first stage;
- Automate testing using Selenium, in the second stage;
- Verify the behavior of the GUI on different browsers and operating systems; TBD
- Verify if the GUI design is as described here: http://yoctoproject.org/webhob;
- Friendly graphical user interface;
- Stress testing (e.g. display appropriate error messages when the system is under stress);
|Test execution cycle|
|#Weekly Test||#Full Pass Test||#Release Test|
- Images built weekly and released through the distribution team.
- Functionality test on most areas with minimum sets of tests;
- Regression test with high probability to find bugs.
Full Pass Test
- Images built as candidates for milestone or final release;
- Passed #Weekly Test
- Ensure functionality of Toaster component.
- Release candidates that pass #Full Pass Test
- All scheduled features are covered, or rescheduled;
- All relevant bugs are fixed and verified.