Toaster testing plan

This article is the test plan for Toaster.

Objectives

Verify all Toaster components to be fully functional.

Components to be verified:

• UI interface
• Backend interaction with bitbake for build purposes
• Backend interaction with database for storing and accessing build information

Test Areas

Toaster consists of two big components, as follows:

Backend

Functionality tests

• REST API verification – create Django tests to detect API calls returning incomplete (fields having null values) or wrong data for a certain set of entries from each table - our goal is to have 90% data collected. The tests can be found here;
• Calculation of the data collection rate - the ratio between the number of the variables having null values ​​and the total number of variables collected (the data collection rate does not include wrong values); Run: ./fail_rate.py path_to_toaster.sqlite_file. Output: table_name field_name value_that_never_changes and for each table a percentage: (the number of occurences of all the fields that never change their values)/(total number of entries for that table);
• Verify that all links in the simple UI are available;
• Verify the quality of the data collected through the simple UI;
• Verify the quality of the data collected by verifying most of the values from the database by connecting directly to the database - our immediate goal is to have 90% correct data collected;

Usability tests

• Verify the easy usage of Toaster (easy to install and start/stop the toaster server)

Frontend

Functionality tests

• Manual testing in the first stage;
• Automate testing using Selenium, in the second stage;

Compatibility tests

• Verify the behavior of the GUI on different browsers and operating systems; TBD

Usability tests

• Verify if the GUI design is as described here: http://yoctoproject.org/webhob;
• Friendly graphical user interface;

Performance tests

• Stress testing (e.g. display appropriate error messages when the system is under stress);

Test Cycle

		Test execution cycle
		#Weekly Test	#Full Pass Test	#Release Test
Build type	Weekly	Yes	Yes
	Release	Yes	Yes	Yes
#Test Areas	#Backend	Yes	Yes	Yes
	#Frontend	Yes	Yes	Yes
Target machine	qemuarm		Yes	Yes
	qemumips			Yes
	qemuppc			Yes
	qemux86	Yes	Yes	Yes
	qemux86-64			Yes
Target image	core-image-minimal	Yes
	core-image-sato-sdk		Yes	Yes

Weekly Test

• Scope

  • Images built weekly and released through the distribution team.

• Objective

  • Functionality test on most areas with minimum sets of tests;
  • Regression test with high probability to find bugs.

Full Pass Test

• Scope

  • Images built as candidates for milestone or final release;
  • Passed #Weekly Test

• Objective

  • Ensure functionality of Toaster component.

Release Test

• Scope

  • Release candidates that pass #Full Pass Test

• Objective

  • All scheduled features are covered, or rescheduled;
  • All relevant bugs are fixed and verified.