Toaster testing plan

Introduction

This article is the test plan for Toaster.

Objectives

The test process focuses on:

• validating the data collected from the build process
• verifying the correct functioning of the Toaster GUI.

Test Areas

Toaster consists of two big components, as follows:

Backend

Functionality tests

• REST API verification – create Django tests to detect API calls returning no data;
• Calculation of the data collection rate - the ratio of the number of the variables having null values ​​and the total number of variables collected;
• Verify that all links in the simple UI are available;
• Verify the quality of the data collected through the simple UI;

Usability tests

• Verify the easy usage of Toaster (easy to install and stop/start the toaster server)

Frontend

Functionality tests

• Manual testing in the first stage;
• Automate testing using Selenium, in the second stage;

Compatibility tests

• Verify the behavior of the GUI on different browsers and operating systems;

Usability tests

• Verify if the GUI design is as described here: http://yoctoproject.org/webhob;
• Friendly graphical user interface;

Performance tests

• Stress testing (e.g. display appropriate error messages when the system is under stress);

Test Cycle

Weekly Test

• Scope

  • Images built weekly and released through the distribution team.
  • Passed #Sanity Test

• Objective

  • Functionality test on most areas with minimum sets of tests;
  • Regression test with high probability to find bugs.

Full Pass Test

• Scope

  • Images built as candidates for milestone or final release;
  • Passed #Weekly Test

• Objective

  • Ensure functionality of all Yocto Project components.

		Test execution cycle
		#Weekly Test	#Full Pass Test
Build type	Weekly	Yes	Yes
Build type	#Backend	Yes	Yes
	#Frontend	Yes	Yes
Target machine	qemuarm	Yes	Yes
	qemumips		Yes
	qemuppc		Yes
	qemux86	Yes	Yes
Target image	core-image-minimal	Yes	Yes
	core-image-sato		Yes