Toaster testing plan - Revision history

Jose G Perez Carranza at 13:28, 19 May 2016

2016-05-19T13:28:19Z

Jose G Perez Carranza at 13:27, 19 May 2016

2016-05-19T13:27:33Z

Alexandru.c.georgescu at 14:17, 19 April 2016

2016-04-19T14:17:58Z

← Older revision		Revision as of 14:17, 19 April 2016
Line 401:		Line 401:


			;Sanity Test is described [https://wiki.yoctoproject.org/wiki/Yocto_2.1_Overall_Test_Plan#Sanity_Testing here].
	*run on toaster-next or poky-contrib.		*run on toaster-next or poky-contrib.
	**TBD		**TBD

Alexandru.c.georgescu at 14:15, 19 April 2016

2016-04-19T14:15:24Z

← Older revision		Revision as of 14:15, 19 April 2016
Line 401:		Line 401:


	~~;Sanity Test is described [https://docs.google.com/document/d/1RN6t-_HfaVWKfcMD4DCl8RYwX_yfsnPcugXVMYz02GE/edit#heading=h.pcitht2lsb0q here].~~
	*run on toaster-next or poky-contrib.		*run on toaster-next or poky-contrib.
	**TBD		**TBD

Mihail Stanciu: Undo revision 17769 by Mihail Stanciu (talk)

2016-03-04T12:34:38Z

Undo revision 17769 by Mihail Stanciu (talk)

← Older revision		Revision as of 12:34, 4 March 2016
Line 219:		Line 219:

	As currently Testopia is structured, all Toaster test cases are stored under Classifications/Build System & Metadata/ Toaster component within Testopia. All the test cases are organized in Test Run Templates as following:		As currently Testopia is structured, all Toaster test cases are stored under Classifications/Build System & Metadata/ Toaster component within Testopia. All the test cases are organized in Test Run Templates as following:
			*Toaster Backend - Weekly: https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4028
			*Toaster Backend - Managed Mode: https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4030
	*Toaster UI: https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4027		*Toaster UI: https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4027
	*Toaster UI - Managed Mode: https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4029		*Toaster UI - Managed Mode: https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4029
			*Toaster Exploratory - https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4043

	== Submitting Bugs ==		== Submitting Bugs ==

Mihail Stanciu: Undo revision 17770 by Mihail Stanciu (talk)

2016-03-04T12:34:20Z

Undo revision 17770 by Mihail Stanciu (talk)

← Older revision		Revision as of 12:34, 4 March 2016
Line 77:		Line 77:
	**currently working on their automation using Selenium		**currently working on their automation using Selenium


			=== BackEnd functionalities ===
			=== LEGACY === Starting with 2.1 M3 the dev team will be taking responsibility for these tests. They will become a part of their internal CI testing.
			Covers data integrity, data transfer between Bitbake and Toaster, Start/Stop, building images, configuration, etc

			Testing needs:
			*Built in python 2.7
			*Direct sql requests to local sqlite instance
			YP2.1 TODO: Direct SQL tests are going to be hard to maintain and could be done more effectively by either using unit tests in django or by accessing the django ORM. This would also make the test database database backend agnostic.
			<!--
			Brian's Comments
			1) I would like to remove the manual element from all the toaster tests.
			2) unless we are using something from within bitbake that I didn't see these tests do seem to be more amenable to django unit tests since they are already using the orm.models.*
			3)we should avoid presuming that we are running from inside poky. referencing ../bitbake/lib/toaster is a bad idea.
			-->

			*Requires sqlite3 client installed (for manual debugging/investigation)
			*BackEnd automated tests are verifying the correctness of the data gathered by the build process as presented in the Backend template: https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4028
			**In order for these tests to run properly, check the the steps for “The oe-selftest toaster tests” from https://wiki.yoctoproject.org/wiki/Testing_Toaster
			*For BackEnd managed mode manual testing follow the steps from the following template https://bugzilla.yoctoproject.org/tr_show_run.cgi?run_id=4030
			*please validate and add info from Testing Toaster wiki in this section(Test Strategy): https://wiki.yoctoproject.org/wiki/Testing_Toaster

	== Test automation ==		== Test automation ==

Mihail Stanciu: Undo revision 17771 by Mihail Stanciu (talk)

2016-03-04T12:34:07Z

Undo revision 17771 by Mihail Stanciu (talk)

← Older revision		Revision as of 12:34, 4 March 2016
Line 102:		Line 102:

	-->		-->

			=== The oe-selftest toaster tests ===
			=== LEGACY === These are no longer valid as of 2.1 M3

			These tests are verifying the correctness of the data gathered by the build process - it tests the bitbake/lib/bb/ui/toasterui.py and bitbake/lib/bb/ui/buildinfohelper.py files and the interface to the bitbake server process.

			In order for these tests to run properly, the following steps need to be taken before running, on a clean toaster setup:

			*Create new project using local poky release
			*Create 1 core-image-minimal image
			*Create 1 core-image-sato image to check building from cache
			*Break a recipe file (by modifying a link, for example)
			*Create a new image using broken recipe to check error detection is working


			To run these tests you need to source an environment by issuing:
			source oe-init-build-env

			then modify the bblayers.conf file to add the meta-selftest layer, create a symbolic link to the toaster.sqlite file:
			ln -s ../toaster.sqlite toaster.sqlite

			and then run the command:
			oe-selftest --run-tests _toaster

	=== The selenium automated UI tests ===		=== The selenium automated UI tests ===

Mihail Stanciu: Undo revision 17772 by Mihail Stanciu (talk)

2016-03-04T12:33:53Z

Undo revision 17772 by Mihail Stanciu (talk)

← Older revision		Revision as of 12:33, 4 March 2016
Line 165:		Line 165:
	===Regression===		===Regression===
	*All Full pass tests for		*All Full pass tests for
			**BackEnd
	**FrontEnd		**FrontEnd

Mihail Stanciu: Undo revision 17773 by Mihail Stanciu (talk)

2016-03-04T12:33:39Z

Undo revision 17773 by Mihail Stanciu (talk)

@@ Line 218: / Line 218: @@
 =Features=
 ==Features to be tested==
 ===Frontend===
 *Verifying if all view elements work (buttons, tables, sort buttons, etc.)

Mihail Stanciu: Undo revision 17774 by Mihail Stanciu (talk)

2016-03-04T12:33:24Z

Undo revision 17774 by Mihail Stanciu (talk)

← Older revision		Revision as of 12:33, 4 March 2016
Line 21:		Line 21:
	*this can sometimes generate new test cases.		*this can sometimes generate new test cases.
	Verifying the bugs and features.		Verifying the bugs and features.

			=LEGACY= Starting with 2.1 M3, the following will no longer be tested by the QA team. The dev team will be taking over responsibility for these components:
			* Backend interaction with bitbake for build purposes
			* Backend interaction with database for storing and accessing build information

	= Team members =		= Team members =