Yocto 1.7 Overall Test Plan - Revision history

Alexandru.c.georgescu: /* Test Cycle */

2014-04-29T15:10:08Z

Test Cycle

← Older revision		Revision as of 15:10, 29 April 2014
Line 293:		Line 293:
	\| \|\| yes \|\| yes \|\|		\| \|\| yes \|\| yes \|\|
	\|-		\|-
	! ~~beagleboard~~		! beaglebone black
	\| \|\| yes \|\| yes \|\| yes		\| \|\| yes \|\| yes \|\| yes
	\|-		\|-
Line 299:		Line 299:
	\| \|\| yes \|\| yes \|\|		\| \|\| yes \|\| yes \|\|
	\|-		\|-
	! ~~routerstationpro~~		! edgerouter
	\| \|\| yes \|\| yes \|\| yes		\| \|\| yes \|\| yes \|\| yes
	\|-		\|-

Alexandru.c.georgescu: /* Build System */

2014-04-28T10:50:37Z

Build System

← Older revision		Revision as of 10:50, 28 April 2014
Line 46:		Line 46:
	; Tool: http://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/scripts/contrib/build-perf-test.sh		; Tool: http://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/scripts/contrib/build-perf-test.sh
	; Results page: [[Performance Test]]		; Results page: [[Performance Test]]
			Additionally, build performance will be tracked with/without security flags enabled. The first step is to have several build combinations of core-image-minimal, core-image-sato and core-image-sato-sdk and see the build time differences between the builds.
	=== Distro Testing ===		=== Distro Testing ===
	Distro Testing is intended to catch bugs that are distribution specific using the yocto-autobuilder. The tests are all run on identical hardware and with all OS-es updated. The distributions used will be Fedora, Ubuntu, CentOS, OpenSuse with their latest update. The buildsets that are running on Distro testing are:		Distro Testing is intended to catch bugs that are distribution specific using the yocto-autobuilder. The tests are all run on identical hardware and with all OS-es updated. The distributions used will be Fedora, Ubuntu, CentOS, OpenSuse with their latest update. The buildsets that are running on Distro testing are:
Line 57:		Line 57:
	* nightly x86-64-lsb		* nightly x86-64-lsb
	* nightly-world		* nightly-world
	Additionally, build performance will be tracked with/without security flags enabled. The first step is to have several build combinations of core-image-minimal, core-image-sato and core-image-sato-sdk and see the build time differences between the builds.

	== Target System ==		== Target System ==

Alexandru.c.georgescu: /* Test Automation */

2014-04-28T10:49:49Z

Test Automation

← Older revision		Revision as of 10:49, 28 April 2014
Line 371:		Line 371:

	*; Tools:		*; Tools:
	** [[Distro Testing]]		** [[#Distro Testing]]
	** [[AutoBuilder]]		** [[AutoBuilder]]
	** On [[AutoBuilder]] the [[Image tests]] are run.		** On [[AutoBuilder]] the [[Image tests]] are run.

Alexandru.c.georgescu: /* Test Automation */

2014-04-28T10:49:32Z

Test Automation

← Older revision		Revision as of 10:49, 28 April 2014
Line 371:		Line 371:

	*; Tools:		*; Tools:
			** [[Distro Testing]]
	** [[AutoBuilder]]		** [[AutoBuilder]]
	** On [[AutoBuilder]] the [[Image tests]] are run.		** On [[AutoBuilder]] the [[Image tests]] are run.

Alexandru.c.georgescu: /* Test Automation */

2014-04-28T10:47:53Z

Test Automation

← Older revision		Revision as of 10:47, 28 April 2014
Line 373:		Line 373:
	** [[AutoBuilder]]		** [[AutoBuilder]]
	** On [[AutoBuilder]] the [[Image tests]] are run.		** On [[AutoBuilder]] the [[Image tests]] are run.
	** [[Oe-selftest]]		** [[Oe-selftest]] can be used. Currently oe-selftest is run on public [[AutoBuilder]] as well.
	** HW automation [https://bugzilla.yoctoproject.org/show_bug.cgi?id=1596 1596]		** HW automation [https://bugzilla.yoctoproject.org/show_bug.cgi?id=1596 1596]

Alexandru.c.georgescu: /* Test Report */

2014-04-28T10:10:30Z

Test Report

← Older revision		Revision as of 10:10, 28 April 2014
Line 408:		Line 408:
	= Test Report =		= Test Report =
	*; Objectives:		*; Objectives:
	** Show a live [[1.6 QA Status]] of the active test runs, on the latest build;		** Show a live [[1.7 QA Status]] of the active test runs, on the latest build;
	** Send out an report email to the {{ns:4}} [https://lists.yoctoproject.org/listinfo/yocto mailing list] at the end of a test cycle;		** Send out an report email to the {{ns:4}} [https://lists.yoctoproject.org/listinfo/yocto mailing list] at the end of a test cycle;
	** Archive reports at [[1.6 qa run history]];		** Archive reports at [[1.6 qa run history]];
	** Use [[QA Status TEMPLATE]] for reporting;		** Use [[QA Status TEMPLATE]] for reporting;
	** Use [[Testopia]] as a tool for reporting;		** Use [[Testopia]] as a tool for reporting;

Alexandru.c.georgescu: /* New features validation and QA integration */

2014-04-28T10:09:36Z

New features validation and QA integration

← Older revision		Revision as of 10:09, 28 April 2014
Line 405:		Line 405:
	** performance using gcc security flags		** performance using gcc security flags
	** Shuku proposal		** Shuku proposal
	** HW automation

	= Test Report =		= Test Report =

Alexandru.c.georgescu: /* Validation */

2014-04-28T09:29:55Z

Validation

← Older revision		Revision as of 09:29, 28 April 2014
Line 378:		Line 378:
	= Validation =		= Validation =
	*; Objective:		*; Objective:
	** Verify the correct functionality of [https://wiki.yoctoproject.org/wiki/Yocto_1.~~6_Features~~ new] changes introduced in version 1.6 of the {{ns:4}}.		** Verify the correct functionality of [https://wiki.yoctoproject.org/wiki/Yocto_1.7_Features new] changes introduced in version 1.7 of the {{ns:4}}.

	*; Entry criteria:		*; Entry criteria:
	** The change is tracked ~~and~~ prioritized in [https://bugzilla.yoctoproject.org/ Bugzilla];		** The change is tracked by filling in the "QA Owner" field for the Medium+/High enhancements
	** Bugzilla entry has a target milestone within version 1.6;		** The change is prioritized in [https://bugzilla.yoctoproject.org/ Bugzilla]
	** The change is documented or pointed out when no documentation is necessary (the doc flag is set accordingly);		** Bugzilla entry has a target milestone within version 1.7
			** The change is documented or pointed out when no documentation is necessary (the doc flag is set accordingly)
	** Bug status is set to RESOLVED.		** Bug status is set to RESOLVED.

	*; Exit criteria:		*; Exit criteria:
	** The change is well documented for writing test case, where applicable;		** The change is well documented for writing test case, where applicable
	** Planned test case has passed;		** Planned test case has passed
	** Bug status is set to VERIFIED.		** Bug status is set to VERIFIED

	*; References:		*; References:
	** [[Yocto 1.6 Features]]		** [[Yocto 1.7 Features]]
	** [[1.6 qa owned bugs]] - this wiki was created for QA to track easily resolved bugs and features		** [[1.7 qa owned bugs]] - this wiki was created for QA to track easily resolved bugs and features
	** [[Yocto Project v1.6 Status]]		** [[Yocto Project v1.7 Status]]

	== New features validation and QA integration ==		== New features validation and QA integration ==
	*; Objective:		*; Objective:
	** QA will follow the [[#Validation]] process on verifying the new [[Yocto 1.6 Features]] and integrating them in the testing process.		** QA will follow the [[#Validation]] process on verifying the new [[Yocto 1.7 Features]] and integrating them in the testing process.

	*; Some of the areas that need to be tested:		*; Some of the areas that need to be tested:
	** uClibc integration		** uClibc integration
	** performance using gcc security flags		** performance using gcc security flags
	** New Kernel functionalities listed in the [[Kernel/1.6 Planning]]
	** Shuku proposal		** Shuku proposal
	** SystemD complete integration
	** HW automation		** HW automation

Alexandru.c.georgescu: /* Compliance Testing */

2014-04-28T09:17:25Z

Compliance Testing

← Older revision		Revision as of 09:17, 28 April 2014
Line 121:		Line 121:

	=== Compliance Testing ===		=== Compliance Testing ===
	Compliance test suites / frameworks used:		Compliance test suites / frameworks used on genericx86-64:
	* LSB tests		* LSB tests
	* POSIX tests		* POSIX tests

Alexandru.c.georgescu: /* Stress Testing */

2014-04-28T09:16:23Z

Stress Testing

← Older revision		Revision as of 09:16, 28 April 2014
Line 128:		Line 128:
	=== Stress Testing ===		=== Stress Testing ===

	Stress tests are run on Beagleboard and ~~Jasperforest~~ BSPs. Details as follow:		Stress tests are run on Beagleboard and genericx86-64 BSPs. Details as follow:
	* Beaglebone core-image-sato-sdk image is tested using LTP and Crashme stress tests		* Beaglebone core-image-sato-sdk image is tested using LTP and Crashme stress tests
	* genericx86-64 core-image-lsb-sdk image is tested using Helltest and Crashme stress tests		* genericx86-64 core-image-lsb-sdk image is tested using Helltest and Crashme stress tests