Yocto 1.1 Overall Test Plan - Revision history

Jiajun: /* Core Build System(Covered in Fullpass Test) */

2011-07-07T14:13:00Z

Core Build System(Covered in Fullpass Test)

← Older revision		Revision as of 14:13, 7 July 2011
Line 233:		Line 233:
	** ... etc		** ... etc
	* Distribution support Test: Test Core buld on latest Ubuntu, Fedora and Opensuse Hosts		* Distribution support Test: Test Core buld on latest Ubuntu, Fedora and Opensuse Hosts
			* Hob Test: Main UI and component functionality are validated (Some cases defined in weekly and some in fulpass)
	* Performance Test: Check Core build performance on standard host(Defined in perfromance test webpage)		* Performance Test: Check Core build performance on standard host(Defined in perfromance test webpage)

Jiajun: /* Test Strategy and Approach */

2011-07-07T14:08:37Z

Test Strategy and Approach

← Older revision		Revision as of 14:08, 7 July 2011
Line 118:		Line 118:
	\| '''Core build'''\|\|yes\|\|yes\|\|yes		\| '''Core build'''\|\|yes\|\|yes\|\|yes
	\|-		\|-
	\|		\| '''Hob'''\|\|yes\|\|n/a\|\|n/a
	\|}
	~~{\| border="1" {{table}}~~
	~~\| align="center" style="background:#f0f0f0;"~~\|'''Hob'''
	\| ~~align="center" style="background:#f0f0f0;"~~\|~~'''Functionality'''~~
	\|-
	\| ~~'''Documentation'''~~\|\|~~yes~~
	\|-		\|-
	\|		\|

Jiajun: /* Test Strategy and Approach */

2011-07-07T14:07:00Z

Test Strategy and Approach

← Older revision		Revision as of 14:07, 7 July 2011
Line 117:		Line 117:
	\|-		\|-
	\| '''Core build'''\|\|yes\|\|yes\|\|yes		\| '''Core build'''\|\|yes\|\|yes\|\|yes
			\|-
			\|
			\|}
			{\| border="1" {{table}}
			\| align="center" style="background:#f0f0f0;"\|'''Hob'''
			\| align="center" style="background:#f0f0f0;"\|'''Functionality'''
			\|-
			\| '''Documentation'''\|\|yes
	\|-		\|-
	\|		\|

Jiajun: /* Features to Be Tested */

2011-07-07T13:48:58Z

Features to Be Tested

← Older revision		Revision as of 13:48, 7 July 2011
Line 18:		Line 18:
	* Yocto ADT: ADT testing only covers toolchain test.		* Yocto ADT: ADT testing only covers toolchain test.
	* Hob: Functionality of all components are validated if work correct.		* Hob: Functionality of all components are validated if work correct.
	* Power/Performance: QA will check the CPU power behavior by software level tool, such as '''powertop'''. Real Power consumption ~~testing will be investigated. Performance results should be~~ collected ~~for analysis and tuning. Memory footprint under specific workload will be recorded~~.		* Power/Performance: QA will check the CPU power behavior by software level tool, such as '''powertop'''. Real Power consumption with idle is collected.

	=== What will not Be Tested in Yocto v1.1 ===		=== What will not Be Tested in Yocto v1.1 ===

Jiajun: /* Features to Be Tested */

2011-07-07T13:47:52Z

Features to Be Tested

← Older revision		Revision as of 13:47, 7 July 2011
Line 17:		Line 17:
	* Core build system: It includes basic core build test and other build feature test. Performance and different distribuitions support will be also tested.		* Core build system: It includes basic core build test and other build feature test. Performance and different distribuitions support will be also tested.
	* Yocto ADT: ADT testing only covers toolchain test.		* Yocto ADT: ADT testing only covers toolchain test.
			* Hob: Functionality of all components are validated if work correct.
	* Power/Performance: QA will check the CPU power behavior by software level tool, such as '''powertop'''. Real Power consumption testing will be investigated. Performance results should be collected for analysis and tuning. Memory footprint under specific workload will be recorded.		* Power/Performance: QA will check the CPU power behavior by software level tool, such as '''powertop'''. Real Power consumption testing will be investigated. Performance results should be collected for analysis and tuning. Memory footprint under specific workload will be recorded.

Jiajun at 16:35, 9 May 2011

2011-05-09T16:35:29Z

← Older revision		Revision as of 16:35, 9 May 2011
Line 5:		Line 5:
	\|-		\|-
	\| 0.3 \|\| Jiajun Xu \|\| Initial Version		\| 0.3 \|\| Jiajun Xu \|\| Initial Version
			\|-
			\| 0.4 \|\| Jiajun Xu \|\| Update with internal review comments from Julie, Saul and Yi Zhao
	\|}		\|}

Jiajun: /* Test Design */

2011-05-09T16:33:55Z

Test Design

← Older revision		Revision as of 16:33, 9 May 2011
Line 220:		Line 220:
	(Basic functionality for tested features will be exercised as well as new functionality added in each release.)		(Basic functionality for tested features will be exercised as well as new functionality added in each release.)

	==== Core Build System ====		==== Core Build System(Covered in Fullpass Test) ====
	* Functionality Test: Test Core build feature on standard host configuration. For example:		* Functionality Test: Test Core build feature on standard host configuration. For example:
	** kernel interactive targets		** kernel interactive targets
Line 230:		Line 230:
	* Performance Test: Check Core build performance on standard host(Defined in perfromance test webpage)		* Performance Test: Check Core build performance on standard host(Defined in perfromance test webpage)

	==== ADT Test ====		==== ADT Test(Covered in Weekly Test) ====
	* Toolchain test, including compiler, libraries, etc.		* Toolchain test, including compiler, libraries, etc.

	==== Stress Test ====		==== Stress Test(Covered in Fullpass Test) ====
	Stress testing refers to tests that determine the robustness of software by testing beyond the limits of normal operation. Stress tests commonly put a greater emphasis on robustness, availability, and error handling under a heavy load, than on what would be considered as correct behavior under normal circumstances.		Stress testing refers to tests that determine the robustness of software by testing beyond the limits of normal operation. Stress tests commonly put a greater emphasis on robustness, availability, and error handling under a heavy load, than on what would be considered as correct behavior under normal circumstances.

	* Test Method: 24 hours workload (Helltest + Crashme) test should pass.		* Test Method: 24 hours workload (Helltest + Crashme) test should pass.

	==== Power/Performance Test ====		==== Power/Performance Test(Covered in Fullpass Test) ====
	Power/Performance Test is to collect the power/performance data and status of Yocto image on defined BSP.		Power/Performance Test is to collect the power/performance data and status of Yocto image on defined BSP.
	* Test Method: Power test will check if Yocto is power saving friendly, that CPU should support and enter different power saving stage. For performance test, memory footprint under specific workload will be collected.		* Test Method: Power test will check if Yocto is power saving friendly, that CPU should support and enter different power saving stage. For performance test, memory footprint under specific workload will be collected.

	==== Compliance Test ====		==== Compliance Test(Covered in Fullpass Test) ====
	Compliance Test is targeted to validate system level health with different system test suite.		Compliance Test is targeted to validate system level health with different system test suite.
	* LTP test suite		* LTP test suite
	* POSIX test		* POSIX test
	* LSB test		* LSB test

Jiajun: /* System & Core OS */

2011-05-09T16:33:07Z

System & Core OS

← Older revision		Revision as of 16:33, 9 May 2011
Line 209:		Line 209:
	This is a overall test design for test components which will be covered QA testing. Detailed test cases design will be covered in Test Execution Plan.		This is a overall test design for test components which will be covered QA testing. Detailed test cases design will be covered in Test Execution Plan.

	==== System & Core OS ====		==== System & Core OS(Covered in Weekly Test) ====
	* Boot Test: Check if Yocto image is bootable with the bootloader.		* Boot Test: Check if Yocto image is bootable with the bootloader.
	* Installation/SW Update Test: Validate the Yocto image installation on hardware platform and software ~~update~~ functions from the end users' perspective. Using zypper/rpm for software update with hardisk installed image.		* Installation/SW Update Test: Validate the Yocto image installation on hardware platform and software install/removal functions from the end users' perspective. Using zypper/rpm for software update with hardisk installed image.
	* Common system usage Test: shutdown/restart, suspend/resume, init 3/5 boot.		* Common system usage Test: shutdown/restart, suspend/resume, init 3/5 boot.
	* Component Test (aka. Feature Test): Based on sato image features to test. For example:		* Component Test (aka. Feature Test): Based on sato image features to test. For example:
Line 218:		Line 218:
	** 3G		** 3G
	** ... etc		** ... etc
			(Basic functionality for tested features will be exercised as well as new functionality added in each release.)

	==== Core Build System ====		==== Core Build System ====

Jiajun: /* Test Strategy and Approach */

2011-05-09T16:26:38Z

Test Strategy and Approach

← Older revision		Revision as of 16:26, 9 May 2011
Line 105:		Line 105:
	Note:		Note:
	* Compliance Test on Beagleboard and routerstation will only cover LTP and POSXI Test, since LSB test suite does not support ARM and MIPS.		* Compliance Test on Beagleboard and routerstation will only cover LTP and POSXI Test, since LSB test suite does not support ARM and MIPS.
	** Jasperforest is picked up for stress test, because it is powerful with 12 CPUs and 2GB memory. Moreover, it is a server, which needs more stress tests. The other test boards, like blacksand and eMenlow do not have powerful CPUs, that's why we do not use them for stress test.		* Jasperforest is picked up for stress test, because it is powerful with 12 CPUs and 2GB memory. Moreover, it is a server, which needs more stress tests. The other test boards, like blacksand and eMenlow do not have powerful CPUs, that's why we do not use them for stress test.

	{\| border="1" {{table}}		{\| border="1" {{table}}

Jiajun: /* Test Strategy and Approach */

2011-05-09T16:26:22Z

Test Strategy and Approach

← Older revision		Revision as of 16:26, 9 May 2011
Line 86:		Line 86:
	\| '''HuronRiver'''\|\|yes\|\|  \|\|  \|\|  \|\|yes\|\|yes\|\|yes		\| '''HuronRiver'''\|\|yes\|\|  \|\|  \|\|  \|\|yes\|\|yes\|\|yes
	\|-		\|-
	\| '''Jasperforest'''\|\|yes\|\|  \|\| \|\|yes\|\|  \|\|  \|\|		\| '''Jasperforest'''\|\|yes\|\|  \|\| \|\|yes**\|\|  \|\|  \|\|
	\|-		\|-
	\| '''Blacksand'''\|\|yes\|\|  \|\| yes \|\|  \|\|  \|\|yes\|\|yes		\| '''Blacksand'''\|\|yes\|\|  \|\| yes \|\|  \|\|  \|\|yes\|\|yes
Line 105:		Line 105:
	Note:		Note:
	* Compliance Test on Beagleboard and routerstation will only cover LTP and POSXI Test, since LSB test suite does not support ARM and MIPS.		* Compliance Test on Beagleboard and routerstation will only cover LTP and POSXI Test, since LSB test suite does not support ARM and MIPS.
			** Jasperforest is picked up for stress test, because it is powerful with 12 CPUs and 2GB memory. Moreover, it is a server, which needs more stress tests. The other test boards, like blacksand and eMenlow do not have powerful CPUs, that's why we do not use them for stress test.

	{\| border="1" {{table}}		{\| border="1" {{table}}