BSP Test Process - Revision history

Yeoh, Ee Peng at 05:55, 12 March 2019

2019-03-12T05:55:40Z

← Older revision		Revision as of 05:55, 12 March 2019
Line 57:		Line 57:
	3. test case id - unique id inside testresults.json file used to identify a specific test case executed		3. test case id - unique id inside testresults.json file used to identify a specific test case executed
	(eg. bsp-hw.bsps-oe-core.Test_Run_Integrity_-_Check_that_image_is_buildable)		(eg. bsp-hw.bsps-oe-core.Test_Run_Integrity_-_Check_that_image_is_buildable)


	== Test Result Store ==		== Test Result Store ==

Yeoh, Ee Peng: /* Defect management */

2019-03-12T05:54:24Z

Defect management

← Older revision		Revision as of 05:54, 12 March 2019
Line 53:		Line 53:
	Information needed to trace bug to testcase inside yocto-testresults:		Information needed to trace bug to testcase inside yocto-testresults:
	1. git tag - tag used to store the failed test case (eg. "{branch}/{commit_number}-g{commit}/{tag_number}")		1. git tag - tag used to store the failed test case (eg. "{branch}/{commit_number}-g{commit}/{tag_number}")
	2. result id - unique id inside testresults.json file that used to identify a test result set consist of a list of test cases executed (eg. manual_bsps-hw_20190311160844)		2. result id - unique id inside testresults.json file that used to identify a test result set consist of a list of test cases executed
	3. test case id - unique id inside testresults.json file used to identify a specific test case executed (eg. bsp-hw.bsps-oe-core.Test_Run_Integrity_-_Check_that_image_is_buildable)		(eg. manual_bsps-hw_20190311160844)
			3. test case id - unique id inside testresults.json file used to identify a specific test case executed
			(eg. bsp-hw.bsps-oe-core.Test_Run_Integrity_-_Check_that_image_is_buildable)

	== Test Result Store ==		== Test Result Store ==

Yeoh, Ee Peng: /* Defect management */

2019-03-12T05:54:02Z

Defect management

← Older revision		Revision as of 05:54, 12 March 2019
Line 52:		Line 52:

	Information needed to trace bug to testcase inside yocto-testresults:		Information needed to trace bug to testcase inside yocto-testresults:
	1. git tag - tag used to store the failed test case (eg. "{branch}/{commit_number}-g{commit}/{tag_number}")		1. git tag - tag used to store the failed test case (eg. "{branch}/{commit_number}-g{commit}/{tag_number}")
	2. result id - unique id inside testresults.json file that used to identify a test result set consist of a list of test cases executed (eg. manual_bsps-hw_20190311160844)		2. result id - unique id inside testresults.json file that used to identify a test result set consist of a list of test cases executed (eg. manual_bsps-hw_20190311160844)
	3. test case id - unique id inside testresults.json file used to identify a specific test case executed (eg. bsp-hw.bsps-oe-core.Test_Run_Integrity_-_Check_that_image_is_buildable)		3. test case id - unique id inside testresults.json file used to identify a specific test case executed (eg. bsp-hw.bsps-oe-core.Test_Run_Integrity_-_Check_that_image_is_buildable)

	== Test Result Store ==		== Test Result Store ==

Yeoh, Ee Peng: /* ROLE: Test Executor */

2019-03-12T05:53:28Z

ROLE: Test Executor

← Older revision		Revision as of 05:53, 12 March 2019
Line 47:		Line 47:
	For runtime tests inside oeqa/manual, use "resulttool manualexecution" to execute these tests (refer to https://wiki.yoctoproject.org/wiki/Resulttool#manualexecution for more detail), collect the "testresults.json" file from the <build-dir>/tmp/log/manual/ directory.		For runtime tests inside oeqa/manual, use "resulttool manualexecution" to execute these tests (refer to https://wiki.yoctoproject.org/wiki/Resulttool#manualexecution for more detail), collect the "testresults.json" file from the <build-dir>/tmp/log/manual/ directory.

	Defect management:		=== Defect management ===
	Once a test case failed during execution, create a bug ticket at Bugzilla.		Once a test case failed during execution, create a bug ticket at Bugzilla.
	To map the bug ticket to the failed test case executed, provide below information inside bug ticket.		To map the bug ticket to the failed test case executed, provide below information inside bug ticket.
Line 55:		Line 55:
	2. result id - unique id inside testresults.json file that used to identify a test result set consist of a list of test cases executed (eg. manual_bsps-hw_20190311160844)		2. result id - unique id inside testresults.json file that used to identify a test result set consist of a list of test cases executed (eg. manual_bsps-hw_20190311160844)
	3. test case id - unique id inside testresults.json file used to identify a specific test case executed (eg. bsp-hw.bsps-oe-core.Test_Run_Integrity_-_Check_that_image_is_buildable)		3. test case id - unique id inside testresults.json file used to identify a specific test case executed (eg. bsp-hw.bsps-oe-core.Test_Run_Integrity_-_Check_that_image_is_buildable)


	== Test Result Store ==		== Test Result Store ==

Yeoh, Ee Peng at 05:53, 12 March 2019

2019-03-12T05:53:00Z

← Older revision		Revision as of 05:53, 12 March 2019
Line 30:		Line 30:
	=== ROLE: Test Executor ===		=== ROLE: Test Executor ===
	Intel and WindRiver test team request subscribe to QA notification by contacting Linux Foundation (Michael Halstead mhalstead@linuxfoundation.org), once QA notification received then each test team will start preparing for BPS testing.		Intel and WindRiver test team request subscribe to QA notification by contacting Linux Foundation (Michael Halstead mhalstead@linuxfoundation.org), once QA notification received then each test team will start preparing for BPS testing.


	== Test Setup==		== Test Setup==
Line 54:		Line 55:
	2. result id - unique id inside testresults.json file that used to identify a test result set consist of a list of test cases executed (eg. manual_bsps-hw_20190311160844)		2. result id - unique id inside testresults.json file that used to identify a test result set consist of a list of test cases executed (eg. manual_bsps-hw_20190311160844)
	3. test case id - unique id inside testresults.json file used to identify a specific test case executed (eg. bsp-hw.bsps-oe-core.Test_Run_Integrity_-_Check_that_image_is_buildable)		3. test case id - unique id inside testresults.json file used to identify a specific test case executed (eg. bsp-hw.bsps-oe-core.Test_Run_Integrity_-_Check_that_image_is_buildable)


	== Test Result Store ==		== Test Result Store ==
Line 70:		Line 72:

	Intel and WindRiver team will notifiy the Overall QA Owner on completion of BSP testing.		Intel and WindRiver team will notifiy the Overall QA Owner on completion of BSP testing.


	== Test Reporting and Regression==		== Test Reporting and Regression==
Line 75:		Line 78:
	=== ROLE: Overall QA Owner ===		=== ROLE: Overall QA Owner ===
	Overall QA Owner team will re-run “resulttool report” & “resulttool regression-git” to create report and regression for both automated and BSP test result (refer to https://wiki.yoctoproject.org/wiki/Resulttool#report and https://wiki.yoctoproject.org/wiki/Resulttool#regression-git for more detail).		Overall QA Owner team will re-run “resulttool report” & “resulttool regression-git” to create report and regression for both automated and BSP test result (refer to https://wiki.yoctoproject.org/wiki/Resulttool#report and https://wiki.yoctoproject.org/wiki/Resulttool#regression-git for more detail).


	== Decision ==		== Decision ==

Yeoh, Ee Peng: /* ROLE: Test Executor */

2019-03-12T05:52:24Z

ROLE: Test Executor

← Older revision		Revision as of 05:52, 12 March 2019
Line 46:		Line 46:
	For runtime tests inside oeqa/manual, use "resulttool manualexecution" to execute these tests (refer to https://wiki.yoctoproject.org/wiki/Resulttool#manualexecution for more detail), collect the "testresults.json" file from the <build-dir>/tmp/log/manual/ directory.		For runtime tests inside oeqa/manual, use "resulttool manualexecution" to execute these tests (refer to https://wiki.yoctoproject.org/wiki/Resulttool#manualexecution for more detail), collect the "testresults.json" file from the <build-dir>/tmp/log/manual/ directory.

			Defect management:
			Once a test case failed during execution, create a bug ticket at Bugzilla.
			To map the bug ticket to the failed test case executed, provide below information inside bug ticket.

			Information needed to trace bug to testcase inside yocto-testresults:
			1. git tag - tag used to store the failed test case (eg. "{branch}/{commit_number}-g{commit}/{tag_number}")
			2. result id - unique id inside testresults.json file that used to identify a test result set consist of a list of test cases executed (eg. manual_bsps-hw_20190311160844)
			3. test case id - unique id inside testresults.json file used to identify a specific test case executed (eg. bsp-hw.bsps-oe-core.Test_Run_Integrity_-_Check_that_image_is_buildable)

	== Test Result Store ==		== Test Result Store ==

Yeoh, Ee Peng: /* ROLE: Overall QA Owner */

2019-03-12T02:32:14Z

ROLE: Overall QA Owner

← Older revision		Revision as of 02:32, 12 March 2019
Line 66:		Line 66:

	=== ROLE: Overall QA Owner ===		=== ROLE: Overall QA Owner ===
	Overall QA Owner team ~~(Linux foundation)~~ will re-run “resulttool report” & “resulttool regression-git” to create report and regression for both automated and BSP test result (refer to https://wiki.yoctoproject.org/wiki/Resulttool#report and https://wiki.yoctoproject.org/wiki/Resulttool#regression-git for more detail).		Overall QA Owner team will re-run “resulttool report” & “resulttool regression-git” to create report and regression for both automated and BSP test result (refer to https://wiki.yoctoproject.org/wiki/Resulttool#report and https://wiki.yoctoproject.org/wiki/Resulttool#regression-git for more detail).


	== Decision ==		== Decision ==

Yeoh, Ee Peng: /* ROLE: Test Executor */

2019-03-12T02:31:38Z

ROLE: Test Executor

← Older revision		Revision as of 02:31, 12 March 2019
Line 51:		Line 51:
	=== ROLE: Test Executor ===		=== ROLE: Test Executor ===
	Intel and WindRiver team will request write access to https://git.yoctoproject.org/git/yocto-testresults remote repository by contacting Linux Foundation (Michael Halstead mhalstead@linuxfoundation.org).		Intel and WindRiver team will request write access to https://git.yoctoproject.org/git/yocto-testresults remote repository by contacting Linux Foundation (Michael Halstead mhalstead@linuxfoundation.org).

	Intel and WindRiver test team will store the testresults.json files collected in previous section to the yocto-testresults git repository (http://git.yoctoproject.org/cgit/cgit.cgi/yocto-testresults/).		Intel and WindRiver test team will store the testresults.json files collected in previous section to the yocto-testresults git repository (http://git.yoctoproject.org/cgit/cgit.cgi/yocto-testresults/).

Line 60:		Line 61:
	Make sure the git tag created (branch and commit) was correct and the correct testresults.json file was stored inside the local yocto-testresults git repository before push to remote.		Make sure the git tag created (branch and commit) was correct and the correct testresults.json file was stored inside the local yocto-testresults git repository before push to remote.

			Intel and WindRiver team will notifiy the Overall QA Owner on completion of BSP testing.

	== Test Reporting and Regression==		== Test Reporting and Regression==

Yeoh, Ee Peng: /* ROLE: Test Executor */

2019-03-12T02:05:32Z

ROLE: Test Executor

← Older revision		Revision as of 02:05, 12 March 2019
Line 29:		Line 29:

	=== ROLE: Test Executor ===		=== ROLE: Test Executor ===
	Intel and WindRiver test team ~~subcribe~~ to QA notification by ~~contact~~ Linux Foundation. ~~will monitor yocto mailing list for "QA notification for completed autobuilder build (yocto-<releasa>~~)", once QA notification received then each test team will start preparing for BPS testing.		Intel and WindRiver test team request subscribe to QA notification by contacting Linux Foundation (Michael Halstead mhalstead@linuxfoundation.org), once QA notification received then each test team will start preparing for BPS testing.

	== Test Setup==		== Test Setup==

Yeoh, Ee Peng: /* ROLE: Test Executor */

2019-03-12T02:04:21Z

ROLE: Test Executor

← Older revision		Revision as of 02:04, 12 March 2019
Line 29:		Line 29:

	=== ROLE: Test Executor ===		=== ROLE: Test Executor ===
	Intel and WindRiver test team will monitor yocto mailing list for "QA notification for completed autobuilder build (yocto-<releasa>)", once QA notification received then each test team will start preparing for BPS testing.		Intel and WindRiver test team subcribe to QA notification by contact Linux Foundation. will monitor yocto mailing list for "QA notification for completed autobuilder build (yocto-<releasa>)", once QA notification received then each test team will start preparing for BPS testing.


	== Test Setup==		== Test Setup==