QA/Test Report Tool - Revision history

Benjamin: Benjamin moved page QA/Test Reporting Tool to QA/Test Report Tool: Name change to simplify its search

2016-11-10T19:47:36Z

Benjamin moved page QA/Test Reporting Tool to QA/Test Report Tool: Name change to simplify its search

← Older revision	Revision as of 19:47, 10 November 2016
(No difference)

Benjamin at 17:08, 15 July 2016

2016-07-15T17:08:03Z

← Older revision		Revision as of 17:08, 15 July 2016
Line 1:		Line 1:
	==General Expectations==		==General Expectations==

			(add link share-ability of results to the list of requirements when it fits somewhere)

	Here's a list of initial expectations of the Test Reporting Tool.		Here's a list of initial expectations of the Test Reporting Tool.

Benjamin at 16:40, 15 July 2016

2016-07-15T16:40:35Z

← Older revision		Revision as of 16:40, 15 July 2016
Line 5:		Line 5:
	# The TRT must be able to receive test reports via a remote communication protocol.		# The TRT must be able to receive test reports via a remote communication protocol.
	## From expected sources of results (Auth)		## From expected sources of results (Auth)
	## To existing test buckets (logical test belonging separation, could be by test component)		## To existing test buckets (logical test belonging separation, could be by test component) through report sessions with:
			##* ability to resume
			##* ability to update existing results
			##* ability to abort/discard
	## Using existing test report protocols like XML		## Using existing test report protocols like XML
	# The TRT should present an interface with views organized towards the following objectives:		# The TRT should present an interface with views organized towards the following objectives:

Joshua Lock: /* Prior Art */ add links to openQA

2016-07-13T13:59:52Z

Prior Art: add links to openQA

← Older revision		Revision as of 13:59, 13 July 2016
Line 24:		Line 24:
	==Prior Art==		==Prior Art==

	* QA maintains [[QA_sanity_history]] reports which list sanity test results and compares them to the previous run (anyone know how these are generated?)		* QA maintains [[QA_sanity_history]] reports which list sanity test results and compares them to the previous run (anyone know how these are generated?).
	* Someone (?) generates graphs of the [https://wiki.yoctoproject.org/charts/perf_milestone/performance_test.html performance data]		* Someone (?) generates graphs of the [https://wiki.yoctoproject.org/charts/perf_milestone/performance_test.html performance data].
	* [https://github.com/fullvlad/CustomReports/ CustomReports] includes visualisation for test run pass rates and graphs changes to success rates.		* [https://github.com/fullvlad/CustomReports/ CustomReports] includes visualisation for test run pass rates and graphs changes to success rates.
	* [https://jenkins.io/ Jenkins] parses and displays JUnit XML output out of the box, a break down of [http://nelsonwells.net/2012/09/how-jenkins-ci-parses-and-displays-junit-output/ How Jenkins Displays JUnit output]		* [https://jenkins.io/ Jenkins] parses and displays JUnit XML output out of the box, a break down of [http://nelsonwells.net/2012/09/how-jenkins-ci-parses-and-displays-junit-output/ How Jenkins Displays JUnit output].
			* [https://openqa.opensuse.org/ OpenQA] from openSUSE has a visual status overview for the tests run vs an installed OS. Code is on github: https://github.com/os-autoinst/openQA and initial release announcement provides an overview: [https://news.opensuse.org/2011/10/11/opensuse-announces-first-public-release-of-openqa/ openSUSE Announces First Public Release of openQA].

Joshua Lock: Start to make a list of QA data sources

2016-07-06T20:40:03Z

Start to make a list of QA data sources

← Older revision		Revision as of 20:40, 6 July 2016
Line 13:		Line 13:
	## Boolean test results Passed/Failed		## Boolean test results Passed/Failed
	## Numeric test results (measurements)		## Numeric test results (measurements)

			==QA Data Sources==
			There are several sources of quality data we'd like to be able to track in a QA dashboard:
			* oe-selftest results
			* bitbake-selftest results
			* build performance metrics
			* buildhistory trends
			* ???

	==Prior Art==		==Prior Art==

Joshua Lock: Add some links to relevant prior art in visualising test results

2016-07-06T16:23:20Z

Add some links to relevant prior art in visualising test results

← Older revision		Revision as of 16:23, 6 July 2016
Line 13:		Line 13:
	## Boolean test results Passed/Failed		## Boolean test results Passed/Failed
	## Numeric test results (measurements)		## Numeric test results (measurements)

			==Prior Art==

			* QA maintains [[QA_sanity_history]] reports which list sanity test results and compares them to the previous run (anyone know how these are generated?)
			* Someone (?) generates graphs of the [https://wiki.yoctoproject.org/charts/perf_milestone/performance_test.html performance data]
			* [https://github.com/fullvlad/CustomReports/ CustomReports] includes visualisation for test run pass rates and graphs changes to success rates.
			* [https://jenkins.io/ Jenkins] parses and displays JUnit XML output out of the box, a break down of [http://nelsonwells.net/2012/09/how-jenkins-ci-parses-and-displays-junit-output/ How Jenkins Displays JUnit output]

Benjamin: Created page with "==General Expectations== Here's a list of initial expectations of the Test Reporting Tool. # The TRT must be able to receive test reports via a remote communication protocol..."

2016-07-01T17:55:57Z

Created page with "==General Expectations== Here's a list of initial expectations of the Test Reporting Tool. # The TRT must be able to receive test reports via a remote communication protocol..."

New page

==General Expectations==

Here's a list of initial expectations of the Test Reporting Tool.

# The TRT must be able to receive test reports via a remote communication protocol.
## From expected sources of results (Auth)
## To existing test buckets (logical test belonging separation, could be by test component)
## Using existing test report protocols like XML
# The TRT should present an interface with views organized towards the following objectives:
## Test Buckets (browsing a component's test results history)
## Test Collections (The type of Milestone/Release or another defined collection)
# The TRT should be able to identify the type of result it is consuming.
## Boolean test results Passed/Failed
## Numeric test results (measurements)