QA/Test Report Tool: Difference between revisions
From Yocto Project
Jump to navigationJump to search
(Created page with "==General Expectations== Here's a list of initial expectations of the Test Reporting Tool. # The TRT must be able to receive test reports via a remote communication protocol...") |
Joshua Lock (talk | contribs) (Add some links to relevant prior art in visualising test results) |
||
| Line 13: | Line 13: | ||
## Boolean test results Passed/Failed | ## Boolean test results Passed/Failed | ||
## Numeric test results (measurements) | ## Numeric test results (measurements) | ||
==Prior Art== | |||
* QA maintains [[QA_sanity_history]] reports which list sanity test results and compares them to the previous run (anyone know how these are generated?) | |||
* Someone (?) generates graphs of the [https://wiki.yoctoproject.org/charts/perf_milestone/performance_test.html performance data] | |||
* [https://github.com/fullvlad/CustomReports/ CustomReports] includes visualisation for test run pass rates and graphs changes to success rates. | |||
* [https://jenkins.io/ Jenkins] parses and displays JUnit XML output out of the box, a break down of [http://nelsonwells.net/2012/09/how-jenkins-ci-parses-and-displays-junit-output/ How Jenkins Displays JUnit output] | |||
Revision as of 16:23, 6 July 2016
General Expectations
Here's a list of initial expectations of the Test Reporting Tool.
- The TRT must be able to receive test reports via a remote communication protocol.
- From expected sources of results (Auth)
- To existing test buckets (logical test belonging separation, could be by test component)
- Using existing test report protocols like XML
- The TRT should present an interface with views organized towards the following objectives:
- Test Buckets (browsing a component's test results history)
- Test Collections (The type of Milestone/Release or another defined collection)
- The TRT should be able to identify the type of result it is consuming.
- Boolean test results Passed/Failed
- Numeric test results (measurements)
Prior Art
- QA maintains QA_sanity_history reports which list sanity test results and compares them to the previous run (anyone know how these are generated?)
- Someone (?) generates graphs of the performance data
- CustomReports includes visualisation for test run pass rates and graphs changes to success rates.
- Jenkins parses and displays JUnit XML output out of the box, a break down of How Jenkins Displays JUnit output
