QA/Test Report Tool

General Expectations

Here's a list of initial expectations of the Test Reporting Tool.

1. The TRT must be able to receive test reports via a remote communication protocol.
   1. From expected sources of results (Auth)
   2. To existing test buckets (logical test belonging separation, could be by test component)
   3. Using existing test report protocols like XML
2. The TRT should present an interface with views organized towards the following objectives:
   1. Test Buckets (browsing a component's test results history)
   2. Test Collections (The type of Milestone/Release or another defined collection)
3. The TRT should be able to identify the type of result it is consuming.
   1. Boolean test results Passed/Failed
   2. Numeric test results (measurements)

QA Data Sources

There are several sources of quality data we'd like to be able to track in a QA dashboard:

• oe-selftest results
• bitbake-selftest results
• build performance metrics
• buildhistory trends
• ???

Prior Art

• QA maintains QA_sanity_history reports which list sanity test results and compares them to the previous run (anyone know how these are generated?)
• Someone (?) generates graphs of the performance data
• CustomReports includes visualisation for test run pass rates and graphs changes to success rates.
• Jenkins parses and displays JUnit XML output out of the box, a break down of How Jenkins Displays JUnit output