QA/Test Report Tool

General Expectations

(add link share-ability of results to the list of requirements when it fits somewhere)

Here's a list of initial expectations of the Test Reporting Tool.

1. The TRT must be able to receive test reports via a remote communication protocol.
   1. From expected sources of results (Auth)
   2. To existing test buckets (logical test belonging separation, could be by test component) through report sessions with:
      • ability to resume
      • ability to update existing results
      • ability to abort/discard
   3. Using existing test report protocols like XML
2. The TRT should present an interface with views organized towards the following objectives:
   1. Test Buckets (browsing a component's test results history)
   2. Test Collections (The type of Milestone/Release or another defined collection)
3. The TRT should be able to identify the type of result it is consuming.
   1. Boolean test results Passed/Failed
   2. Numeric test results (measurements)

QA Data Sources

There are several sources of quality data we'd like to be able to track in a QA dashboard:

• oe-selftest results
• bitbake-selftest results
• build performance metrics
• buildhistory trends
• ???

Prior Art

• QA maintains QA_sanity_history reports which list sanity test results and compares them to the previous run (anyone know how these are generated?).
• Someone (?) generates graphs of the performance data.
• CustomReports includes visualisation for test run pass rates and graphs changes to success rates.
• Jenkins parses and displays JUnit XML output out of the box, a break down of How Jenkins Displays JUnit output.
• OpenQA from openSUSE has a visual status overview for the tests run vs an installed OS. Code is on github: https://github.com/os-autoinst/openQA and initial release announcement provides an overview: openSUSE Announces First Public Release of openQA.