QA/Test Report Tool: Difference between revisions

From Yocto Project
Jump to navigationJump to search
(→‎Prior Art: add links to openQA)
m (Benjamin moved page QA/Test Reporting Tool to QA/Test Report Tool: Name change to simplify its search)
 
(2 intermediate revisions by the same user not shown)
Line 1: Line 1:
==General Expectations==
==General Expectations==
(add link share-ability of results to the list of requirements when it fits somewhere)


Here's a list of initial expectations of the Test Reporting Tool.
Here's a list of initial expectations of the Test Reporting Tool.
Line 5: Line 7:
# The TRT must be able to receive test reports via a remote communication protocol.
# The TRT must be able to receive test reports via a remote communication protocol.
## From expected sources of results (Auth)
## From expected sources of results (Auth)
## To existing test buckets (logical test belonging separation, could be by test component)
## To existing test buckets (logical test belonging separation, could be by test component) through report sessions with:
##* ability to resume
##* ability to update existing results
##* ability to abort/discard
## Using existing test report protocols like XML  
## Using existing test report protocols like XML  
# The TRT should present an interface with views organized towards the following objectives:
# The TRT should present an interface with views organized towards the following objectives:

Latest revision as of 19:47, 10 November 2016

General Expectations

(add link share-ability of results to the list of requirements when it fits somewhere)

Here's a list of initial expectations of the Test Reporting Tool.

  1. The TRT must be able to receive test reports via a remote communication protocol.
    1. From expected sources of results (Auth)
    2. To existing test buckets (logical test belonging separation, could be by test component) through report sessions with:
      • ability to resume
      • ability to update existing results
      • ability to abort/discard
    3. Using existing test report protocols like XML
  2. The TRT should present an interface with views organized towards the following objectives:
    1. Test Buckets (browsing a component's test results history)
    2. Test Collections (The type of Milestone/Release or another defined collection)
  3. The TRT should be able to identify the type of result it is consuming.
    1. Boolean test results Passed/Failed
    2. Numeric test results (measurements)

QA Data Sources

There are several sources of quality data we'd like to be able to track in a QA dashboard:

  • oe-selftest results
  • bitbake-selftest results
  • build performance metrics
  • buildhistory trends
  • ???

Prior Art