BSP Test Process

Summary

Given the resulttool command line tool was ready to replace Testopia and Wiki Page in QA Testing for execution and test result management. New BSP test process was needed to integrate BSP testing with resulttool.

BSP Test Process

New BSP test process consists of:

- Test Triggering
- Test Setup
- Test Execution 
- Test Result Store 
- Test Reporting and Regression
- Decision

Roles

- Test Executor - Intel and WindRiver test team

- Overall QA Owner - Linux Foundation team

- QA Stakeholder - Team that owned final release decision

Test Triggering

ROLE: Test Executor

Intel and WindRiver test team will monitor yocto mailing list for "QA notification for completed autobuilder build (yocto-<releasa>)", once QA notification received then each test team will start preparing for BPS testing.

Test Setup

ROLE: Test Executor

Intel and WindRiver test team will setup test hardware and environment for BSP testing.

Test Execution

ROLE: Test Executor

Intel and WindRiver test team will execute BSP runtime testing on related hardware accordingly.

For runtime tests available from oeqa/runtime, once executed this runtime tests, collect the "testresults.json" file contain the configuration and status from the <build-dir>/tmp/log/oeqa/ directory.

For runtime tests inside oeqa/manual, use "resulttool manualexecution" to execute these tests (refer to https://wiki.yoctoproject.org/wiki/Resulttool#manualexecution for more detail), collect the "testresults.json" file from the <build-dir>/tmp/log/manual/ directory.

Test Result Store

ROLE: Test Executor

Intel and WindRiver test team will store the testresults.json files collected in previous section to the yocto-testresults git repository (http://git.yoctoproject.org/cgit/cgit.cgi/yocto-testresults/).

1. Git clone https://git.yoctoproject.org/git/yocto-testresults to local machine. 

2. Use "resulttool store" to store testresults.json files to local yocto-testresults git repository (refer to https://wiki.yoctoproject.org/wiki/Resulttool#store for more detail).

3. Once verified the results are saved properly to local git repository, push the commit & tag to remote repository. 

Make sure the git tag created (branch and commit) was correct and the correct testresults.json file was stored inside the local yocto-testresults git repository before push to remote.

Test Reporting and Regression

ROLE: Overall QA Owner

Overall QA Owner team (Linux foundation) will re-run “resulttool report” & “resulttool regression-git” to create report and regression for both automated and BSP test result (refer to https://wiki.yoctoproject.org/wiki/Resulttool#report and https://wiki.yoctoproject.org/wiki/Resulttool#regression-git for more detail).

Decision

ROLE: QA Stakeholder

QA Stakeholder will review test report and regression information created in previous section to make the release decision.