BSP Test Process

Summary

Given the resulttool command line tool was ready to replace Testopia and Wiki Page in QA Testing for execution and test result management. New BSP test process was needed to integrate BSP testing with resulttool.

The resulttool is currently only available for 2.7, and it will be ported to 2.6 (thud) and 2.5 (sumo).

BSP Test Process

New BSP test process consists of:

- Test Triggering
- Test Setup
- Test Execution 
- Test Result Store 
- Test Reporting and Regression
- Decision

Roles

- Test Executor - Intel and WindRiver test team

- Overall QA Owner - Intel (2.7)

- QA Stakeholder - Team that owned final release decision

Test Triggering

ROLE: Test Executor

Intel and WindRiver test team request subscribe to QA notification by contacting Linux Foundation (Michael Halstead mhalstead@linuxfoundation.org), once QA notification received then each test team will start preparing for BPS testing.

Test Setup

ROLE: Test Executor

Intel and WindRiver test team will setup test hardware and environment for BSP testing.

Test Execution

ROLE: Test Executor

Intel and WindRiver test team will execute BSP runtime testing on related hardware accordingly.

For runtime tests available from oeqa/runtime, once executed this runtime tests, collect the "testresults.json" file contain the configuration and status from the <build-dir>/tmp/log/oeqa/ directory.

For runtime tests inside oeqa/manual, use "resulttool manualexecution" to execute these tests (refer to https://wiki.yoctoproject.org/wiki/Resulttool#manualexecution for more detail), collect the "testresults.json" file from the <build-dir>/tmp/log/manual/ directory.

Defect management

Once a test case failed during execution, create a bug ticket at Bugzilla. To map the bug ticket to the failed test case executed, provide below information inside bug ticket.

Information needed to trace bug to testcase inside yocto-testresults:

1. git tag - tag used to store the failed test case (eg. "{branch}/{commit_number}-g{commit}/{tag_number}")
2. result id - unique id inside testresults.json file that used to identify a test result set consist of a list of test cases executed 

(eg. manual_bsps-hw_20190311160844)

3. test case id - unique id inside testresults.json file used to identify a specific test case executed 

(eg. bsp-hw.bsps-oe-core.Test_Run_Integrity_-_Check_that_image_is_buildable)

Test Result Store

ROLE: Test Executor

Intel and WindRiver team will request write access to https://git.yoctoproject.org/git/yocto-testresults remote repository by contacting Linux Foundation (Michael Halstead mhalstead@linuxfoundation.org).

Intel and WindRiver test team will store the testresults.json files collected in previous section to the yocto-testresults git repository (http://git.yoctoproject.org/cgit/cgit.cgi/yocto-testresults/).

1. Git clone https://git.yoctoproject.org/git/yocto-testresults to local machine. 

2. Use "resulttool store" to store testresults.json files to local yocto-testresults git repository (refer to https://wiki.yoctoproject.org/wiki/Resulttool#store for more detail).

3. Once verified the results are saved properly to local git repository, push the commit & tag to remote repository. 

Make sure the git tag created (branch and commit) was correct and the correct testresults.json file was stored inside the local yocto-testresults git repository before push to remote.

Intel and WindRiver team will notifiy the Overall QA Owner on completion of BSP testing.

Test Reporting and Regression

ROLE: Overall QA Owner

Overall QA Owner team will re-run “resulttool report” & “resulttool regression-git” to create report and regression for both automated and BSP test result (refer to https://wiki.yoctoproject.org/wiki/Resulttool#report and https://wiki.yoctoproject.org/wiki/Resulttool#regression-git for more detail).

Decision

ROLE: QA Stakeholder

QA Stakeholder will review test report and regression information created in previous section to make the release decision.