BSP Test Process: Difference between revisions

From Yocto Project
Jump to navigationJump to search
No edit summary
 
(18 intermediate revisions by the same user not shown)
Line 2: Line 2:


Given the resulttool command line tool was ready to replace Testopia and Wiki Page in QA Testing for execution and test result management. New BSP test process was needed to integrate BSP testing with resulttool.
Given the resulttool command line tool was ready to replace Testopia and Wiki Page in QA Testing for execution and test result management. New BSP test process was needed to integrate BSP testing with resulttool.
The resulttool is currently only available for 2.7, and it will be ported to 2.6 (thud) and 2.5 (sumo).


= BSP Test Process =  
= BSP Test Process =  
Line 13: Line 15:
  - Test Reporting and Regression
  - Test Reporting and Regression
  - Decision
  - Decision
= Roles =
- Test Executor - Intel and WindRiver test team
- Overall QA Owner - Intel (2.7)
- QA Stakeholder - Team that owned final release decision


== Test Triggering ==
== Test Triggering ==


=== ROLE: Test Executor ===
=== ROLE: Test Executor ===
Intel and WindRiver test team will monitor yocto mailing list for "QA notification for completed autobuilder build (yocto-<releasa>)", once QA notification received then each test team will start preparing for BPS testing.
Intel and WindRiver test team request subscribe to QA notification by contacting Linux Foundation (Michael Halstead mhalstead@linuxfoundation.org), once QA notification received then each test team will start preparing for BPS testing.
 


== Test Setup==
== Test Setup==
Line 23: Line 36:
=== ROLE: Test Executor ===
=== ROLE: Test Executor ===
Intel and WindRiver test team will setup test hardware and environment for BSP testing.
Intel and WindRiver test team will setup test hardware and environment for BSP testing.


== Test Execution ==
== Test Execution ==
Line 32: Line 46:


For runtime tests inside oeqa/manual, use "resulttool manualexecution" to execute these tests (refer to https://wiki.yoctoproject.org/wiki/Resulttool#manualexecution for more detail), collect the "testresults.json" file from the <build-dir>/tmp/log/manual/ directory.
For runtime tests inside oeqa/manual, use "resulttool manualexecution" to execute these tests (refer to https://wiki.yoctoproject.org/wiki/Resulttool#manualexecution for more detail), collect the "testresults.json" file from the <build-dir>/tmp/log/manual/ directory.
=== Defect management ===
Once a test case failed during execution, create a bug ticket at Bugzilla.
To map the bug ticket to the failed test case executed, provide below information inside bug ticket.
Information needed to trace bug to testcase inside yocto-testresults:
1. git tag - tag used to store the failed test case (eg. "{branch}/{commit_number}-g{commit}/{tag_number}")
2. result id - unique id inside testresults.json file that used to identify a test result set consist of a list of test cases executed
(eg. manual_bsps-hw_20190311160844)
3. test case id - unique id inside testresults.json file used to identify a specific test case executed
(eg. bsp-hw.bsps-oe-core.Test_Run_Integrity_-_Check_that_image_is_buildable)


== Test Result Store ==
== Test Result Store ==


=== ROLE: Test Executor ===
=== ROLE: Test Executor ===
Intel and WindRiver team will request write access to https://git.yoctoproject.org/git/yocto-testresults remote repository by contacting Linux Foundation (Michael Halstead mhalstead@linuxfoundation.org).
Intel and WindRiver test team will store the testresults.json files collected in previous section to the yocto-testresults git repository (http://git.yoctoproject.org/cgit/cgit.cgi/yocto-testresults/).
Intel and WindRiver test team will store the testresults.json files collected in previous section to the yocto-testresults git repository (http://git.yoctoproject.org/cgit/cgit.cgi/yocto-testresults/).


Line 42: Line 70:
  2. Use "resulttool store" to store testresults.json files to local yocto-testresults git repository (refer to https://wiki.yoctoproject.org/wiki/Resulttool#store for more detail).
  2. Use "resulttool store" to store testresults.json files to local yocto-testresults git repository (refer to https://wiki.yoctoproject.org/wiki/Resulttool#store for more detail).


  3. Once verified the results are saved properly to local git repository, push the commit & tag to remote repository.
  3. Once verified the results are saved properly to local git repository, push the commit & tag to remote repository.  
Make sure the git tag created (branch and commit) was correct and the correct testresults.json file was stored inside the local yocto-testresults git repository before push to remote.
 
Intel and WindRiver team will notifiy the Overall QA Owner on completion of BSP testing.
 


== Test Reporting and Regression==
== Test Reporting and Regression==


=== ROLE: Overall QA Owner ===
=== ROLE: Overall QA Owner ===
Overall QA Owner team (Linux foundation) will re-run “resulttool report” & “resulttool regression-git” to create report and regression for both automated and BSP test result (refer to https://wiki.yoctoproject.org/wiki/Resulttool#report and https://wiki.yoctoproject.org/wiki/Resulttool#regression-git for more detail).
Overall QA Owner team will re-run “resulttool report” & “resulttool regression-git” to create report and regression for both automated and BSP test result (refer to https://wiki.yoctoproject.org/wiki/Resulttool#report and https://wiki.yoctoproject.org/wiki/Resulttool#regression-git for more detail).
 


== Decision ==
== Decision ==

Latest revision as of 05:55, 12 March 2019

Summary

Given the resulttool command line tool was ready to replace Testopia and Wiki Page in QA Testing for execution and test result management. New BSP test process was needed to integrate BSP testing with resulttool.

The resulttool is currently only available for 2.7, and it will be ported to 2.6 (thud) and 2.5 (sumo).

BSP Test Process

New BSP test process consists of:

- Test Triggering
- Test Setup
- Test Execution 
- Test Result Store 
- Test Reporting and Regression
- Decision


Roles

- Test Executor - Intel and WindRiver test team
- Overall QA Owner - Intel (2.7)
- QA Stakeholder - Team that owned final release decision


Test Triggering

ROLE: Test Executor

Intel and WindRiver test team request subscribe to QA notification by contacting Linux Foundation (Michael Halstead mhalstead@linuxfoundation.org), once QA notification received then each test team will start preparing for BPS testing.


Test Setup

ROLE: Test Executor

Intel and WindRiver test team will setup test hardware and environment for BSP testing.


Test Execution

ROLE: Test Executor

Intel and WindRiver test team will execute BSP runtime testing on related hardware accordingly.

For runtime tests available from oeqa/runtime, once executed this runtime tests, collect the "testresults.json" file contain the configuration and status from the <build-dir>/tmp/log/oeqa/ directory.

For runtime tests inside oeqa/manual, use "resulttool manualexecution" to execute these tests (refer to https://wiki.yoctoproject.org/wiki/Resulttool#manualexecution for more detail), collect the "testresults.json" file from the <build-dir>/tmp/log/manual/ directory.

Defect management

Once a test case failed during execution, create a bug ticket at Bugzilla. To map the bug ticket to the failed test case executed, provide below information inside bug ticket.

Information needed to trace bug to testcase inside yocto-testresults:

1. git tag - tag used to store the failed test case (eg. "{branch}/{commit_number}-g{commit}/{tag_number}")
2. result id - unique id inside testresults.json file that used to identify a test result set consist of a list of test cases executed 

(eg. manual_bsps-hw_20190311160844)

3. test case id - unique id inside testresults.json file used to identify a specific test case executed 

(eg. bsp-hw.bsps-oe-core.Test_Run_Integrity_-_Check_that_image_is_buildable)


Test Result Store

ROLE: Test Executor

Intel and WindRiver team will request write access to https://git.yoctoproject.org/git/yocto-testresults remote repository by contacting Linux Foundation (Michael Halstead mhalstead@linuxfoundation.org).

Intel and WindRiver test team will store the testresults.json files collected in previous section to the yocto-testresults git repository (http://git.yoctoproject.org/cgit/cgit.cgi/yocto-testresults/).

1. Git clone https://git.yoctoproject.org/git/yocto-testresults to local machine. 
2. Use "resulttool store" to store testresults.json files to local yocto-testresults git repository (refer to https://wiki.yoctoproject.org/wiki/Resulttool#store for more detail).
3. Once verified the results are saved properly to local git repository, push the commit & tag to remote repository. 

Make sure the git tag created (branch and commit) was correct and the correct testresults.json file was stored inside the local yocto-testresults git repository before push to remote.

Intel and WindRiver team will notifiy the Overall QA Owner on completion of BSP testing.


Test Reporting and Regression

ROLE: Overall QA Owner

Overall QA Owner team will re-run “resulttool report” & “resulttool regression-git” to create report and regression for both automated and BSP test result (refer to https://wiki.yoctoproject.org/wiki/Resulttool#report and https://wiki.yoctoproject.org/wiki/Resulttool#regression-git for more detail).


Decision

ROLE: QA Stakeholder

QA Stakeholder will review test report and regression information created in previous section to make the release decision.