Resulttool

Summary

The resulttool is command line tool available inside poky/scripts, it was developed to manage test result files for both automated tests (eg. oeqa/runtime, oeqa/selftest, oeqa/sdk, etc) and manual tests (eg. oeqa/manual). It was developed as an alternative tool to replace test result storage and reporting using Testopia and Wiki Page.

To view the features available for resulttool, git clone poky repository, source poky environment, type "resulttool --help".

List of features

The resulttool provide plugin to:

- "manualexecution": helper script for results populating during manual test execution
- "merge": merge test result files/directories
- "store": store test results into a git repository
- "regression": regression file/directory analysis
- "regression-git": regression git analysis
- "report": summarise test results

manualexecution

The manualexecution plugin was developed to provide command line interface to execute manual tests inside oeqa/manual and it will create testresults.json file that contain the configurations and status of the testing (result format compatible to store, report, regression).

Current "Branch" and "Commit" used by the poky repository to run resulttool will be used to create the layers metadata inside testresults.json configuration. The layers metadata were important data required by the "resulttool store" for creating the git tag. Where the git tag will be used for "resulttool report" and "resulttool regression-git".

The "MACHINE" and "IMAGE_BASENAME" configurations were mandatory inputs for manualexecution as these configurations will be used for "resulttool regression" and "resulttool regression-git".

manualexecution --help

helper script for results populating during manual test execution. You can find manual test case JSON file in meta/lib/oeqa/manual/

arguments:

 file        specify path to manual test case JSON file.Note: Please use "" to encapsulate the file path.

manualexecution example

To run manual sdk testcases inside oeqa/manual/ directory, use "resulttool manualexecution":

$ cd poky
$ source oe-init-build-env
$ resulttool manualexecution ../meta/lib/oeqa/manual/sdk.json

merge

The merge plugin was developed to facilitate test result file(s) merging.

merge --help

merge the results from multiple files/directories into the target file or directory

arguments:

 base_results    the results file/directory to import
 target_results  the target file or directory to merge the base_results with

store

The store plugin was developed to store testresults.json file and other artifacts files to local git repository. Where the local git repository will served as database containing test result information to be used by "report", "regression" and "regression-git" plugin.

store --help

takes a results file or directory of results files and stores them into the destination git repository, splitting out the results files as configured

arguments:

 source             source file or directory that contain the test result
                    files to be stored
 git_dir            the location of the git repository to store the results
                    in

options:

 -h, --help         show this help message and exit
 -a, --all          include all files, not just testresults.json files
 -e, --allow-empty  don't error if no results to store are found

store example

Assume "resulttool manualexecution" create the testresults.json file inside tmp/log/manual/testresults.json. Assume yocto-testresults git repository was clone at <yocto-testresults-dir>.

$ cd poky
$ source oe-init-build-env
$ resulttool store tmp/log/manual/ <yocto-testresults-dir>

regression

The regression plugin was developed to provide regression analysis between two set of results (eg. compare base set results to target set results). This "regression" shall be applied to specific test result file or directory.

regression --help

regression analysis comparing the base set of results to the target results

arguments:

 base_result           base result file/directory for the comparison
 target_result         target result file/directory to compare with

options:

 -h, --help            show this help message and exit
 -b BASE_RESULT_ID, --base-result-id BASE_RESULT_ID
                       (optional) filter the base results to this result ID
 -t TARGET_RESULT_ID, --target-result-id TARGET_RESULT_ID
                       (optional) filter the target results to this result ID

regression example

Assume base test result files was at <base-result-dir>. Assume target test result files was at <target-result-dir>.

$ cd poky
$ source oe-init-build-env
$ resulttool regression <base-result-dir> <target-result-dir>

regression-git

The regression-git plugin was developed to provide regression analysis to test result available inside a local git repository.

regression-git --help

regression analysis comparing base result set to target result set

arguments:

 repo                  the git repository containing the data

options:

 -h, --help            show this help message and exit
 -b BASE_RESULT_ID, --base-result-id BASE_RESULT_ID
                       (optional) default select regression based on
                       configurations unless base result id was provided
 -t TARGET_RESULT_ID, --target-result-id TARGET_RESULT_ID
                       (optional) default select regression based on
                       configurations unless target result id was provided
 --branch BRANCH, -B BRANCH
                       Branch to find commit in
 --branch2 BRANCH2     Branch to find comparision revisions in
 --commit COMMIT       Revision to search for
 --commit-number COMMIT_NUMBER
                       Revision number to search for, redundant if --commit
                       is specified
 --commit2 COMMIT2     Revision to compare with
 --commit-number2 COMMIT_NUMBER2
                       Revision number to compare with, redundant if
                       --commit2 is specified

regression-git example

Assume base test result set and target result set was stored inside local git repository at <yocto-testresults-dir>.

$ cd poky
$ source oe-init-build-env
$ resulttool regression-git <yocto-testresults-dir> --commit <commit-of-base> --commit2 <commit-of-target>

report

The report plugin was developed to summarise test result available in a file/directory/git repository.

report --help

print a text-based summary of the test results

arguments:

 source_dir         source file/directory that contain the test result files
                    to summarise

options:

 -h, --help         show this help message and exit
 -t TAG, --tag TAG  source_dir is a git repository, report on the tag
                    specified from that repository

report example

$ cd poky
$ source oe-init-build-env
$ resulttool report <yocto-testresults-dir>