Resulttool

Summary

The resulttool is command line tool available inside poky/scripts, it was developed to manage test result files for both automated tests (eg. oeqa/runtime, oeqa/selftest, oeqa/sdk, etc) and manual tests (eg. oeqa/manual). It was developed as an alternative tool to replace test result storage and reporting using Testopia and Wiki Page.

To view the features available for resulttool, git clone poky repository, source poky environment, type "resulttool --help".

List of features

The resulttool provide plugin to:

- "manualexecution": helper script for results populating during manual test execution
- "merge": merge test result files/directories
- "store": store test results into a git repository
- "regression": regression file/directory analysis
- "regression-git": regression git analysis
- "report": summarise test results
- "log": show logs

manualexecution

The manualexecution plugin was developed to provide command line interface to execute manual tests inside oeqa/manual and it will create testresults.json file that contain the configurations and status of the testing (result format compatible to store, report, regression).

Current "Branch" and "Commit" used by the poky repository to run resulttool will be used to create the layers metadata inside testresults.json configuration. The layers metadata were important data required by the "resulttool store" for creating the git tag. Where the git tag will be used for "resulttool report" and "resulttool regression-git".

The "MACHINE" and "IMAGE_BASENAME" configurations were mandatory inputs for manualexecution as these configurations will be used for "resulttool regression" and "resulttool regression-git", where regression used the configurations data to find matching result set for regression comparison if specific result id was not provided.

manualexecution --help

helper script for results populating during manual test execution. You can find manual test case JSON file in meta/lib/oeqa/manual/

arguments:

 file        specify path to manual test case JSON file.Note: Please use "" to encapsulate the file path.

options:

 -h, --help            show this help message and exit
 -c CONFIG_OPTIONS_FILE, --config-options-file CONFIG_OPTIONS_FILE
                       the config options file to import and used as
                       available configuration option selection or make
                       config option file
 -m, --make-config-options-file
                       make the configuration options file based on provided
                       inputs
 -t TESTCASE_CONFIG_FILE, --testcase-config-file TESTCASE_CONFIG_FILE
                       the testcase configuration file to enable user to run
                       a selected set of test case or make a testcase
                       configuration file
 -d, --make-testcase-config-file
                       make the testcase configuration file to run a set of
                       test cases based on user selection

manualexecution example

To run manual sdk testcases inside oeqa/manual/sdk.json, use "resulttool manualexecution":

$ cd poky
$ source oe-init-build-env
$ resulttool manualexecution ../meta/lib/oeqa/manual/sdk.json

merge

The merge plugin was developed to facilitate test result file(s) merging.

merge --help

merge the results from multiple files/directories into the target file or directory

arguments:

 base_results    the results file/directory to import
 target_results  the target file or directory to merge the base_results with

options:

 -h, --help            show this help message and exit
 -t, --not-add-testseries
                       do not add testseries configuration to results
 -x EXECUTED_BY, --executed-by EXECUTED_BY
                       add executed-by configuration to each result file

merge example

To merge base testresults.json file to target directory.

Assume base at tmp/log/manual/testresults.json and target directory at <target-testresults-dir>.

$ cd poky
$ source oe-init-build-env
$ resulttool merge tmp/log/manual/testresults.json <target-testresults-dir>

store

The store plugin was developed to store testresults.json file and other artifacts files to local git repository (https://git.yoctoproject.org/git/yocto-testresults). Where the local git repository serves as a database containing test result information to be used by "report", "regression" and "regression-git" plugin.

store --help

takes a results file or directory of results files and stores them into the destination git repository, splitting out the results files as configured

arguments:

 source             source file or directory that contain the test result
                    files to be stored
 git_dir            the location of the git repository to store the results
                    in

options:

 -h, --help         show this help message and exit
 -a, --all          include all files, not just testresults.json files
 -e, --allow-empty  don't error if no results to store are found
 -x EXECUTED_BY, --executed-by EXECUTED_BY
                       add executed-by configuration to each result file
 -t EXTRA_TEST_ENV, --extra-test-env EXTRA_TEST_ENV
                       add extra test environment data to each result file
                       configuration

store example

To store tmp/log/manual/testresults.json to local result repository.

Assume "resulttool manualexecution" create the testresults.json file inside tmp/log/manual/testresults.json and yocto-testresults local git repository was clone at <yocto-testresults-dir>.

$ cd poky
$ source oe-init-build-env
$ resulttool store tmp/log/manual/ <yocto-testresults-dir>

regression

The regression plugin was developed to provide regression analysis between two set of results (eg. compare base set results to target set results). This "regression" shall be applied to specific test result file or directory.

regression --help

regression analysis comparing the base set of results to the target results

arguments:

 base_result           base result file/directory for the comparison
 target_result         target result file/directory to compare with

options:

 -h, --help            show this help message and exit
 -b BASE_RESULT_ID, --base-result-id BASE_RESULT_ID
                       (optional) filter the base results to this result ID
 -t TARGET_RESULT_ID, --target-result-id TARGET_RESULT_ID
                       (optional) filter the target results to this result ID

regression example

To have regression analysis on base result set and target result set.

Assume base test result files was at <base-result-dir>. Assume target test result files was at <target-result-dir>.

$ cd poky
$ source oe-init-build-env
$ resulttool regression <base-result-dir> <target-result-dir>

regression-git

The regression-git plugin was developed to provide regression analysis to test result available inside a local git repository. User need to clone yocto-testresults (https://git.yoctoproject.org/git/yocto-testresults) to local machine.

regression-git --help

regression analysis comparing base result set to target result set

arguments:

 repo                  the git repository containing the data

options:

 -h, --help            show this help message and exit
 -b BASE_RESULT_ID, --base-result-id BASE_RESULT_ID
                       (optional) default select regression based on
                       configurations unless base result id was provided
 -t TARGET_RESULT_ID, --target-result-id TARGET_RESULT_ID
                       (optional) default select regression based on
                       configurations unless target result id was provided
 --branch BRANCH, -B BRANCH
                       Branch to find commit in
 --branch2 BRANCH2     Branch to find comparision revisions in
 --commit COMMIT       Revision to search for
 --commit-number COMMIT_NUMBER
                       Revision number to search for, redundant if --commit
                       is specified
 --commit2 COMMIT2     Revision to compare with
 --commit-number2 COMMIT_NUMBER2
                       Revision number to compare with, redundant if
                       --commit2 is specified

regression-git example

To perform regression analysis on local result repository.

Assume base test result set and target result set was stored inside local git repository at <yocto-testresults-dir>.

$ cd poky
$ source oe-init-build-env
$ resulttool regression-git <yocto-testresults-dir> --commit <commit-of-base> --commit2 <commit-of-target>

report

The report plugin was developed to summarise test result available in a file/directory/git repository.

report --help

print a text-based summary of the test results

arguments:

 source_dir         source file/directory that contain the test result files
                    to summarise

options:

 -h, --help            show this help message and exit
 --branch BRANCH, -B BRANCH
                       Branch to find commit in
 --commit COMMIT       Revision to report
 -t TAG, --tag TAG     source_dir is a git repository, report on the tag
                       specified from that repository
 -m, --use_regression_map
                       instead of the default "store_map", use the
                       "regression_map" for report
 -r RAW_TEST_ONLY, --raw_test_only RAW_TEST_ONLY
                       output raw test result only for the user provided test
                       result id
 -s SELECTED_TEST_CASE_ONLY, --selected_test_case_only SELECTED_TEST_CASE_ONLY
                       output selected test case result for the user provided
                       test case id, if both test result id and test case id
                       are provided then output the selected test case result
                       from the provided test result id

report example

To get summary report for test result stored inside local git repository.

$ cd poky
$ source oe-init-build-env
$ resulttool report <yocto-testresults-dir>

log

The log plugin was developed to print log available in a file/directory/URL.

log --help

show the logs from test results

arguments:

 source_dir         the results file/directory/URL to import

options:

 -h, --help            show this help message and exit
 --ptest PTEST         show logs for a ptest
 --dump-ptest DIR      Dump all ptest log files to the specified directory.
 --reproducible REPRODUCIBLE
                       show logs for a reproducible test
 --prepend-run         Dump ptest results to a subdirectory named after the
                       test run when using --dump-ptest. Required if more
                       than one test run is present in the result file
 --raw                 show raw (ptest) logs. Deprecated. Alias for "--raw-
                       ptest"
 --raw-ptest           show raw ptest log
 --raw-reproducible    show raw reproducible build logs

log example

To dump ptest log to a directory.

$ cd poky
$ source oe-init-build-env
$ resulttool log <yocto-testresults-dir> --dump-ptest <output-dir-for-ptest>

To print custom ptest log from a custom ptest.

$ cd poky
$ source oe-init-build-env
$ resulttool log <yocto-testresults-dir> --ptest <package-name-that-provide-ptest>