Qa-tools
Summary
The qa-tools git repository is the recipient for the QA automation scripts and tools. Is intended to store any tool that helps on the QA activities related to the Yocto Project
You can take a look at it via the gitweb interface.
For cloning the qa-tools repository:
$ git clone git://git.yoctoproject.org/qa-tools
List of Tools
Below is a list with detailed information about the tools/scripts already implemented
Testopia Update
Objective
This tool was designed to give an automated way to interact with Testopia activities.
Structure
This tool allows to create a test run based on Templates already defined for every release, and based on that, the tool is able to create the appropriate test run by cloning the exiting template and generating a new Test Run that will show the results of the test execution. The Tool consist in a python script that can be used for different actions and with different parameters, with a configuration file to get connected to the Testopia DB and then using the Testopia API to perform different actions on the DB.
The structure is as follows:
-> qa-tools/
|-testopia_update.py: Main script
|-testopia_update.config : Configuration file with the URL and the login info to the Testopia DB
|-> testopia_update/
|- __init__.py
|-> product/ : This directory contains all the available products used by QA on Testopia
|-> External/
|- testopia.py: This file contains the API that provides methods to work on Testopia DB.
How to use it
1- Clone repo
2- Install Virtual Environment tool
$ sudo apt-get install python-virtualenv
3- Create virtual environment
$ virtualenv venv
4- Activate virtual environment
$ source venv/bin/activate
5- Create configuration file, there is already a configuration file example that can be used
testopia_update.config.example [snapshot]
[Testopia] url = <TESTOPIA URL> - https://bugzilla.yoctoproject.org/xmlrpc.cgi username = <SOME-USER> - Bugzilla user password = <VERY-SECURE-PWD> - Bugzilla pswd store_location = /tmp/testopia_update
6- Install requirements
$ pip install -r requeriments.txt
7- Start using the testopia_update.py script
./testopia_update.py -h
Below is a brief explanation of each one of the parameters available for the script:
-h, --help
Show this help message and exit
--config
Configuration file with the updated data to get connected to the Testopia DB
--url
URL of Testopia instance. This is used in case a configuration file is not being specified
--username
Username of Testopia instance. This is used in case a configuration file is not being specified
--password
Username of Testopia instance. This is used in case a configuration file is not being specified
--list-products
List available products. This option retrieves the list of the available products that can be used on Testopia instance
-a --action {create,update, setup}
Action to execute can be create, update or setup. This should be combined with other option check Example section to see how can be used
-p --product
Product to create or update. This is the name of the product available on Testopia. This should be one of the valid products retrieved when using option --list-products
-c --category {AUTO,MANUAL}
Category for create or update. This represents the way how the Test Cases of the test run are being executed
-b --branch {master,jethro,dizzy,daisy,noexists}
Branch for create or update. Thi is the branch that is currently tested. Generally we used master branch as it represents the current release under test
-e --environment
Environment for create or update.This is the environment where the test cases are being executed, could be a Distro OS, VM or real HW
-o --optional
Optional parameter for get Test runs. This is a string added to the name of the Test Run usually used on BSPs to specify the image where the test cases will be executed
--project-version
Version of the project. This is the release of the project that is being validated, to check current version go to Releases
--project-milestone
Milestone of the project. This represents the milestone of the project where Test Run will be applicable, it should be added also the release (i.e 2.3 M1)
--project-revision
SCM Revision of the project. This is a combination of the Release candidate and the Hash of that release candidate on Poky go to Examples to check how this is used
--project-date
SCM version/revision date of the project. The date where the release candidate outs at main auto builder, format is YYYY-MM-DD
--test-plan
Name of the test plan of the product, used when test plan name is different from product name. Normally the test plans on Testopia are composed as "<PRDOUCT_NAME>: <BRANCH_NAME> branch" but sometimes the test plan name is not the same as the product name so the correct test plan name should be specified.
--testrun-id
Number of the test run to be updated, this option should be used along with update action. If you already know the ID of the test run to be updated this option should be used to avoid all other parameters to search for the test run.
--results-log
Results log. This is the log generated by the execution were are reported the Test Cases IDs with the corresponding status (PASSED | FAILED)
--clone-templates
Clone templates from one branch to another or same branch (only available) with action=setup
--old-project-version
Version of the project to clone template from. This option is only used when action=setup and --clone-templates called
--create-wiki
Create a wiki page for the final report of the QA test cycle (only available with setup action)
--new-branch
New branch to clone the Templates. The branch name where the new templates will be cloned to.
--verbose
Enable verbose mode.
--debug
Enable debug mode.
Examples
On all the examples add the --verbose variable to show details of the actions
Create a Test Run
./testopia_update.py
--config testopia_update.config
-a create
-p ADT
-c AUTO
-b master
-e "Ubuntu 16.04 x86_64"
--project-version 2.3
--project-milestone "2.3 M2"
--project-date 2017-03-09
--project-revision "rc1-7e37e6178358c9868bf0a62419f0e4d6c693d531"
--verbose
Update a Test Run
./testopia_update.py
--config testopia_update.config
-a update
-p ADT
-c AUTO
-b master
-e "Ubuntu 16.04 x86_64"
--project-version 2.3
--project-milestone "2.3 M2"
--project-date 2017-03-09
--project-revision "rc1-7e37e6178358c9868bf0a62419f0e4d6c693d531"
--results-log results.log
--verbose
Update a Test Run by ID
./testopia_update.py
--config testopia_update.config
-a update
--results-log results.log
--testrun-id 7049
Clone Templates on the same branch
./testopia_update.py
--config testopia_update.config
-a setup
-b master
--clone-templates
--project-version 2.4
--old-project-version 2.3
--product Kernel
Clone Templates from one branch to another
./testopia_update.py
--config testopia_update.config
-a setup
-b master
--clone-templates
--project-version 2.4
--old-project-version 2.3
--product Kernel
--new-branch pyro
Create wiki for Milestone report
This will create a wiki.page file with the content that should be copy and pasted on the new wiki created for the milestone QA cycle report
./testopia_update.py
--config testopia_update.config
-a setup
--create-wiki
--project-date 2017-06-23
--project-version 2.3
Improvements
- Add option to create templates by adding defined TC's
- Create a log of the tasks executed for future reference
Full Cycle Wrapper
Objective
This script was designed to help in the creation of a full test QA cycle for every release of Yocto Project
Structure
This script is a wrapper that creates a full test cycle for QA team, before all defined test runs were created manually and sometimes the correct templates or format was not used, with this script all the test runs for a full test cycle will be created.
-> qa-tools/scripts
|- full-test-cycle-wrapper.sh
This scrips calls the [Update Testopia] tool with already defined parameters and is capable to create all the Test Runs for Normal YOCTO cycle and also for META-INTEL cycle.
How To use it
1. Clone repo
2. Execute the script with required parameters
./full-test-cycle-wrapper.sh
Below is a brief explanation of the parameters used on the script:
-h --help
Show help message and exit.
-t {YOCTO,META-INTEL}
Type of test cycle, could be YOCTO or META-INTEL. The cycle that is wanted to be created YOCTO for a full test cycle and META-INTEL for only test against products with Intel architecture
-b {master,jethro,dizzy,daisy,noexists}
Branch for create or update. This is the branch that is currently tested. Generally we used master branch as it represents the current release under test
--project-version
Version of the project. This is the release of the project that is being validated, to check current version go to Releases
--project-milestone Milestone of the project. This represents the milestone of the project where Test Run will be applicable, in this case only the milestone should be set (M1, M2. M3)
-r {XXXXXX}
SCM Revision of the project, commit of branch tested.
-c {rc1 rc2 rcX}
Release candidate to be used.
--project-date PROJECT_DATE
SCM version/revision date of the project. The date where the release candidate outs at main auto builder, format is YYYY-MM-DD
Examples
Create Cycle for YOCTO
./full-test-cycle-wrapper.sh
-t YOCTO
-b master
-p 2.3
-m M1
-r fabf5d7597955ec664c232ab6597f6dd8ed0ade3
-c rc1
-d 2017-03-00
Create Cycle for META-INTEL
./full-test-cycle-wrapper.sh
-t META-INTEL
-b master
-p 2.3
-m M1
-r fabf5d7597955ec664c232ab6597f6dd8ed0ade3
-c rc1
-d 2017-03-00
Improvements
- Add option to create Test Runs per single component
- Create configuration file where components are defined instead of hard coded on the script
Ptest Parser
Objective
This script is designed to provide a report of ptest execution compared with previous results provided inf a format of wikitable with graphical indication of increase, decrease pass rate on components, as well as, indication of components that are not being executed during the tests, all this data is stored in a file named <release_under_test>-wiki.page
Structure
This script is a parser written in python that receives 2 logs as parameters as well as the release and commit of each log, the script parse the logs and generates a table with compared results, this table is stored in a file on wikimedia format to be copied to the final wiki page created.
-> qa-tools/scripts
|- ptest-parser.py
This scrips calls the [Update Testopia] tool with already defined parameters and is capable to create all the Test Runs for Normal YOCTO cycle and also for META-INTEL cycle.
How To use it
1. Clone repo
2. Execute the script with required parameters
./ptest-parser.py
Below is a description of the parameters used on the script:
optional arguments
-h, --help
show this help message and exit
required arguments:
-p PRE_LOG
prvious log of the pTest result
--pre_commit PRE_COMMIT
commit of the previous log.
--pre_release PRE_RELEASE
release of the previous log.
-c CURR_LOG
current log of pTets results.
--curr_commit CURR_COMMIT
commit of the current log.
--curr_release CURR_RELEASE
release of the current log.
Example
./ptest-parser.py
-p Ptest2.4_m3_rc1.txt
-c ptest.log
--pre_commit 5f6945f5031e1a4ca116cc1eccf4c2f9dc228547
--pre_release 2.4_M3_rc1
--curr_commit 592c3449697ce39bf07864fafd7f0a61ca230561
--curr_release 2.4_M4_rc1
Here is an example of one the reports generated Ptest 592c3449697ce39bf07864fafd7f0a61ca230561
Example
Proposed tools
This section is intended to add new tools to be implemented
- Create a tool to compare the automated test list against the Testopia Test list DB Bug 10147
How To Contribute
Contributing to any of the yoctoproject.org repositories, including qa-tools, can be achieved with the Poky Contributions guidelines.
Mantainers Anibal Limon José Pérez Carranza
