Qa-tools
Summary
The qa-tools git repository is the recipient for the QA automation scripts and tools. Is inteended to store any tool that helps on the QA activities related to the Yocto Porject
You can take a look at it via the gitweb interface.
For cloning the qa-tools repository:
$ git clone git://git.yoctoproject.org/qa-tools
List of Tools
Below is a list with detailed information about the tools/scripts already implemented
Testopia Update
Objective
This tool was designed to give an automated way to interact with Testopia activities.
Structure
This tool allows to create a test run based on [ Templates ] already defined for every release, and based on that the tool is able to create the appropiate test run by cloning the exiting template and generating a new Test Run that will show the results of the test execution. Tool consist in a python script that can be used for different actions and with different parameters, with a configuration file to get connected to the Testopia DB and then using the Testopia API to perform different actions on the DB.
The structure is as follows:
-> qa-tools/
|-testopia_update.py: Main script
|-testopia_update.config : Configuration file with the URL and the login info to the Testopia DB
|-> testopia_update/
|- __init__.py
|-> product/ : This directory contains all the available products used by QA on Testopia
|-> External/
|- testopia.py: This file contains the API that provides methods to work on Testopia DB.
How to use it
1- Clone repo
2- Install Virtual Environment tool
$ sudo apt-get install python-virtualenv
3- Create virtual environment
$ virtualenv venv
4- Activate virtual environment
$ source venv/bin/activate
5- Create configuration file, there is already a configuration file example that can be used
testopia_update.config.example [snapshot]
[Testopia] url = TESTOPIA URL - https://bugzilla.yoctoproject.org/xmlrpc.cgi username = someuser - Bugzilla user password = verysecurepass - Bugzilla pswd store_location = /tmp/testopia_update
6- Install requirements
$ pip install -r requeriments.txt
7- Start using the testopia_update.py script
./testopia_update.py -h
Below is a brief explanation of each one of the parameters available for the script:
-h, --help
Show this help message and exit
--config
Configuration file with the updated data to get connected to the Testopia DB
--url
URL of Testopia instance. This is used in case a configuration file is not being specified
--username
Username of Testopia instance. This is used in case a configuration file is not being specified
--password
Username of Testopia instance. This is used in case a configuration file is not being specified
--list-products
List available products. This option retrieves the list of the available products that can be used on Testopia instance
-a --action {create,update}
Action to execute can be create or update. This should be combined with other option check [example] section to see how can be used
-p --product
Product to create or update. This is the name of the product available on Testopia. This should be one of the valid products retrieved when using option --list-products
-c --category {AUTO,MANUAL}
Category for create or update. This represents the way how the Test Cases of the test run are being executed
-b --branch {master,jethro,dizzy,daisy,noexists}
Branch for create or update. Thi is the branch that is currently tested. Generally we used master branch as it represente the curren release under test
-e --environment
Environment for create or update.This is teh environemt where the test cases are being executed, could be a Distro OS, VM or real HW
-o --optional
Optional parameter for get Test runs. This is a string added to the name of the Test Run usually used on BSPs to specify the image where the test cases will be executed
--project-version
Version of the project. This is the release of the project that is being validated, to check current version go to [Releases]
--project-milestone
Milestone of the project. This representes the milestone of the project where Tes Run will be applicable, it should be added also the release (i.e 2.3 M1)
--project-revision
SCM Revision of the project. This is a combination of the Release candidate and the Hash of that release candidate on Poky go to [Examples] to check how this is used
--project-date PROJECT_DATE
SCM version/revision date of the project. The date where the release candidate outs at main auto builder, format is YYYY-MM-DD
--test-plan
Name of the test plan of the product, used when test plan name is different from product name. Normally the test plans on Testopia are composed as "<PRDOUCT_NAME>: <BRANCH_NAME> branch" but sometimes the test plan name is not the same as the product name so the correct test plan name should be specified.
--testrun-id
Number of the test run to be updated, this option should be used along with update action. If you already know the ID of the test run to be updated this option should be used to avoid all other parameters to search for the test run.
--results-log
Results log. This is the log generated by the execution were are reported the Test Cases IDs with the corresponding status (PASSED | FAILED)
--verbose
Enable verbose mode.
--debug
Enable debug mode.
Examples
Create a Test Run
./testopia_update.py
--config testopia_update.config
-a create
-p ADT
-c AUTO
-b master
-e "Ubuntu 16.04 x86_64"
--project-version 2.3
--project-milestone "2.3 M2"
--project-date 2017-03-09
--project-revision "rc1-7e37e6178358c9868bf0a62419f0e4d6c693d531"
--verbose
Update a Test Run
./testopia_update.py
--config testopia_update.config
-a update
-p ADT
-c AUTO
-b master
-e "Ubuntu 16.04 x86_64"
--project-version 2.3
--project-milestone "2.3 M2"
--project-date 2017-03-09
--project-revision "rc1-7e37e6178358c9868bf0a62419f0e4d6c693d531"
--results-log results.log
--verbose
Update a Test Run by ID
./testopia_update.py
--config testopia_update.config
-a update
--results-log results.log
--testrun-id 7049
Improvements
- Add option to create templates by adding defined TC's
- Create a log of the tasks executed for future reference
Full Cycle Wrapper
Objective
This script was designed to help in the creation of a full test QA cycle for every release of Yocto Project
Structure
This script is a wrapper that creates a full test cycle for QA team, before all defined test runs were created manually and sometimes the correct templates or format was not used, with this script all the test runs for a full test cycle will be created.
-> qa-tools/scripts
|- full-test-cycle-wrapper.sh
This scrips calls the [Update Testopia] tool with already defined parameters and is capable to create all the Test Runs for Normal YOCTO cycle and also for META-INTEL cycle.
Below is a brief explanation of the parameters used on the script:
-h --help
Show help message and exit.
-t {YOCTO,META-INTEL}
Type of test cycle, could be YOCTO or META-INTEL. The cycle that is wanted to be created YOCTO for a full test cycle and META-INTEL for only test against products with Intel architecture
How To use it
Examples
Improvements
Proposed tools
This section is intended to add new tools to be implemented
- Create a tool to compare the automated test list against the Testopia Test list DB - Bug 10147
How To Contribute
Contributing to any of the yoctoproject.org repositories, including qa-tools, can be achieved with the Poky Contributions guidelines.
Mantainer Anibal Limon
