Qa-tools

Summary

The qa-tools git repository is the recipient for the QA automation scripts and tools. Is intended to store any tool that helps on the QA activities related to the Yocto Project

You can take a look at it via the gitweb interface.

For cloning the qa-tools repository:

$ git clone git://git.yoctoproject.org/qa-tools

List of Tools

Below is a list with detailed information about the tools/scripts already implemented

Testopia Update

Objective

This tool was designed to give an automated way to interact with Testopia activities.

Structure

This tool allows to create a test run based on Templates already defined for every release, and based on that, the tool is able to create the appropriate test run by cloning the exiting template and generating a new Test Run that will show the results of the test execution. The Tool consist in a python script that can be used for different actions and with different parameters, with a configuration file to get connected to the Testopia DB and then using the Testopia API to perform different actions on the DB.

The structure is as follows:

-> qa-tools/ 
  |-testopia_update.py: Main script 
  |-testopia_update.config : Configuration file with the URL and the login info to the Testopia DB
  |-> testopia_update/
     |- __init__.py
     |-> product/ : This directory contains all the available products used by QA on Testopia                
  |-> External/
     |- testopia.py: This file contains the API that provides methods to work on Testopia DB.

How to use it

1- Clone repo

2- Install Virtual Environment tool

$ sudo apt-get install python-virtualenv

3- Create virtual environment

$ virtualenv venv

4- Activate virtual environment

$ source venv/bin/activate

5- Create configuration file, there is already a configuration file example that can be used

testopia_update.config.example [snapshot]

[Testopia]
url = <TESTOPIA URL> - https://bugzilla.yoctoproject.org/xmlrpc.cgi 
username = <SOME-USER> - Bugzilla user
password = <VERY-SECURE-PWD> - Bugzilla pswd
store_location = /tmp/testopia_update

6- Install requirements

$ pip install -r requeriments.txt

7- Start using the testopia_update.py script

./testopia_update.py -h 

Below is a brief explanation of each one of the parameters available for the script:

-h, --help

Show this help message and exit

--config

Configuration file with the updated data to get connected to the Testopia DB

--url

URL of Testopia instance. This is used in case a configuration file is not being specified

--username

Username of Testopia instance. This is used in case a configuration file is not being specified

--password

Username of Testopia instance. This is used in case a configuration file is not being specified

--list-products

List available products. This option retrieves the list of the available products that can be used on Testopia instance

-a --action {create,update, setup}

Action to execute can be create, update or setup. This should be combined with other option check Example section to see how can be used

-p --product

Product to create or update. This is the name of the product available on Testopia. This should be one of the valid products retrieved when using option --list-products

-c --category {AUTO,MANUAL}

Category for create or update. This represents the way how the Test Cases of the test run are being executed

-b --branch {master,jethro,dizzy,daisy,noexists}

Branch for create or update. Thi is the branch that is currently tested. Generally we used master branch as it represents the current release under test

-e --environment

Environment for create or update.This is the environment where the test cases are being executed, could be a Distro OS, VM or real HW

-o --optional

Optional parameter for get Test runs. This is a string added to the name of the Test Run usually used on BSPs to specify the image where the test cases will be executed

--project-version

Version of the project. This is the release of the project that is being validated, to check current version go to Releases

--project-milestone

Milestone of the project. This represents the milestone of the project where Test Run will be applicable, it should be added also the release (i.e 2.3 M1)

--project-revision

SCM Revision of the project. This is a combination of the Release candidate and the Hash of that release candidate on Poky go to Examples to check how this is used

--project-date

SCM version/revision date of the project. The date where the release candidate outs at main auto builder, format is YYYY-MM-DD

--test-plan

Name of the test plan of the product, used when test plan name is different from product name. Normally the test plans on Testopia are composed as "<PRDOUCT_NAME>: <BRANCH_NAME> branch" but sometimes the test plan name is not the same as the product name so the correct test plan name should be specified.

--testrun-id

Number of the test run to be updated, this option should be used along with update action. If you already know the ID of the test run to be updated this option should be used to avoid all other parameters to search for the test run.

--results-log

Results log. This is the log generated by the execution were are reported the Test Cases IDs with the corresponding status (PASSED | FAILED)

--clone-templates

Clone templates from one branch to another or same branch (only available) with action=setup

--old-project-version

Version of the project to clone template from. This option is only used when action=setup and --clone-templates called

--create-wiki

Create a wiki page for the final report of the QA test cycle (only available with setup action)

--new-branch

New branch to clone the Templates. The branch name where the new templates will be cloned to.

--verbose

Enable verbose mode.

--debug

Enable debug mode.

Examples

On all the examples add the --verbose variable to show details of the actions

Create a Test Run

./testopia_update.py 
                        --config testopia_update.config
                        -a create 
                        -p ADT 
                        -c AUTO 
                        -b master 
                        -e "Ubuntu 16.04 x86_64"
                        --project-version 2.3 
                        --project-milestone "2.3 M2" 
                        --project-date 2017-03-09 
                        --project-revision "rc1-7e37e6178358c9868bf0a62419f0e4d6c693d531" 
                        --verbose

Update a Test Run

./testopia_update.py 
                        --config testopia_update.config
                        -a update
                        -p ADT 
                        -c AUTO 
                        -b master 
                        -e "Ubuntu 16.04 x86_64"
                        --project-version 2.3 
                        --project-milestone "2.3 M2" 
                        --project-date 2017-03-09 
                        --project-revision "rc1-7e37e6178358c9868bf0a62419f0e4d6c693d531" 
                        --results-log results.log
                        --verbose

Update a Test Run by ID

./testopia_update.py 
                        --config testopia_update.config
                        -a update
                        --results-log  results.log
                        --testrun-id 7049

Clone Templates on the same branch

./testopia_update.py  
                        --config testopia_update.config
                        -a setup
                        -b master 
                        --clone-templates  
                        --project-version 2.4 
                        --old-project-version 2.3 
                        --product Kernel

Clone Templates from one branch to another

./testopia_update.py  
                        --config testopia_update.config
                        -a setup 
                        -b master 
                        --clone-templates  
                        --project-version 2.4 
                        --old-project-version 2.3 
                        --product Kernel
                        --new-branch pyro

Create wiki for Milestone report

This will create a wiki.page file with the content that should be copy and pasted on the new wiki created for the milestone QA cycle report

./testopia_update.py  
                        --config testopia_update.config
                        -a setup 
                        --create-wiki  
                        --project-date 2017-06-23
                        --project-version 2.3

Improvements

• Add option to create templates by adding defined TC's
• Create a log of the tasks executed for future reference

Full Cycle Wrapper

Objective

This script was designed to help in the creation of a full test QA cycle for every release of Yocto Project

Structure

This script is a wrapper that creates a full test cycle for QA team, before all defined test runs were created manually and sometimes the correct templates or format was not used, with this script all the test runs for a full test cycle will be created.

-> qa-tools/scripts
          |-  full-test-cycle-wrapper.sh

This scrips calls the [Update Testopia] tool with already defined parameters and is capable to create all the Test Runs for Normal YOCTO cycle and also for META-INTEL cycle.

How To use it

1. Clone repo

2. Execute the script with required parameters

./full-test-cycle-wrapper.sh

Below is a brief explanation of the parameters used on the script:

-h --help

Show help message and exit.

-t {YOCTO,META-INTEL}

Type of test cycle, could be YOCTO or META-INTEL. The cycle that is wanted to be created YOCTO for a full test cycle and META-INTEL for only test against products with Intel architecture

-b {master,jethro,dizzy,daisy,noexists}

Branch for create or update. This is the branch that is currently tested. Generally we used master branch as it represents the current release under test

--project-version

Version of the project. This is the release of the project that is being validated, to check current version go to Releases

--project-milestone Milestone of the project. This represents the milestone of the project where Test Run will be applicable, in this case only the milestone should be set (M1, M2. M3)

-r {XXXXXX}

SCM Revision of the project, commit of branch tested.

-c {rc1 rc2 rcX}

Release candidate to be used.

--project-date PROJECT_DATE

SCM version/revision date of the project. The date where the release candidate outs at main auto builder, format is YYYY-MM-DD

Examples

Create Cycle for YOCTO

./full-test-cycle-wrapper.sh
                               -t YOCTO 
                               -b master 
                               -p 2.3  
                               -m M1 
                               -r fabf5d7597955ec664c232ab6597f6dd8ed0ade3
                               -c rc1 
                               -d 2017-03-00

Create Cycle for META-INTEL

./full-test-cycle-wrapper.sh
                               -t META-INTEL 
                               -b master 
                               -p 2.3  
                               -m M1 
                               -r fabf5d7597955ec664c232ab6597f6dd8ed0ade3
                               -c rc1 
                               -d 2017-03-00

Improvements

• Add option to create Test Runs per single component
• Create configuration file where components are defined instead of hard coded on the script

Ptest Parser

Objective

This script is designed to provide a report of ptest execution compared with previous results provided inf a format of wikitable with graphical indication of increase, decrease pass rate on components, as well as, indication of components that are not being executed during the tests, all this data is stored in a file named <release_under_test>-wiki.page

Structure

This script is a parser written in python that receives 2 logs as parameters as well as the release and commit of each log, the script parse the logs and generates a table with compared results, this table is stored in a file on wikimedia format to be copied to the final wiki page created.

-> qa-tools/scripts
          |-  ptest-parser.py

This scrips calls the [Update Testopia] tool with already defined parameters and is capable to create all the Test Runs for Normal YOCTO cycle and also for META-INTEL cycle.

How To use it

1. Clone repo

2. Execute the script with required parameters

./ptest-parser.py

Below is a description of the parameters used on the script:

optional arguments

-h, --help

show this help message and exit

required arguments:

-p PRE_LOG

prvious log of the pTest result

--pre_commit PRE_COMMIT

commit of the previous log.

--pre_release PRE_RELEASE

release of the previous log.

-c CURR_LOG

current log of pTets results.

--curr_commit CURR_COMMIT

commit of the current log.

--curr_release CURR_RELEASE

release of the current log.

Example

./ptest-parser.py
                    -p Ptest2.4_m3_rc1.txt 
                    -c ptest.log 
                    --pre_commit 5f6945f5031e1a4ca116cc1eccf4c2f9dc228547 
                    --pre_release 2.4_M3_rc1 
                    --curr_commit 592c3449697ce39bf07864fafd7f0a61ca230561 
                    --curr_release 2.4_M4_rc1

Here is an example of one the reports generated Ptest 592c3449697ce39bf07864fafd7f0a61ca230561

Proposed tools

This section is intended to add new tools to be implemented

• Create a tool to compare the automated test list against the Testopia Test list DB Bug 10147

How To Contribute

Contributing to any of the yoctoproject.org repositories, including qa-tools, can be achieved with the Poky Contributions guidelines.

Mantainers Anibal Limon José Pérez Carranza