View source for Qa-tools

= Summary =

The qa-tools git repository is the recipient for the QA automation scripts and tools. Is intended to store any tool that helps on the QA activities related to the Yocto Project 


You can take a look at it via the [http://git.yoctoproject.org/cgit/cgit.cgi/qa-tools/ gitweb interface].

For cloning the qa-tools repository:

<pre>$ git clone git://git.yoctoproject.org/qa-tools</pre>

= List of Tools =

Below is a list with detailed information about the tools/scripts already implemented

== Testopia Update ==

=== Objective ===

This tool was designed to give an automated way to interact with Testopia activities.

=== Structure ===

This tool allows to create a test run based on [[Testopia#Test_Runs | Templates]]  already defined for every release, and based on that, the tool is able to create the appropriate test run by cloning the exiting template and generating a new Test Run that will show the results of the test execution. The Tool consist in a python script that can be used for different actions and with different parameters, with a configuration file to get connected to the Testopia DB and then using the Testopia API to perform different actions on the DB.

The structure is as follows:

 -> qa-tools/ 
   |-testopia_update.py: Main script 
   |-testopia_update.config : Configuration file with the URL and the login info to the Testopia DB
   |-> testopia_update/
      |- __init__.py
      |-> product/ : This directory contains all the available products used by QA on Testopia                
   |-> External/
      |- testopia.py: This file contains the API that provides methods to work on Testopia DB.

=== How to use it ===

1- Clone repo

2- Install Virtual Environment tool 
 $ sudo apt-get install python-virtualenv

3- Create virtual environment
 $ virtualenv venv

4- Activate virtual environment 
 $ source venv/bin/activate

5- Create configuration file, there is already a configuration file example that can be used

'''testopia_update.config.example [snapshot]'''
 [Testopia]
 url = <TESTOPIA URL> - https://bugzilla.yoctoproject.org/xmlrpc.cgi 
 username = <SOME-USER> - Bugzilla user
 password = <VERY-SECURE-PWD> - Bugzilla pswd
 store_location = /tmp/testopia_update

6- Install requirements 
 $ pip install -r requeriments.txt

7- Start using the '''testopia_update.py''' script

 ./testopia_update.py -h 


Below is a brief explanation of each one of the parameters available for the script:


'''-h, --help'''     
   
Show this help message and exit

'''--config ''' 

Configuration file with the updated data to get connected to the Testopia DB 

'''--url'''

URL of Testopia instance. This is used in case a configuration file is not being specified 

'''--username'''   

Username of Testopia instance. This is used in case a configuration file is not being specified

'''--password'''   

Username of Testopia instance. This is used in case a configuration file is not being specified

'''--list-products'''

List available products. This option retrieves the list of the available products that can be used on Testopia instance 

''' -a --action {create,update, setup}'''
                        
Action to execute can be create, update or setup. This should be combined with other option check [[#Examples | Example]] section to see how can be used

'''-p --product'''

Product to create or update. This is the name of the product available on Testopia. This should be one of the valid products retrieved when using option  --list-products

''' -c --category {AUTO,MANUAL} '''

Category for create or update. This represents the way how the Test Cases of the test run are being executed

'''-b --branch {master,jethro,dizzy,daisy,noexists}'''

Branch for create or update. Thi is the branch that is currently tested. Generally we used master branch as it represents the current release under test 

'''-e --environment'''

Environment for create or update.This is the environment where the test cases are being executed, could be a Distro OS, VM or real HW

'''-o --optional'''

Optional parameter for get Test runs. This is a string added to the name of the Test Run usually used on BSPs to specify the image where the test cases will be executed

'''--project-version '''

Version of the project. This is the release of the project that is being validated, to check current version go to [[Releases]]

'''--project-milestone '''

Milestone of the project. This represents the milestone of the project where Test Run will be applicable, it should be added also the release (i.e 2.3 M1)

'''--project-revision '''

SCM Revision of the project. This is a combination of the Release candidate and the Hash of that release candidate on Poky go to [[#Examples  | Examples]] to check how this is used

'''--project-date '''

SCM version/revision date of the project. The date where the release candidate outs at main auto builder, format is YYYY-MM-DD

'''--test-plan '''

Name of the test plan of the product, used when test plan name is different from product name. Normally  the test plans on Testopia are composed as "<PRDOUCT_NAME>: <BRANCH_NAME> branch" but sometimes the test plan name is not the same as the product name so the correct test plan name should be specified. 

'''--testrun-id  '''

Number of the test run to be updated, this option should be used along with update action. If you already know the ID of the test run to be updated this option should be used to avoid all other parameters to search for the test run. 

'''--results-log '''

Results log. This is the log generated by the execution were are reported the Test Cases IDs with the corresponding status (PASSED | FAILED)

'''--clone-templates '''

Clone templates from one branch to another or same branch (only available) with action=setup

'''--old-project-version'''

Version of the project to clone template from. This option is only used when action=setup and --clone-templates called

'''--create-wiki'''

Create a wiki page for the final report of the QA test cycle (only available with setup action)

'''--new-branch'''

New branch to clone the Templates. The branch name where the new templates will be cloned to. 

'''--verbose'''

Enable verbose mode.

'''--debug'''

Enable debug mode.

=== Examples ===

On all the examples add the '''--verbose variable''' to show details of the actions 

==== Create a Test Run ====

 '''./testopia_update.py '''
                         '''--config''' testopia_update.config
                         '''-a''' create 
                         '''-p''' ADT 
                         '''-c''' AUTO 
                         '''-b''' master 
                         '''-e''' "Ubuntu 16.04 x86_64"
                         '''--project-version''' 2.3 
                         '''--project-milestone''' "2.3 M2" 
                         '''--project-date''' 2017-03-09 
                         '''--project-revision''' "rc1-7e37e6178358c9868bf0a62419f0e4d6c693d531" 
                         '''--verbose'''

==== Update a Test Run ====

 '''./testopia_update.py '''
                         '''--config''' testopia_update.config
                         '''-a''' update
                         '''-p''' ADT 
                         '''-c''' AUTO 
                         '''-b''' master 
                         '''-e''' "Ubuntu 16.04 x86_64"
                         '''--project-version''' 2.3 
                         '''--project-milestone''' "2.3 M2" 
                         '''--project-date''' 2017-03-09 
                         '''--project-revision''' "rc1-7e37e6178358c9868bf0a62419f0e4d6c693d531" 
                         '''--results-log''' results.log
                         '''--verbose'''

==== Update a Test Run by ID ====

 '''./testopia_update.py '''
                         '''--config''' testopia_update.config
                         '''-a''' update
                         '''--results-log'''  results.log
                         '''--testrun-id''' 7049

==== Clone Templates on the same branch ====
 '''./testopia_update.py ''' 
                         '''--config''' testopia_update.config
                         '''-a''' setup
                         '''-b''' master 
                         '''--clone-templates'''  
                         '''--project-version''' 2.4 
                         '''--old-project-version''' 2.3 
                         '''--product''' Kernel

==== Clone Templates from one branch to another ====
 '''./testopia_update.py ''' 
                         '''--config''' testopia_update.config
                         '''-a''' setup 
                         '''-b''' master 
                         '''--clone-templates'''  
                         '''--project-version''' 2.4 
                         '''--old-project-version''' 2.3 
                         '''--product''' Kernel
                         '''--new-branch''' pyro

==== Create wiki for Milestone report ====

This will create a wiki.page file with the content that should be copy and pasted on the new wiki created for the milestone QA cycle report 

 '''./testopia_update.py ''' 
                         '''--config''' testopia_update.config
                         '''-a''' setup 
                         '''--create-wiki'''  
                         '''--project-date''' 2017-06-23
                         '''--project-version''' 2.3

=== Improvements ===

* Add option to create templates by adding defined TC's
* Create a log of the tasks executed for future reference

== Full Cycle Wrapper ==

=== Objective ===

This script was designed to help in the creation of a full test QA cycle for every release of Yocto Project

=== Structure ===

This script is a wrapper that creates a full test cycle for QA team, before all defined test runs were created manually and sometimes the correct templates or format was not used, with this script all the test runs for a full test cycle will be created. 

 -> qa-tools/scripts
           |-  full-test-cycle-wrapper.sh

This scrips calls the [Update Testopia] tool with already defined parameters and is capable to create all the Test Runs for Normal YOCTO cycle and also for META-INTEL cycle.


=== How To use it ===

1. Clone repo 

2. Execute the script with required parameters 

 ./full-test-cycle-wrapper.sh


Below is a brief explanation of the parameters used on the script:

'''-h --help'''

Show help message and exit.

''' -t {YOCTO,META-INTEL} '''

Type of test cycle, could be YOCTO or META-INTEL. The cycle that is wanted to be created YOCTO for a full test cycle and META-INTEL for only test against products with Intel architecture 

'''-b  {master,jethro,dizzy,daisy,noexists}'''

Branch for create or update. This is the branch that is currently tested. Generally we used master branch as it represents the current release under test

'''--project-version'''

Version of the project. This is the release of the project that is being validated, to check current version go to [[Releases]]

'''--project-milestone'''
Milestone of the project. This represents the milestone of the project where Test Run will be applicable, in this case only the milestone should be set (M1, M2. M3) 

'''-r  {XXXXXX} '''
                   
SCM Revision of the project, commit of branch tested.

'''-c {rc1 rc2 rcX}'''

Release candidate to be used.

'''--project-date PROJECT_DATE'''

SCM version/revision date of the project. The date where the release candidate outs at main auto builder, format is YYYY-MM-DD

=== Examples ===

==== Create Cycle for YOCTO ====

 '''./full-test-cycle-wrapper.sh'''
                                '''-t''' YOCTO 
                                '''-b''' master 
                                '''-p''' 2.3  
                                '''-m''' M1 
                                '''-r''' fabf5d7597955ec664c232ab6597f6dd8ed0ade3
                                '''-c''' rc1 
                                '''-d''' 2017-03-00


==== Create Cycle for META-INTEL ====

 '''./full-test-cycle-wrapper.sh'''
                                '''-t''' META-INTEL 
                                '''-b''' master 
                                '''-p''' 2.3  
                                '''-m''' M1 
                                '''-r''' fabf5d7597955ec664c232ab6597f6dd8ed0ade3
                                '''-c''' rc1 
                                '''-d''' 2017-03-00

=== Improvements ===

* Add option to create Test Runs per single component 
* Create configuration file where components are defined instead of hard coded on the script

== Ptest Parser ==

=== Objective ===

This script is designed to provide a report of ptest execution compared with previous results provided inf a format of wikitable with graphical indication of increase, decrease pass rate on components, as well as, indication of components that are not being executed during the tests, all this data is stored in a file named <release_under_test>-wiki.page

=== Structure ===

This script is a parser written in python that receives 2 logs as parameters as well as the release and commit of each log, the script parse the logs and generates a table with compared results, this table is stored in a file on wikimedia format to be copied to the final wiki page created.  

 -> qa-tools/scripts
           |-  ptest-parser.py

This scrips calls the [Update Testopia] tool with already defined parameters and is capable to create all the Test Runs for Normal YOCTO cycle and also for META-INTEL cycle.


=== How To use it ===

1. Clone repo 

2. Execute the script with required parameters 

 ./ptest-parser.py

Below is a description of the parameters used on the script:

'''optional arguments'''

'''-h, --help'''

show this help message and exit

'''required arguments:'''

'''-p PRE_LOG'''

prvious log of the pTest result

'''--pre_commit PRE_COMMIT'''

commit of the previous log.

'''--pre_release PRE_RELEASE'''

release of the previous log.

'''-c CURR_LOG'''

current log of pTets results.

'''--curr_commit CURR_COMMIT'''

commit of the current log.

'''--curr_release CURR_RELEASE'''

release of the current log.

=== Example ===
 '''./ptest-parser.py'''
                     '''-p''' Ptest2.4_m3_rc1.txt 
                     '''-c''' ptest.log 
                     '''--pre_commit''' 5f6945f5031e1a4ca116cc1eccf4c2f9dc228547 
                     '''--pre_release''' 2.4_M3_rc1 
                     '''--curr_commit''' 592c3449697ce39bf07864fafd7f0a61ca230561 
                     '''--curr_release''' 2.4_M4_rc1

Here is an example of one the reports generated [[ Ptest 592c3449697ce39bf07864fafd7f0a61ca230561 ]]

=== Example ===

= Proposed tools =

This section is intended to add new tools to be implemented

* Create a tool to compare the automated test list against the Testopia Test list DB [https://bugzilla.yoctoproject.org/show_bug.cgi?id=10147  Bug 10147]

= How To Contribute =

Contributing to any of the yoctoproject.org repositories, including qa-tools, can be achieved with the [[Poky Contributions]] guidelines.

Mantainers
[mailto:anibal.limon@linux.intel.com Anibal Limon]
[mailto:jose.perez.carranza@linux.intel.com José Pérez Carranza]