Coverage: Difference between revisions

From Yocto Project
Jump to navigationJump to search
No edit summary
 
(8 intermediate revisions by 3 users not shown)
Line 1: Line 1:
=Code Coverage=
Gathering code coverage is an important metric to ponder when analyzing the effectiveness of testcases. It consists not only in retrieving which lines were run and which ones weren't on a certain test, but it can also contain relevant information such as branch or function coverage.
Gathering code coverage is an important metric to ponder when analyzing the effectiveness of testcases. It consists not only in retrieving which lines were run and which ones weren't on a certain test, but it can also contain relevant information such as branch or function coverage.


Line 38: Line 40:
====Combining data files====
====Combining data files====


Sometimes we have more than one data file, each one with coverage information from different scripts or testcases. If this is the case, we can combine them into a single data file with the '''coverage combine''' subcommand. This will merge the coverage information and will show us which code was run by at least one of the testcases.
Sometimes we have more than one data file, each one with coverage information from different scripts or testcases. If this is the case, we can combine them into a single data file with the '''coverage combine''' subcommand. This will merge the coverage information and will show us which code was ran by at least one of the testcases.


   $ '''coverage combine'''
   $ '''coverage combine'''
Line 79: Line 81:
   directory = coverage_html_report
   directory = coverage_html_report


A configuration file is also of great help when using '''subprocessing''', which is explained in the next section.
==Source code Coverage in the Yocto Project==
 
Source code coverage is done through the <code>oe-selftest</code>. To enable this feature, include the <code>--coverage</code> (this option must be present, even when filtering options are used) argument into the script:
 
  $ '''oe-selftest --run-tests''' ''bblayers.BitbakeLayers.test_bitbakelayers_flatten'' '''--coverage'''
 
This command will run the given testcase with coverage tracking, generating a datafile and a configuration file in the current folder. The data file name is generated with the current timestamp, while the configuration file is always named '''.coveragerc'''.
 
===Filtering===


===Subprocessing===
By default, source code coverage looks into these Poky directories:


'''Coverage.py''' has a feature to track subprocesses, this means that it can gather coverage data from commands that were not directly triggered from the command line. When this option is enabled, it will track every python process in the system, so it is important to specify the directories that are not required for the measurement.
* <code>bitbake</code>
* <code>scripts</code>
* meta layers specified in the <code>conf/bblayers.conf</code> file


To set the subprocess tracking there are a couple of things that need to be done. First, we need to export the <code>'''COVERAGE_PROCESS_START'''</code> variable, which will point to our configuration file. Also, the <code>sitecustomize.py</code> file located in out python folder needs to be modified to contain the following:
This is not always the required behavior so the following sections explain the filtering features that narrow the coverage.  


  import coverage
 
  coverage.process_startup()


==Coverage in the Yocto Project==
* Selecting source directories
 
The <code>--coverage-source</code> option lets the user specify source directories. This option overrides the default behavior mentioned above. For example, let's coverage just the directory <code>bitbake/bin</code>:
 
  $ '''oe-selftest --run-tests''' ''bblayers.BitbakeLayers.test_bitbakelayers_flatten'' '''--coverage''' '''--coverage-source''' ''bitbake/bin''


In the Yocto Project there are a large variety of python scripts that can be tracked with this tool. However, one of the best parts to work is in the <code>scripts/oe-selftest</code> script. The <code>oe-selftest</code> script groups testcases from the system in modules, allowing us to check the status of different parts of the Yocto Project.
* Including files using patterns


Another important advantage of working with <code>oe-selftest</code> script is that it isolates the changes away from the system's code, keeping the code to be measured unaltered.
The <code>--coverage-include</code> option specifies a pattern to filter files inside the given sources. Unlike the previous option (which is a folder or a list of folders), this option works with patterns (see [http://coverage.readthedocs.org/en/latest/source.html] to check supported patterns). The following example only covers those files that contain the word ''bitbake'':


The <code>oe-selftest</code> file has been modified to include coverage in the testcases that it runs. To use coverage in this command the following is required:
  $ '''oe-selftest --run-tests''' ''bblayers.BitbakeLayers.test_bitbakelayers_flatten'' '''--coverage''' '''--coverage-include''' ''"*bitbake*"''


# Install Coverage.py
* Omitting files using patterns
# Modify <code>sitecustomize.py</code> file to include subprocessing. (As mentioned in the previous section)
# Issue <code>oe-selftest</code> script with coverage enabled


For the last step, a ''--coverage'' option has been added to <code>oe-selftest</code>. Here is an example of how to execute this feature:
The <code>--coverage-omit</code> option behaves just like <code>--coverage-include</code>, but the behaviour is exactly the opposite: it omits files depending on the given pattern (see [http://coverage.readthedocs.org/en/latest/source.html] to check supported patterns). The following example omits coverage for <code>wic</code> related files:


   $ '''oe-selftest --run-tests''' ''bblayers.BitbakeLayers.test_bitbakelayers_flatten'' '''--coverage'''
   $ '''oe-selftest --run-tests''' ''bblayers.BitbakeLayers.test_bitbakelayers_flatten'' '''--coverage''' '''--coverage-omit''' ''"*wic*"''


This command will run the given testcase with coverage tracking, generating a datafile and a configuration file in the current folder. For the previous example, the datafile generated will be named '''.coverage.bblayers.BitbakeLayers.test_bitbakelayers_flatten''' and the configuration file will be '''.coveragerc'''.


It is important to notice that multiple filtering options can be provided in the same command allowing more complex scenarios.


==References==
==References==


<references />
<references />

Latest revision as of 22:13, 7 April 2016

Code Coverage

Gathering code coverage is an important metric to ponder when analyzing the effectiveness of testcases. It consists not only in retrieving which lines were run and which ones weren't on a certain test, but it can also contain relevant information such as branch or function coverage.

For python code, there is an utility developed by Ned Batchelder which can give us these metrics, Coverage.py.[1] This article is going to explain how to use this tool to gather meaningful data from the Yocto Project.[2]

Version 4.0.2 of Coverage.py[1] was used to write the following instructions, changes might need to be made if using a different version.

Installation

There are a few ways of installing coverage, which are explained at the tool's site. [3] The simplest way is using pip:

 $ pip install coverage

Usage

Executing from the Command Line

Gathering the coverage data

To get the coverage information from a specific script, we need to use the run subcommand. This is done by adding coverage run to the command we want to get the data from, all of the command's arguments remain unaltered:

 $ coverage run command.py arg1 arg2 

This will generate a .coverage file in the current directory with all the coverage data from the script. For a complete explanation of the coverage run arguments, please check the Coverage.py documentation.[1]

Reporting the coverage data

Once we have the .coverage file we can analyze all the information present there. A full report of that data can be obtained by running the coverage report subcommand in the same folder where the data file is:

 $ coverage report

All this data can also be shown in other formats, such as html or xml, for better easier analysis or parsing.

 $ coverage html
 $ coverage xml

These commands also require to be run in the same folder where the data file is. They can be used with a few options, which are detailed in the documentation of the tool also.

Combining data files

Sometimes we have more than one data file, each one with coverage information from different scripts or testcases. If this is the case, we can combine them into a single data file with the coverage combine subcommand. This will merge the coverage information and will show us which code was ran by at least one of the testcases.

 $ coverage combine

As well as the previous subcommands, this one also needs to be run in the same folder where the data files are. All data files to be combined need to be called .coverage.<testcase>, where <testcase> is any sufix given to the file; the resulting data file will be named .coverage and will be placed in the same folder.

IMPORTANT. When combining data files, the original files are lost in the process. If these are needed, they should be copied to a different folder before the combine.

Using a configuration file

Instead of giving all options to coverage through the command line, a configuration file can be built. Coverage.py looks for this file in the same directory where it runs, the default name of this configuration file is .coveragerc. This allows a great customization alternative, since it can be stored with all the desired parameters and coverage will use those parameters each time that runs.

There is a whole section in the coverage documentation given to the construction of a configuration file. Here is the sample file from the docs:

 # .coveragerc to control coverage.py
 [run]
 branch = True
 
 [report]
 # Regexes for lines to exclude from consideration
 exclude_lines =
     # Have to re-enable the standard pragma
     pragma: no cover
     
     # Don't complain about missing debug-only code:
     def __repr__
     if self\.debug
     
     # Don't complain if tests don't hit defensive assertion code:
     raise AssertionError
     raise NotImplementedError
     
     # Don't complain if non-runnable code isn't run:
     if 0:
     if __name__ == .__main__.:
 
 ignore_errors = True
 
 [html]
 directory = coverage_html_report

Source code Coverage in the Yocto Project

Source code coverage is done through the oe-selftest. To enable this feature, include the --coverage (this option must be present, even when filtering options are used) argument into the script:

 $ oe-selftest --run-tests bblayers.BitbakeLayers.test_bitbakelayers_flatten --coverage

This command will run the given testcase with coverage tracking, generating a datafile and a configuration file in the current folder. The data file name is generated with the current timestamp, while the configuration file is always named .coveragerc.

Filtering

By default, source code coverage looks into these Poky directories:

  • bitbake
  • scripts
  • meta layers specified in the conf/bblayers.conf file

This is not always the required behavior so the following sections explain the filtering features that narrow the coverage.


  • Selecting source directories

The --coverage-source option lets the user specify source directories. This option overrides the default behavior mentioned above. For example, let's coverage just the directory bitbake/bin:

 $ oe-selftest --run-tests bblayers.BitbakeLayers.test_bitbakelayers_flatten --coverage --coverage-source bitbake/bin
  • Including files using patterns

The --coverage-include option specifies a pattern to filter files inside the given sources. Unlike the previous option (which is a folder or a list of folders), this option works with patterns (see [1] to check supported patterns). The following example only covers those files that contain the word bitbake:

 $ oe-selftest --run-tests bblayers.BitbakeLayers.test_bitbakelayers_flatten --coverage --coverage-include "*bitbake*"
  • Omitting files using patterns

The --coverage-omit option behaves just like --coverage-include, but the behaviour is exactly the opposite: it omits files depending on the given pattern (see [2] to check supported patterns). The following example omits coverage for wic related files:

 $ oe-selftest --run-tests bblayers.BitbakeLayers.test_bitbakelayers_flatten --coverage --coverage-omit "*wic*"


It is important to notice that multiple filtering options can be provided in the same command allowing more complex scenarios.

References