Coverage

2016-04-07T19:57:49Z

Humberto Ibarra:

=Code Coverage=

Gathering code coverage is an important metric to ponder when analyzing the effectiveness of testcases. It consists not only in retrieving which lines were run and which ones weren't on a certain test, but it can also contain relevant information such as branch or function coverage.

For python code, there is an utility developed by Ned Batchelder which can give us these metrics, Coverage.py.<ref name="coverage" >[http://coverage.readthedocs.org/en/latest/ Coverage.py]</ref> This article is going to explain how to use this tool to gather meaningful data from the Yocto Project.<ref name="yocto" >[http://www.yoctoproject.org/ Yocto Project]</ref>

Version 4.0.2 of Coverage.py<ref name="coverage"></ref> was used to write the following instructions, changes might need to be made if using a different version.

==Installation==

There are a few ways of installing coverage, which are explained at the tool's site. <ref name="install">[http://coverage.readthedocs.org/en/latest/install.html Coverage.py Installation]</ref> The simplest way is using pip:

$ pip install coverage

==Usage==

===Executing from the Command Line===

====Gathering the coverage data====

To get the coverage information from a specific script, we need to use the '''run''' subcommand. This is done by adding '''coverage run''' to the command we want to get the data from, all of the command's arguments remain unaltered:

$ '''coverage run''' ''command.py arg1 arg2''

This will generate a <code>.coverage</code> file in the current directory with all the coverage data from the script. For a complete explanation of the '''coverage run''' arguments, please check the Coverage.py documentation.<ref name="coverage"></ref>

====Reporting the coverage data====

Once we have the <code>.coverage</code> file we can analyze all the information present there. A full report of that data can be obtained by running the '''coverage report''' subcommand in the same folder where the data file is:

$ '''coverage report'''

All this data can also be shown in other formats, such as html or xml, for better easier analysis or parsing.

$ '''coverage html'''
$ '''coverage xml'''

These commands also require to be run in the same folder where the data file is. They can be used with a few options, which are detailed in the documentation of the tool also.

====Combining data files====

Sometimes we have more than one data file, each one with coverage information from different scripts or testcases. If this is the case, we can combine them into a single data file with the '''coverage combine''' subcommand. This will merge the coverage information and will show us which code was ran by at least one of the testcases.

$ '''coverage combine'''

As well as the previous subcommands, this one also needs to be run in the same folder where the data files are. All data files to be combined need to be called <code>.coverage.''<testcase>''</code>, where ''<testcase>'' is any sufix given to the file; the resulting data file will be named <code>.coverage</code> and will be placed in the same folder.

'''IMPORTANT.''' When combining data files, the original files are lost in the process. If these are needed, they should be copied to a different folder before the '''combine'''.

===Using a configuration file===

Instead of giving all options to coverage through the command line, a configuration file can be built. '''Coverage.py''' looks for this file in the same directory where it runs, the default name of this configuration file is <code>.coveragerc</code>. This allows a great customization alternative, since it can be stored with all the desired parameters and coverage will use those parameters each time that runs.

There is a whole section in the coverage documentation given to the construction of a configuration file. Here is the sample file from the docs:

# .coveragerc to control coverage.py
[run]
branch = True

[report]
# Regexes for lines to exclude from consideration
exclude_lines =
# Have to re-enable the standard pragma
pragma: no cover

# Don't complain about missing debug-only code:
def __repr__
if self\.debug

# Don't complain if tests don't hit defensive assertion code:
raise AssertionError
raise NotImplementedError

# Don't complain if non-runnable code isn't run:
if 0:
if __name__ == .__main__.:

ignore_errors = True

[html]
directory = coverage_html_report

==Coverage in the Yocto Project==

In the Yocto Project there are a large variety of python scripts that can be tracked with this tool. However, one of the best parts to work is in the <code>scripts/oe-selftest</code> script. The <code>oe-selftest</code> script groups testcases from the system in modules, allowing us to check the status of different parts of the Yocto Project.

Another important advantage of working with <code>oe-selftest</code> script is that it isolates the changes away from the system's code, keeping the code to be measured unaltered.

The <code>oe-selftest</code> file has been modified to include coverage in the testcases that it runs, it just requires to have Coverage.py installed.

To issue <code>oe-selftest</code> with coverage measurement the ''--coverage'' option is used.. Here is an example of how to execute this feature:

$ '''oe-selftest --run-tests''' ''bblayers.BitbakeLayers.test_bitbakelayers_flatten'' '''--coverage'''

This command will run the given testcase with coverage tracking, generating a datafile and a configuration file in the current folder. The data file name is generated with the current timestamp, while the configuration file is always named '''.coveragerc'''.

===Filtering===

The coverage measurement returns only the information from the files under the following directories:

* {poky}/bitbake
* {poky}/scripts
* Layer folders specified in the <code>bblayers.conf</code> file

This is not always the required behavior so three options have been added to <code>oe-selftest</code> to improve the filtering of the files reported.

The <code>--coverage-source</code> option specifies from which directories to gather coverage data. Giving this option will override the default behavior mentioned above. The following example only tracks coverage from the bitbake/bin folder:

$ '''oe-selftest --run-tests''' ''bblayers.BitbakeLayers.test_bitbakelayers_flatten'' '''--coverage''' '''--coverage-source''' ''bitbake/bin''

The <code>--coverage-include</code> option specifies a pattern to filter files inside the sources. Unlike the previous option (which is a folder or a list of folders), this option works with selecting files based on patterns, accepting wildcards. Only shows the files that match the pattern and that are inside one of the folders specified in the source directories. The following example only shows the files that contain the sequence ''bitbake'' and that are contained in the bitbake directory:

$ '''oe-selftest --run-tests''' ''bblayers.BitbakeLayers.test_bitbakelayers_flatten'' '''--coverage''' '''--coverage-source''' ''bitbake'' '''--coverage-include''' ''"*bitbake*"''

The <code>--coverage-omit</code> behaves just like <code>--coverage-include</code>, except that it specifies files not to be included in the report. The following example works similar to the last one, but it omits the files in ''bitbake/lib/bb'':

$ '''oe-selftest --run-tests''' ''bblayers.BitbakeLayers.test_bitbakelayers_flatten'' '''--coverage''' '''--coverage-source''' ''bitbake'' '''--coverage-include''' ''"*bitbake*"'' '''--coverage-omit''' ''"*/lib/bb/*"''

==References==

<references />

Yocto Project - User contributions [en]

Coverage