Coverage - Revision history

Leonardo Sandoval: /* Coverage in the Yocto Project */

2016-04-07T22:13:27Z

Coverage in the Yocto Project

← Older revision		Revision as of 22:13, 7 April 2016
Line 81:		Line 81:
	directory = coverage_html_report		directory = coverage_html_report

	==Coverage in the Yocto Project==		==Source code Coverage in the Yocto Project==

	~~In the Yocto Project there are a large variety of python scripts that can be tracked with this tool. However, one of the best parts to work~~ is in the <code>~~scripts/~~oe-selftest</code> ~~script~~. ~~The <code>oe-selftest</code> script groups testcases from the system in modules~~, ~~allowing us to check~~ the ~~status of different parts of the Yocto Project.~~		Source code coverage is done through the <code>oe-selftest</code>. To enable this feature, include the <code>--coverage</code> (this option must be present, even when filtering options are used) argument into the script:

	~~Another important advantage of working with~~ <code>oe-~~selftest</code> script is that it isolates the changes away from the system's code, keeping the code to be measured unaltered.~~

	~~The <code>oe~~-~~selftest</code> file has been modified to include~~ coverage ~~in the testcases that it runs, it just requires to have Coverage.py installed.~~

	~~To issue <code>oe-selftest~~</code> ~~with coverage measurement the ''--coverage''~~ option is used~~.. Here is an example of how to execute this feature~~:

	$ '''oe-selftest --run-tests''' ''bblayers.BitbakeLayers.test_bitbakelayers_flatten'' '''--coverage'''		$ '''oe-selftest --run-tests''' ''bblayers.BitbakeLayers.test_bitbakelayers_flatten'' '''--coverage'''
Line 97:		Line 91:
	===Filtering===		===Filtering===

	~~The~~ coverage ~~measurement returns only~~ the ~~information from~~ the ~~files under~~ the following ~~directories:~~		By default, source code coverage looks into these Poky directories:

			* <code>bitbake</code>
			* <code>scripts</code>
			* meta layers specified in the <code>conf/bblayers.conf</code> file

			This is not always the required behavior so the following sections explain the filtering features that narrow the coverage.

	* {poky}/bitbake
	* {poky}/scripts
	* Layer folders specified in the <code>bblayers.conf</code> file

	~~This is not always the required behavior so three options have been added to <code>oe-selftest</code> to improve the filtering of the files reported.~~		* Selecting source directories

	The <code>--coverage-source</code> option ~~specifies from which~~ directories ~~to gather coverage data~~. ~~Giving this~~ option ~~will override~~ the default behavior mentioned above. ~~The following~~ example ~~only tracks~~ coverage ~~from~~ the bitbake/bin ~~folder~~:		The <code>--coverage-source</code> option lets the user specify source directories. This option overrides the default behavior mentioned above. For example, let's coverage just the directory <code>bitbake/bin</code>:

	$ '''oe-selftest --run-tests''' ''bblayers.BitbakeLayers.test_bitbakelayers_flatten'' '''--coverage''' '''--coverage-source''' ''bitbake/bin''		$ '''oe-selftest --run-tests''' ''bblayers.BitbakeLayers.test_bitbakelayers_flatten'' '''--coverage''' '''--coverage-source''' ''bitbake/bin''

	The <code>--coverage-include</code> option specifies a pattern to filter files inside the sources. Unlike the previous option (which is a folder or a list of folders), this option works with ~~selecting files based on~~ patterns~~, accepting wildcards~~. ~~Only shows the files that match the pattern and that are inside one of the folders specified in the~~ source ~~directories~~. The following example only ~~shows the~~ files that contain the ~~sequence~~ ''bitbake'' ~~and that are contained in the~~ bitbake ~~directory:~~		* Including files using patterns

			The <code>--coverage-include</code> option specifies a pattern to filter files inside the given sources. Unlike the previous option (which is a folder or a list of folders), this option works with patterns (see [http://coverage.readthedocs.org/en/latest/source.html] to check supported patterns). The following example only covers those files that contain the word ''bitbake'':

			$ '''oe-selftest --run-tests''' ''bblayers.BitbakeLayers.test_bitbakelayers_flatten'' '''--coverage''' '''--coverage-include''' ''"bitbake"''

			* Omitting files using patterns

	~~$ '''oe~~-~~selftest~~ --~~run-tests''' ''bblayers.BitbakeLayers.test_bitbakelayers_flatten'' '''~~--coverage~~''' '''-~~-coverage-source~~''' ''bitbake'' '''--~~coverage~~-include''' ''"bitbake"''~~		The <code>--coverage-omit</code> option behaves just like <code>--coverage-include</code>, but the behaviour is exactly the opposite: it omits files depending on the given pattern (see [http://coverage.readthedocs.org/en/latest/source.html] to check supported patterns). The following example omits coverage for <code>wic</code> related files:

	~~The <code>~~--coverage~~-omit</code> behaves just like <code>~~--coverage-~~include</code>, except that it specifies files not to be included in the report. The following example works similar to the last one, but it omits the files in~~ ''~~bitbake/lib/bb~~'':		$ '''oe-selftest --run-tests''' ''bblayers.BitbakeLayers.test_bitbakelayers_flatten'' '''--coverage''' '''--coverage-omit''' ''"wic"''

	$ '''oe-selftest --run-tests''' ''bblayers.BitbakeLayers.test_bitbakelayers_flatten'' '''--coverage''' '''--coverage-source''' ''bitbake'' '''--coverage-include''' ''"bitbake"'' '''--coverage-omit''' ''"/lib/bb/"''

			It is important to notice that multiple filtering options can be provided in the same command allowing more complex scenarios.

	==References==		==References==

	<references />		<references />

Humberto Ibarra at 19:57, 7 April 2016

2016-04-07T19:57:49Z

← Older revision		Revision as of 19:57, 7 April 2016
Line 40:		Line 40:
	====Combining data files====		====Combining data files====

	Sometimes we have more than one data file, each one with coverage information from different scripts or testcases. If this is the case, we can combine them into a single data file with the '''coverage combine''' subcommand. This will merge the coverage information and will show us which code was ~~run~~ by at least one of the testcases.		Sometimes we have more than one data file, each one with coverage information from different scripts or testcases. If this is the case, we can combine them into a single data file with the '''coverage combine''' subcommand. This will merge the coverage information and will show us which code was ran by at least one of the testcases.

	$ '''coverage combine'''		$ '''coverage combine'''
Line 80:		Line 80:
	[html]		[html]
	directory = coverage_html_report		directory = coverage_html_report

	~~A configuration file is also of great help when using '''subprocessing''', which is explained in the next section.~~

	==Coverage in the Yocto Project==		==Coverage in the Yocto Project==
Line 95:		Line 93:
	$ '''oe-selftest --run-tests''' ''bblayers.BitbakeLayers.test_bitbakelayers_flatten'' '''--coverage'''		$ '''oe-selftest --run-tests''' ''bblayers.BitbakeLayers.test_bitbakelayers_flatten'' '''--coverage'''

	This command will run the given testcase with coverage tracking, generating a datafile and a configuration file in the current folder. ~~For~~ the ~~previous example~~, ~~the datafile generated will be named '''.coverage.bblayers.BitbakeLayers.test_bitbakelayers_flatten''' and~~ the configuration file ~~will be~~ '''.coveragerc'''. ~~<ref name="subprocessing" >[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8930 Subprocessing Handling]</ref>~~		This command will run the given testcase with coverage tracking, generating a datafile and a configuration file in the current folder. The data file name is generated with the current timestamp, while the configuration file is always named '''.coveragerc'''.

	===Filtering===		===Filtering===
Line 105:		Line 103:
	* Layer folders specified in the <code>bblayers.conf</code> file		* Layer folders specified in the <code>bblayers.conf</code> file

	This is not always the required behavior so three options have been added to <code>oe-selftest</code> to improve the filtering of the files reported. ~~<ref name="filtering" >[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8920 Coverage Filtering]</ref>~~		This is not always the required behavior so three options have been added to <code>oe-selftest</code> to improve the filtering of the files reported.

	The <code>--coverage-source</code> option specifies from which directories to gather coverage data. Giving this option will override the default behavior mentioned above. The following example only tracks coverage from the bitbake/bin folder:		The <code>--coverage-source</code> option specifies from which directories to gather coverage data. Giving this option will override the default behavior mentioned above. The following example only tracks coverage from the bitbake/bin folder:

Juan Humberto Ibarra at 21:49, 18 February 2016

2016-02-18T21:49:37Z

← Older revision		Revision as of 21:49, 18 February 2016
Line 105:		Line 105:
	* Layer folders specified in the <code>bblayers.conf</code> file		* Layer folders specified in the <code>bblayers.conf</code> file

	This is not always the required behavior so three options have been added to <code>oe-selftest</code> to improve the filtering of the files reported. <ref name="~~subprocessing~~" >[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8920 Coverage Filtering]</ref>		This is not always the required behavior so three options have been added to <code>oe-selftest</code> to improve the filtering of the files reported. <ref name="filtering" >[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8920 Coverage Filtering]</ref>

	The <code>--coverage-source</code> option specifies from which directories to gather coverage data. Giving this option will override the default behavior mentioned above. The following example only tracks coverage from the bitbake/bin folder:		The <code>--coverage-source</code> option specifies from which directories to gather coverage data. Giving this option will override the default behavior mentioned above. The following example only tracks coverage from the bitbake/bin folder:

Juan Humberto Ibarra at 21:49, 18 February 2016

2016-02-18T21:49:06Z

← Older revision		Revision as of 21:49, 18 February 2016
Line 95:		Line 95:
	$ '''oe-selftest --run-tests''' ''bblayers.BitbakeLayers.test_bitbakelayers_flatten'' '''--coverage'''		$ '''oe-selftest --run-tests''' ''bblayers.BitbakeLayers.test_bitbakelayers_flatten'' '''--coverage'''

	This command will run the given testcase with coverage tracking, generating a datafile and a configuration file in the current folder. For the previous example, the datafile generated will be named '''.coverage.bblayers.BitbakeLayers.test_bitbakelayers_flatten''' and the configuration file will be '''.coveragerc'''.		This command will run the given testcase with coverage tracking, generating a datafile and a configuration file in the current folder. For the previous example, the datafile generated will be named '''.coverage.bblayers.BitbakeLayers.test_bitbakelayers_flatten''' and the configuration file will be '''.coveragerc'''. <ref name="subprocessing" >[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8930 Subprocessing Handling]</ref>

	===Filtering===		===Filtering===
Line 105:		Line 105:
	* Layer folders specified in the <code>bblayers.conf</code> file		* Layer folders specified in the <code>bblayers.conf</code> file

	This is not always the required behavior so three options have been added to <code>oe-selftest</code> to improve the filtering of the files reported.		This is not always the required behavior so three options have been added to <code>oe-selftest</code> to improve the filtering of the files reported. <ref name="subprocessing" >[https://bugzilla.yoctoproject.org/show_bug.cgi?id=8920 Coverage Filtering]</ref>

	The <code>--coverage-source</code> option specifies from which directories to gather coverage data. Giving this option will override the default behavior mentioned above. The following example only tracks coverage from the bitbake/bin folder:		The <code>--coverage-source</code> option specifies from which directories to gather coverage data. Giving this option will override the default behavior mentioned above. The following example only tracks coverage from the bitbake/bin folder:
Line 117:		Line 117:
	The <code>--coverage-omit</code> behaves just like <code>--coverage-include</code>, except that it specifies files not to be included in the report. The following example works similar to the last one, but it omits the files in ''bitbake/lib/bb'':		The <code>--coverage-omit</code> behaves just like <code>--coverage-include</code>, except that it specifies files not to be included in the report. The following example works similar to the last one, but it omits the files in ''bitbake/lib/bb'':

	$ '''oe-selftest --run-tests''' ''bblayers.BitbakeLayers.test_bitbakelayers_flatten'' '''--coverage''' '''--coverage-source''' ''bitbake'' '''--coverage-include''' ''"bitbake"'' '''--coverage-omit''' ''/lib/bb/"''		$ '''oe-selftest --run-tests''' ''bblayers.BitbakeLayers.test_bitbakelayers_flatten'' '''--coverage''' '''--coverage-source''' ''bitbake'' '''--coverage-include''' ''"bitbake"'' '''--coverage-omit''' ''"/lib/bb/"''

Juan Humberto Ibarra at 21:46, 18 February 2016

2016-02-18T21:46:17Z

← Older revision		Revision as of 21:46, 18 February 2016
Line 107:		Line 107:
	This is not always the required behavior so three options have been added to <code>oe-selftest</code> to improve the filtering of the files reported.		This is not always the required behavior so three options have been added to <code>oe-selftest</code> to improve the filtering of the files reported.

	The <code>--coverage-source</code> specifies from which directories to gather coverage data. Giving this option will override the default behavior mentioned above. The following example only tracks coverage from the bitbake/bin folder:		The <code>--coverage-source</code> option specifies from which directories to gather coverage data. Giving this option will override the default behavior mentioned above. The following example only tracks coverage from the bitbake/bin folder:

	$ '''oe-selftest --run-tests''' ''bblayers.BitbakeLayers.test_bitbakelayers_flatten'' '''--coverage''' '''--coverage-source''' ''bitbake/bin''		$ '''oe-selftest --run-tests''' ''bblayers.BitbakeLayers.test_bitbakelayers_flatten'' '''--coverage''' '''--coverage-source''' ''bitbake/bin''

	The <code>--coverage-include</code> specifies a pattern to filter files inside the sources. Unlike the previous option (which is a folder or a list of folders), this option works with selecting files based on patterns, accepting wildcards. Only shows the files that match the pattern and that are inside one of the folders specified in the source directories. The following example only shows the files that contain the sequence ''bitbake'' and that are contained in the bitbake directory:		The <code>--coverage-include</code> option specifies a pattern to filter files inside the sources. Unlike the previous option (which is a folder or a list of folders), this option works with selecting files based on patterns, accepting wildcards. Only shows the files that match the pattern and that are inside one of the folders specified in the source directories. The following example only shows the files that contain the sequence ''bitbake'' and that are contained in the bitbake directory:

	$ '''oe-selftest --run-tests''' ''bblayers.BitbakeLayers.test_bitbakelayers_flatten'' '''--coverage''' '''--coverage-source''' ''bitbake'' '''--coverage-include''' ''bitbake''		$ '''oe-selftest --run-tests''' ''bblayers.BitbakeLayers.test_bitbakelayers_flatten'' '''--coverage''' '''--coverage-source''' ''bitbake'' '''--coverage-include''' ''"bitbake"''

			The <code>--coverage-omit</code> behaves just like <code>--coverage-include</code>, except that it specifies files not to be included in the report. The following example works similar to the last one, but it omits the files in ''bitbake/lib/bb'':

			$ '''oe-selftest --run-tests''' ''bblayers.BitbakeLayers.test_bitbakelayers_flatten'' '''--coverage''' '''--coverage-source''' ''bitbake'' '''--coverage-include''' ''"bitbake"'' '''--coverage-omit''' ''/lib/bb/"''

Juan Humberto Ibarra at 21:38, 18 February 2016

2016-02-18T21:38:40Z

← Older revision		Revision as of 21:38, 18 February 2016
Line 96:		Line 96:

	This command will run the given testcase with coverage tracking, generating a datafile and a configuration file in the current folder. For the previous example, the datafile generated will be named '''.coverage.bblayers.BitbakeLayers.test_bitbakelayers_flatten''' and the configuration file will be '''.coveragerc'''.		This command will run the given testcase with coverage tracking, generating a datafile and a configuration file in the current folder. For the previous example, the datafile generated will be named '''.coverage.bblayers.BitbakeLayers.test_bitbakelayers_flatten''' and the configuration file will be '''.coveragerc'''.

			===Filtering===

			The coverage measurement returns only the information from the files under the following directories:

			* {poky}/bitbake
			* {poky}/scripts
			* Layer folders specified in the <code>bblayers.conf</code> file

			This is not always the required behavior so three options have been added to <code>oe-selftest</code> to improve the filtering of the files reported.

			The <code>--coverage-source</code> specifies from which directories to gather coverage data. Giving this option will override the default behavior mentioned above. The following example only tracks coverage from the bitbake/bin folder:

			$ '''oe-selftest --run-tests''' ''bblayers.BitbakeLayers.test_bitbakelayers_flatten'' '''--coverage''' '''--coverage-source''' ''bitbake/bin''

			The <code>--coverage-include</code> specifies a pattern to filter files inside the sources. Unlike the previous option (which is a folder or a list of folders), this option works with selecting files based on patterns, accepting wildcards. Only shows the files that match the pattern and that are inside one of the folders specified in the source directories. The following example only shows the files that contain the sequence ''bitbake'' and that are contained in the bitbake directory:

			$ '''oe-selftest --run-tests''' ''bblayers.BitbakeLayers.test_bitbakelayers_flatten'' '''--coverage''' '''--coverage-source''' ''bitbake'' '''--coverage-include''' ''bitbake''

Juan Humberto Ibarra at 21:24, 18 February 2016

2016-02-18T21:24:19Z

← Older revision		Revision as of 21:24, 18 February 2016
Line 82:		Line 82:

	A configuration file is also of great help when using '''subprocessing''', which is explained in the next section.		A configuration file is also of great help when using '''subprocessing''', which is explained in the next section.

	~~===Subprocessing===~~

	'''Coverage.py''' has a feature to track subprocesses, this means that it can gather coverage data from commands that were not directly triggered from the command line. When this option is enabled, it will track every python process in the system, so it is important to specify the directories that are not required for the measurement.

	To set the subprocess tracking there are a couple of things that need to be done. First, we need to export the <code>'''COVERAGE_PROCESS_START'''</code> variable, which will point to our configuration file. Also, the <code>sitecustomize.py</code> file located in out python folder needs to be modified to contain the following:

	~~import coverage~~

	~~coverage.process_startup()~~

	==Coverage in the Yocto Project==		==Coverage in the Yocto Project==
Line 99:		Line 89:
	Another important advantage of working with <code>oe-selftest</code> script is that it isolates the changes away from the system's code, keeping the code to be measured unaltered.		Another important advantage of working with <code>oe-selftest</code> script is that it isolates the changes away from the system's code, keeping the code to be measured unaltered.

	The <code>oe-selftest</code> file has been modified to include coverage in the testcases that it runs~~. To use coverage in this command the following is required:~~		The <code>oe-selftest</code> file has been modified to include coverage in the testcases that it runs, it just requires to have Coverage.py installed.

	~~# Install~~ Coverage.py
	~~# Modify <code>sitecustomize.py</code> file to include subprocessing~~. ~~(As mentioned in the previous section)~~
	~~# Issue <code>oe-selftest</code> script with coverage enabled~~

	~~For the last step, a ''--coverage'' option has been added to~~ <code>oe-selftest</code>. Here is an example of how to execute this feature:		To issue <code>oe-selftest</code> with coverage measurement the ''--coverage'' option is used.. Here is an example of how to execute this feature:

	$ '''oe-selftest --run-tests''' ''bblayers.BitbakeLayers.test_bitbakelayers_flatten'' '''--coverage'''		$ '''oe-selftest --run-tests''' ''bblayers.BitbakeLayers.test_bitbakelayers_flatten'' '''--coverage'''

Benjamin at 16:45, 15 January 2016

2016-01-15T16:45:19Z

← Older revision		Revision as of 16:45, 15 January 2016
Line 1:		Line 1:
			=Code Coverage=

	Gathering code coverage is an important metric to ponder when analyzing the effectiveness of testcases. It consists not only in retrieving which lines were run and which ones weren't on a certain test, but it can also contain relevant information such as branch or function coverage.		Gathering code coverage is an important metric to ponder when analyzing the effectiveness of testcases. It consists not only in retrieving which lines were run and which ones weren't on a certain test, but it can also contain relevant information such as branch or function coverage.

Juan Humberto Ibarra: Juan Humberto Ibarra moved page Code Coverage in the Yocto Project to Coverage: More condensed title

2016-01-14T15:37:12Z

Juan Humberto Ibarra moved page Code Coverage in the Yocto Project to Coverage: More condensed title

← Older revision	Revision as of 15:37, 14 January 2016
(No difference)

Juan Humberto Ibarra at 21:09, 13 January 2016

2016-01-13T21:09:29Z

← Older revision		Revision as of 21:09, 13 January 2016
Line 48:		Line 48:
	===Using a configuration file===		===Using a configuration file===

	Instead of giving all options to coverage through the command line, a configuration file can be built. '''Coverage.py''' looks for this file in the same directory where it runs, the default name of this configuration file is <code>.coveragerc</code>. This allows a great customization alternative, since it can be stored with all the desired parameters and coverage will use those ~~paramteres~~ each time that runs.		Instead of giving all options to coverage through the command line, a configuration file can be built. '''Coverage.py''' looks for this file in the same directory where it runs, the default name of this configuration file is <code>.coveragerc</code>. This allows a great customization alternative, since it can be stored with all the desired parameters and coverage will use those parameters each time that runs.

	There is a whole section in the coverage documentation given to the construction of a configuration file. Here is the sample file from the docs:		There is a whole section in the coverage documentation given to the construction of a configuration file. Here is the sample file from the docs:
Line 83:		Line 83:
	===Subprocessing===		===Subprocessing===

	'''Coverage.py''' has a feature to track subprocesses, this means that it can gather coverage data from ~~which~~ were not directly triggered from the command line. When this option is enabled, it will track every python process in the system, so it is important to specify the directories that are not required for the measurement.		'''Coverage.py''' has a feature to track subprocesses, this means that it can gather coverage data from commands that were not directly triggered from the command line. When this option is enabled, it will track every python process in the system, so it is important to specify the directories that are not required for the measurement.

	To set the subprocess tracking there are a couple of things that need to be done. First, we need to export the <code>'''COVERAGE_PROCESS_START'''</code> variable, which will point to our configuration file. Also, the <code>sitecustomize.py</code> file located in out python folder needs to be modified to contain the following:		To set the subprocess tracking there are a couple of things that need to be done. First, we need to export the <code>'''COVERAGE_PROCESS_START'''</code> variable, which will point to our configuration file. Also, the <code>sitecustomize.py</code> file located in out python folder needs to be modified to contain the following:
Line 97:		Line 97:
	Another important advantage of working with <code>oe-selftest</code> script is that it isolates the changes away from the system's code, keeping the code to be measured unaltered.		Another important advantage of working with <code>oe-selftest</code> script is that it isolates the changes away from the system's code, keeping the code to be measured unaltered.

	~~There is currently a branch in poky-contrib that has implemented '''Coverage.py''' inside oe-selftest. This branch can be checked out with the following command:~~		The <code>oe-selftest</code> file has been modified to include coverage in the testcases that it runs. To use coverage in this command the following is required:

	~~'''git -b''' lsandov1/coverage '''git://git.yoctoproject.org/poky-contrib'''~~

	~~In this branch, the~~ <code>oe-selftest</code> file has been modified to include coverage in the testcases that it runs. To use this ~~version of <code>oe-selftest</code>~~ the following is required:

	# Install Coverage.py		# Install Coverage.py
Line 107:		Line 103:
	# Issue <code>oe-selftest</code> script with coverage enabled		# Issue <code>oe-selftest</code> script with coverage enabled

	For the last step, a ''--coverage'' option ~~was~~ added to <code>oe-selftest</code> ~~in this branch~~. Here is an example of how to ~~implement~~ this feature:		For the last step, a ''--coverage'' option has been added to <code>oe-selftest</code>. Here is an example of how to execute this feature:

	$ '''oe-selftest --run-tests''' ''bblayers.BitbakeLayers.test_bitbakelayers_flatten'' '''--coverage'''		$ '''oe-selftest --run-tests''' ''bblayers.BitbakeLayers.test_bitbakelayers_flatten'' '''--coverage'''

	This command will run the given testcase with coverage tracking, generating a datafile in the current folder. For the previous example, the datafile generated will be named '''.coverage.bblayers.BitbakeLayers.test_bitbakelayers_flatten'''.		This command will run the given testcase with coverage tracking, generating a datafile and a configuration file in the current folder. For the previous example, the datafile generated will be named '''.coverage.bblayers.BitbakeLayers.test_bitbakelayers_flatten''' and the configuration file will be '''.coveragerc'''.