Handling optional module packaging - Revision history

PaulEggleton: Content moved to the dev manual

2013-05-29T13:07:07Z

Content moved to the dev manual

← Older revision		Revision as of 13:07, 29 May 2013
Line 1:		Line 1:
	~~Many pieces of software split functionality out into optional modules (or plugins), and which of these plugins are built may depend on configuration options~~. ~~To avoid having to duplicate~~ the ~~logic for determining which modules are available in your recipe or packaging each module by hand, the build system provides functionality to handle module packaging dynamically.~~		Content previously here has been moved. Please see the [http://www.yoctoproject.org/docs/current/dev-manual/dev-manual.html#handling-optional-module-packaging "Handling optional module packaging"] section of the Yocto Project Development Manual.

	~~== Packaging ==~~

	~~Firstly, you need to ensure that the module packaging actually gets done. This is accomplished using the <tt>do_split_packages<~~/~~tt> function within the <tt>populate_packages<~~/tt> python function in your recipe; this searches for a pattern of files or directories under a specified path and creates a package for each one it finds, by appending to the PACKAGES variable and setting the appropriate values for FILES_packagename, RDEPENDS_packagename, DESCRIPTION_packagename etc. Here is an example from the recipe for lighttpd:

	~~<nowiki>~~
	~~python populate_packages_prepend () {~~
	~~lighttpd_libdir = d.expand('${libdir}')~~
	~~do_split_packages(d, lighttpd_libdir, '^mod_(.*)\.so$', 'lighttpd-module-%s', 'Lighttpd module for %s', extra_depends='')~~
	~~}</nowiki>~~

	In the above example a number of things are specified in the call to do_split_packages: a directory in which to search (within the files installed by your recipe via do_install), a regular expression to match module files in that directory - note the parentheses () which mark the part of the expression from which the module name should be derived; a pattern to use for the package names, a description for each package, and an empty string for extra_depends which disables the default dependency on the main lighttpd package. ~~Thus, if a file in ${libdir} called mod_alias~~.~~so is found, a package called lighttpd-module-alias will be created for it, with DESCRIPTION set to "Lighttpd module for alias".~~

	Often packaging modules is as simple as above; however, there are more advanced options you can supply to do_split_packages to modify its behaviour, and if you need to you can add more logic by specifying a hook function to be called for each package. It is also perfectly acceptable to call do_split_packages multiple times if you have more than one set of modules to package.

	~~For some other examples of do_split_packages usage, see meta~~/~~recipes-connectivity~~/~~connman/connman.inc and meta/classes/kernel.bbclass.~~

	~~=== do_split_packages reference ===~~

	~~==== Mandatory arguments ====~~
	~~; root : the path in which to search~~
	; file_regex : regular expression to match searched files. Use parentheses () to mark the part of this expression that should be used to derive the module name (to be substituted where %s is used in other function arguments as noted below)
	~~; output_pattern : pattern to use for the package names. Must include %s.~~
	~~; description : description to set for each package. Must include %s.~~

	~~==== Optional arguments ====~~
	~~; postinst : postinstall script to use for all packages (as a string)~~
	~~; recursive : True to perform a recursive search - default False~~
	~~; hook : a hook function to be called for every match. The function will be called with the following arguments (in the order listed):~~
	~~:; f: full path to the file~~/~~directory match~~
	~~:; pkg: the package name~~
	~~:; file_regex: as above~~
	~~:; output_pattern: as above~~
	~~:; modulename: the module name derived using file_regex~~
	~~; extra_depends : extra runtime dependencies (RDEPENDS) to be set for all packages. The default value of None causes a dependency on the main package (${PN})~~ - ~~if you do not want this, pass empty string <nowiki>''<~~/~~nowiki> for this parameter.~~
	~~; aux_files_pattern : extra item(s) to be added to FILES for each package. Can be a single string item or a list of strings for multiple items. Must include %s.~~
	~~; postrm : postrm script to use for all packages (as a string)~~
	~~; allow_dirs : True to allow directories to be matched~~ - ~~default False~~
	~~; prepend : if True, prepend created packages to PACKAGES instead of the default False which appends them~~
	~~; match_path : match file_regex on the whole relative path to the root rather than just the file name~~
	~~; aux_files_pattern_verbatim : extra item(s) to be added to FILES for each package, using the actual derived module name rather than converting it to something legal for a package name~~. ~~Can be a single string item or a list of strings for multiple items. Must include %s.~~
	~~; allow_links : True to allow symlinks to be matched~~ - ~~default False~~

	~~== Satisfying dependencies ==~~

	~~The second step is to ensure that any dependencies from other recipes on~~ optional ~~modules are satisfied by your recipe. To do this, use PACKAGES_DYNAMIC. Here is an example, again from the lighttpd recipe:~~

	~~PACKAGES_DYNAMIC = "lighttpd~~-module-.*"

	~~The name here can of course be anything - here "lighttpd-~~module-" is specified as the prefix, and the <code>.*</code> ensures that any RDEPENDS / RRECOMMENDS on a package name starting with that prefix will be satisfied, at least at build time (PACKAGES_DYNAMIC is a regular expression). If you are using do_split_packages as described above, the value you put in PACKAGES_DYNAMIC should correspond to the name pattern specified in the call to do_split_packages.

	~~''Note:'' PACKAGES_DYNAMIC is simply a promise in order to satisfy dependencies, it does not actually do anything else by itself. If an RDEPENDS relationship (a hard runtime dependency)~~ of another package is satisfied at build time via PACKAGES_DYNAMIC, but a package with that name is never actually produced, then the other package will be broken, and if you attempt to include it in an image you will get a dependency failure from the packaging system during do_rootfs. Typically, if there is a chance that such a situation will arise, and the package not being created is valid and not cause for an error, then RRECOMMENDS (a soft runtime dependency) would be used instead of RDEPENDS.

PaulEggleton: Clarify that PACKAGES_DYNAMIC is a regex

2012-11-12T15:52:14Z

Clarify that PACKAGES_DYNAMIC is a regex

← Older revision		Revision as of 15:52, 12 November 2012
Line 47:		Line 47:
	The second step is to ensure that any dependencies from other recipes on optional modules are satisfied by your recipe. To do this, use PACKAGES_DYNAMIC. Here is an example, again from the lighttpd recipe:		The second step is to ensure that any dependencies from other recipes on optional modules are satisfied by your recipe. To do this, use PACKAGES_DYNAMIC. Here is an example, again from the lighttpd recipe:

	PACKAGES_DYNAMIC = "lighttpd-module-*"		PACKAGES_DYNAMIC = "lighttpd-module-.*"

	The name here can of course be anything - here "lighttpd-module-" is specified as the prefix, and the ~~wildcard~~ * ensures that any RDEPENDS / RRECOMMENDS on a package name starting with that prefix will be satisfied, at least at build time. If you are using do_split_packages as described above, the value you put in PACKAGES_DYNAMIC should correspond to the name pattern specified in the call to do_split_packages.		The name here can of course be anything - here "lighttpd-module-" is specified as the prefix, and the <code>.*</code> ensures that any RDEPENDS / RRECOMMENDS on a package name starting with that prefix will be satisfied, at least at build time (PACKAGES_DYNAMIC is a regular expression). If you are using do_split_packages as described above, the value you put in PACKAGES_DYNAMIC should correspond to the name pattern specified in the call to do_split_packages.

	''Note:'' PACKAGES_DYNAMIC is simply a promise in order to satisfy dependencies, it does not actually do anything else by itself. If an RDEPENDS relationship (a hard runtime dependency) of another package is satisfied at build time via PACKAGES_DYNAMIC, but a package with that name is never actually produced, then the other package will be broken, and if you attempt to include it in an image you will get a dependency failure from the packaging system during do_rootfs. Typically, if there is a chance that such a situation will arise, and the package not being created is valid and not cause for an error, then RRECOMMENDS (a soft runtime dependency) would be used instead of RDEPENDS.		''Note:'' PACKAGES_DYNAMIC is simply a promise in order to satisfy dependencies, it does not actually do anything else by itself. If an RDEPENDS relationship (a hard runtime dependency) of another package is satisfied at build time via PACKAGES_DYNAMIC, but a package with that name is never actually produced, then the other package will be broken, and if you attempt to include it in an image you will get a dependency failure from the packaging system during do_rootfs. Typically, if there is a chance that such a situation will arise, and the package not being created is valid and not cause for an error, then RRECOMMENDS (a soft runtime dependency) would be used instead of RDEPENDS.

PaulEggleton at 13:06, 24 August 2012

2012-08-24T13:06:46Z

← Older revision		Revision as of 13:06, 24 August 2012
Line 3:		Line 3:
	== Packaging ==		== Packaging ==

	Firstly, you need to ensure that the module packaging actually gets done. This is accomplished using the <tt>do_split_packages</tt> function within the <tt>~~populate_packages_prepend~~</tt> python function in your recipe; this searches for a pattern of files or directories under a specified path and creates a package for each one it finds, by appending to the PACKAGES variable and setting the appropriate values for FILES_packagename, RDEPENDS_packagename, DESCRIPTION_packagename etc. Here is an example from the recipe for lighttpd:		Firstly, you need to ensure that the module packaging actually gets done. This is accomplished using the <tt>do_split_packages</tt> function within the <tt>populate_packages</tt> python function in your recipe; this searches for a pattern of files or directories under a specified path and creates a package for each one it finds, by appending to the PACKAGES variable and setting the appropriate values for FILES_packagename, RDEPENDS_packagename, DESCRIPTION_packagename etc. Here is an example from the recipe for lighttpd:

	<nowiki>		<nowiki>
	python populate_packages_prepend () {		python populate_packages_prepend () {
	lighttpd_libdir = d.expand('${libdir}')		lighttpd_libdir = d.expand('${libdir}')
	do_split_packages(d, lighttpd_libdir, '^mod_(.*)\.so$', 'lighttpd-module-%s', 'Lighttpd module for %s', extra_depends='')		do_split_packages(d, lighttpd_libdir, '^mod_(.*)\.so$', 'lighttpd-module-%s', 'Lighttpd module for %s', extra_depends='')
	}		}</nowiki>
	</nowiki>

	In the above example a number of things are specified in the call to do_split_packages: a directory in which to search (within the files installed by your recipe via do_install), a regular expression to match module files in that directory - note the parentheses () which mark the part of the expression from which the module name should be derived; a pattern to use for the package names, a description for each package, and an empty string for extra_depends which disables the default dependency on the main lighttpd package. Thus, if a file in ${libdir} called mod_alias.so is found, a package called lighttpd-module-alias will be created for it, with DESCRIPTION set to "Lighttpd module for alias".		In the above example a number of things are specified in the call to do_split_packages: a directory in which to search (within the files installed by your recipe via do_install), a regular expression to match module files in that directory - note the parentheses () which mark the part of the expression from which the module name should be derived; a pattern to use for the package names, a description for each package, and an empty string for extra_depends which disables the default dependency on the main lighttpd package. Thus, if a file in ${libdir} called mod_alias.so is found, a package called lighttpd-module-alias will be created for it, with DESCRIPTION set to "Lighttpd module for alias".

PaulEggleton at 12:12, 22 August 2012

2012-08-22T12:12:22Z

PaulEggleton at 11:08, 22 August 2012

2012-08-22T11:08:56Z

← Older revision		Revision as of 11:08, 22 August 2012
Line 27:		Line 27:
	; postinst : postinstall script to use for all packages (as a string)		; postinst : postinstall script to use for all packages (as a string)
	; recursive : True to perform a recursive search - default False		; recursive : True to perform a recursive search - default False
	; hook : a hook function to be called for ~~each~~ package		; hook : a hook function to be called for every match. The function will be called with the following arguments (in the order listed):
			:; f: full path to the file/directory match
			:; pkg: the package name
			:; file_regex: as above
			:; output_pattern: as above
			:; modulename: the module name derived using file_regex
	; extra_depends : extra runtime dependencies (RDEPENDS) to be set for all packages. The default value of None causes a dependency on the main package (${PN}) - if you do not want this, pass '' for this parameter.		; extra_depends : extra runtime dependencies (RDEPENDS) to be set for all packages. The default value of None causes a dependency on the main package (${PN}) - if you do not want this, pass '' for this parameter.
	; aux_files_pattern : extra item(s) to be added to FILES for each package. Can be a single string item or a list of strings for multiple items. Must include %s.		; aux_files_pattern : extra item(s) to be added to FILES for each package. Can be a single string item or a list of strings for multiple items. Must include %s.

PaulEggleton at 10:56, 22 August 2012

2012-08-22T10:56:42Z

← Older revision		Revision as of 10:56, 22 August 2012
Line 13:		Line 13:

	Often packaging modules is as simple as above; however, there are more advanced options you can supply to do_split_packages to modify its behaviour, and if you need to you can add more logic by specifying a hook function to be called for each package. It is also perfectly acceptable to call do_split_packages multiple times if you have more than one set of modules to package.		Often packaging modules is as simple as above; however, there are more advanced options you can supply to do_split_packages to modify its behaviour, and if you need to you can add more logic by specifying a hook function to be called for each package. It is also perfectly acceptable to call do_split_packages multiple times if you have more than one set of modules to package.

			For some other examples of do_split_packages usage, see meta/recipes-connectivity/connman/connman.inc and meta/classes/kernel.bbclass.

	=== do_split_packages reference ===		=== do_split_packages reference ===

PaulEggleton at 10:38, 22 August 2012

2012-08-22T10:38:50Z

← Older revision		Revision as of 10:38, 22 August 2012
Line 24:		Line 24:
	==== Optional arguments ====		==== Optional arguments ====
	; postinst : postinstall script to use for all packages (as a string)		; postinst : postinstall script to use for all packages (as a string)
	; recursive : perform a recursive search		; recursive : True to perform a recursive search - default False
	; hook : a hook function to be called for each package		; hook : a hook function to be called for each package
	; extra_depends : extra runtime dependencies (RDEPENDS) to be set for all packages. The default value of None causes a dependency on the main package (${PN}) - if you do not want this, pass '' for this parameter.		; extra_depends : extra runtime dependencies (RDEPENDS) to be set for all packages. The default value of None causes a dependency on the main package (${PN}) - if you do not want this, pass '' for this parameter.
	; aux_files_pattern : extra item(s) to be added to FILES for each package. Can be a single string item or a list of strings for multiple items. Must include %s.		; aux_files_pattern : extra item(s) to be added to FILES for each package. Can be a single string item or a list of strings for multiple items. Must include %s.
	; postrm : postrm script to use for all packages (as a string)		; postrm : postrm script to use for all packages (as a string)
	; allow_dirs : allow directories to be matched		; allow_dirs : True to allow directories to be matched - default False
	; prepend : if True, prepend created packages to PACKAGES instead of ~~appending~~ them		; prepend : if True, prepend created packages to PACKAGES instead of the default False which appends them
	; match_path : match file_regex on the whole relative path to the root rather than just the file name		; match_path : match file_regex on the whole relative path to the root rather than just the file name
	; aux_files_pattern_verbatim : extra item(s) to be added to FILES for each package, using the actual derived module name rather than converting it to something legal for a package name. Can be a single string item or a list of strings for multiple items. Must include %s.		; aux_files_pattern_verbatim : extra item(s) to be added to FILES for each package, using the actual derived module name rather than converting it to something legal for a package name. Can be a single string item or a list of strings for multiple items. Must include %s.
	; allow_links : allow symlinks to be matched		; allow_links : True to allow symlinks to be matched - default False

	== Satisfying dependencies ==		== Satisfying dependencies ==

PaulEggleton at 10:33, 22 August 2012

2012-08-22T10:33:46Z

← Older revision		Revision as of 10:33, 22 August 2012
Line 16:		Line 16:
	=== do_split_packages reference ===		=== do_split_packages reference ===

	~~<nowiki>~~		==== Mandatory arguments ====
	~~Arguments~~:		; root : the path in which to search
	~~root --~~ the path in which to search		; file_regex : regular expression to match searched files. Use parentheses () to mark the part of this expression that should be used to derive the module name (to be substituted where %s is used in other function arguments as noted below)
	file_regex -- regular expression to match searched files. Use		; output_pattern : pattern to use for the package names. Must include %s.
	parentheses () to mark the part of this expression		; description : description to set for each package. Must include %s.
	that should be used to derive the module name (to be
	substituted where %s is used in other function		==== Optional arguments ====
	arguments as noted below)		; postinst : postinstall script to use for all packages (as a string)
	output_pattern -- pattern to use for the package names. Must include %s.		; recursive : perform a recursive search
	description -- description to set for each package. Must include %s.		; hook : a hook function to be called for each package
	postinst -- postinstall script to use for all packages (as a		; extra_depends : extra runtime dependencies (RDEPENDS) to be set for all packages. The default value of None causes a dependency on the main package (${PN}) - if you do not want this, pass '' for this parameter.
	string)		; aux_files_pattern : extra item(s) to be added to FILES for each package. Can be a single string item or a list of strings for multiple items. Must include %s.
	recursive -- perform a recursive search		; postrm : postrm script to use for all packages (as a string)
	hook -- a hook function to be called for each package		; allow_dirs : allow directories to be matched
	extra_depends -- extra runtime dependencies (RDEPENDS) to be set for		; prepend : if True, prepend created packages to PACKAGES instead of appending them
	all packages. The default value of None causes a		; match_path : match file_regex on the whole relative path to the root rather than just the file name
	dependency on the main package (${PN}) - if you do		; aux_files_pattern_verbatim : extra item(s) to be added to FILES for each package, using the actual derived module name rather than converting it to something legal for a package name. Can be a single string item or a list of strings for multiple items. Must include %s.
	not want this, pass '' for this parameter.		; allow_links : allow symlinks to be matched
	aux_files_pattern -- extra item(s) to be added to FILES for each
	package. Can be a single string item or a list of
	strings for multiple items. Must include %s.
	postrm -- postrm script to use for all packages (as a string)
	allow_dirs -- allow directories to be matched
	prepend -- if True, prepend created packages to PACKAGES instead
	of appending them
	match_path -- match file_regex on the whole relative path to the
	root rather than just the file name
	aux_files_pattern_verbatim -- extra item(s) to be added to FILES for
	each package, using the actual derived module name
	rather than converting it to something legal for a
	package name. Can be a single string item or a list
	of strings for multiple items. Must include %s.
	allow_links -- allow symlinks to be matched
	~~</nowiki>~~

	== Satisfying dependencies ==		== Satisfying dependencies ==

PaulEggleton at 10:10, 22 August 2012

2012-08-22T10:10:07Z

← Older revision		Revision as of 10:10, 22 August 2012
Line 35:		Line 35:
	not want this, pass '' for this parameter.		not want this, pass '' for this parameter.
	aux_files_pattern -- extra item(s) to be added to FILES for each		aux_files_pattern -- extra item(s) to be added to FILES for each
	package. Must include %s.		package. Can be a single string item or a list of
			strings for multiple items. Must include %s.
	postrm -- postrm script to use for all packages (as a string)		postrm -- postrm script to use for all packages (as a string)
	allow_dirs -- allow directories to be matched		allow_dirs -- allow directories to be matched
Line 45:		Line 46:
	each package, using the actual derived module name		each package, using the actual derived module name
	rather than converting it to something legal for a		rather than converting it to something legal for a
	package name. Must include %s.		package name. Can be a single string item or a list
			of strings for multiple items. Must include %s.
	allow_links -- allow symlinks to be matched		allow_links -- allow symlinks to be matched
	</nowiki>		</nowiki>

PaulEggleton at 10:06, 22 August 2012

2012-08-22T10:06:08Z

← Older revision		Revision as of 10:06, 22 August 2012
Line 1:		Line 1:
	Many pieces of software split functionality out into optional modules (or plugins), and which of these plugins are built may depend on configuration options. To avoid having to duplicate the logic for determining which modules are available in your recipe or packaging each module by hand, the build system provides functionality to handle module packaging ~~automatically~~.		Many pieces of software split functionality out into optional modules (or plugins), and which of these plugins are built may depend on configuration options. To avoid having to duplicate the logic for determining which modules are available in your recipe or packaging each module by hand, the build system provides functionality to handle module packaging dynamically.

	== Packaging ==		== Packaging ==

← Older revision		Revision as of 12:12, 22 August 2012
Line 5:		Line 5:
	Firstly, you need to ensure that the module packaging actually gets done. This is accomplished using the <tt>do_split_packages</tt> function within the <tt>populate_packages_prepend</tt> python function in your recipe; this searches for a pattern of files or directories under a specified path and creates a package for each one it finds, by appending to the PACKAGES variable and setting the appropriate values for FILES_packagename, RDEPENDS_packagename, DESCRIPTION_packagename etc. Here is an example from the recipe for lighttpd:		Firstly, you need to ensure that the module packaging actually gets done. This is accomplished using the <tt>do_split_packages</tt> function within the <tt>populate_packages_prepend</tt> python function in your recipe; this searches for a pattern of files or directories under a specified path and creates a package for each one it finds, by appending to the PACKAGES variable and setting the appropriate values for FILES_packagename, RDEPENDS_packagename, DESCRIPTION_packagename etc. Here is an example from the recipe for lighttpd:

			<nowiki>
	python populate_packages_prepend () {		python populate_packages_prepend () {
	lighttpd_libdir = d.expand('${libdir}')		lighttpd_libdir = d.expand('${libdir}')
	do_split_packages(d, lighttpd_libdir, '^mod_(.*)\.so$', 'lighttpd-module-%s', 'Lighttpd module for %s', extra_depends='')		do_split_packages(d, lighttpd_libdir, '^mod_(.*)\.so$', 'lighttpd-module-%s', 'Lighttpd module for %s', extra_depends='')
	}		}
			</nowiki>

	In the above example a number of things are specified in the call to do_split_packages: a directory in which to search (within the files installed by your recipe via do_install), a regular expression to match module files in that directory - note the parentheses () which mark the part of the expression from which the module name should be derived; a pattern to use for the package names, a description for each package, and an empty string for extra_depends which disables the default dependency on the main lighttpd package. Thus, if a file in ${libdir} called mod_alias.so is found, a package called lighttpd-module-alias will be created for it, with DESCRIPTION set to "Lighttpd module for alias".		In the above example a number of things are specified in the call to do_split_packages: a directory in which to search (within the files installed by your recipe via do_install), a regular expression to match module files in that directory - note the parentheses () which mark the part of the expression from which the module name should be derived; a pattern to use for the package names, a description for each package, and an empty string for extra_depends which disables the default dependency on the main lighttpd package. Thus, if a file in ${libdir} called mod_alias.so is found, a package called lighttpd-module-alias will be created for it, with DESCRIPTION set to "Lighttpd module for alias".
Line 33:		Line 35:
	:; output_pattern: as above		:; output_pattern: as above
	:; modulename: the module name derived using file_regex		:; modulename: the module name derived using file_regex
	; extra_depends : extra runtime dependencies (RDEPENDS) to be set for all packages. The default value of None causes a dependency on the main package (${PN}) - if you do not want this, pass '' for this parameter.		; extra_depends : extra runtime dependencies (RDEPENDS) to be set for all packages. The default value of None causes a dependency on the main package (${PN}) - if you do not want this, pass empty string <nowiki>''</nowiki> for this parameter.
	; aux_files_pattern : extra item(s) to be added to FILES for each package. Can be a single string item or a list of strings for multiple items. Must include %s.		; aux_files_pattern : extra item(s) to be added to FILES for each package. Can be a single string item or a list of strings for multiple items. Must include %s.
	; postrm : postrm script to use for all packages (as a string)		; postrm : postrm script to use for all packages (as a string)