Layer Tooling

2011-05-17T16:57:37Z

Jpuhlman: /* Jeremy Puhlman */

== Tasks ==

Extracted from the brainstorming info below.

=== 1.1 tasks ===

* Show a parse error if a bbappend matches no existing bb [Dexuan]
* bitbake-layers: stop it displaying the warning message about not being run from the wrapper [Dexuan]
* bitbake-layers: extend it to show overridden ("cloaked") recipes [Dexuan?]
* Update banner info to show layers being used and revisions/branches for each one if available [Dexuan]
* Dependency handling [Paul]
** Add metadata to layers - can go in layer.conf
*** Layer name
*** Layer version
*** Layer dependencies (as a list of layer names + minimum versions only)
** Add dependency/version check to sanity.bbclass
* Consider integrating Jeremy Puhlman's remote layers patch (see comments below) [Ke]
* Tool which merges multiple layer components in one repo (e.g. the poky repo situation) [Paul/Ke]
** Definition file that indicates what components come from where (git remote URI as well as subdirectory of that repo, and perhaps an optional pegged branch/rev?)
** For non-pegged components, may need to actually record latest revision integrated from the component?
** Simplify fetching and applying changes from upstream components onto the merged repo
** Enable splitting of a patch into separate parts for different components (use splitdiff for this?)
** Could be initially created with git-stitch-repo? (Depends on whether you want the history)
* Tool to merge (flatten) several layers into one [Paul/Dexuan]
** Take current layer config and apply all .bbappend / remove overridden .bb files and output to a separate directory

=== Future tasks ===

Nice to have - may be added in 1.1 time allowing but not mandatory:

* Break out / make accessible functionality from OE's newcollection.bbclass to allow splitting recipes out to another layer
* Able to visualise dependencies between layers
* bitbake -e on steroids to highlight differences due to layer changes? (long term goal)
* Recipe maintenance tools - this is long term

== Brainstorming submissions ==

=== Richard Purdie ===
* There is no dependency information between layers (layer X depends on layer Y and Z)
* There is no version information present (layer X needs version A of layer Y)
* There is no layer "ABI" version number present in layers
* Layer specific sanity checks? (other types of sanity checks on Yocto 1.1 roadmap)
* Ability to fetch layers from other repositories
** Multi SCM?
** Use bitbake fetcher?
* Ability to generate repositories representing composited layers (Poky, RP has hacky scripts)
* Tool to seperate patch into one for different composited parts
* Ability to include partial components of repositories (e.g. only beagleboard from meta-ti)
* Tool to merge (flatten) several layers into one
* Able to visualise dependencies between layers
* When bitbake prints banner of branch/commit, need to cover all layer components present
* bitbake -e on steroids to highlight differences due to layer changes? (long term goal)

Implementation thoughts:

* This layer tooling probably belongs at a higher level on the stack than bitbake itself
* Maybe need to split into "bootstrap" steps (e.g where pseudo is established, layers downloaded etc)
* Need some kind of config file defining layer's properties or at least enhance layer.conf significantly
* git submodules and repo should be looked at but due to specific needs and ability to control tool evolution, likely need own tool.
** Paul says: am guessing a blocker for submodules (other than the fact that they look painful to deal with) is that they don't help us with the "part of another repo" thing; as well as being read-only.

=== Martin Jansa ===
* Parsing error when ie foo_1.0.bbappend is found, but foo_1.0.bb in upper layer was upgraded to foo_1.1.bb

=== Paul Eggleton / Chris Larson ===

* Show easily which recipes are being used and which are overridden. We have bitbake-layers, but this clearly needs some extension. I think long term we would want to be able to analyse overrides across a set of layers to figure out where common stuff in non-core layers needs pushing up to oe-core.

* Allow layer maintainers to easily pull a version of a recipe into their own layer if it's about to be removed from oe-core - with some complicated recipes that could be painful/annoying to pick out all the necessary files by hand. We have newcollection.bbclass in OE which does this, may be useful to break it out to a separate script somehow.

* Recipe maintenance tools that take advantage of bitbake's parsing logic to aid blanket updates (e.g. variable renames). This should help mitigate the increased difficulty of having recipes spread out over multiple repositories when doing these kinds of updates.
** Chris says: This is non-trivial, but doable given zecke's ast implementation. I expect we need an API function in the parse package to parse a string/file and hand back the AST nodes, which we could then manipulate, and write code to generate the file back out of the ast.

=== Jeremy Puhlman ===

[http://lists.linuxtogo.org/pipermail/openembedded-core/2011-April/001708.html Patch to add remote layers]

Paul says: a good start but naturally we want to avoid the bits that hard-code variable values. I wonder about situations where people want their own versions of layers or to share remote layers (e.g. between an oe-core setup and a poky setup); however people can just use local layers for this.

Jeremy says: In reality they are setting "reasonable" defaults. Hardcoding implies something that is unchangeable. All the values are "set if unset", so they can be changed in the bblayers.conf
file. Prior to the introduction of layers, all the values were hardcoded, just in a bitbake.conf file, so you always had working fetchers. With out that the fetchers are basically broken until
after you have loaded up the layers. Given the fact that nearly every bitbake.conf file contains the same settings for defining fetchcmd and a like, simply having a reasonable default setting for
the things the fetchers need is not really that terrible an idea irrespective of the remote layering.

As for the second question. I have some content tools that I am cleaning up that basically provide methods for defining layers and groups of layers together for mirroring and sharing. That might
address what your talking about there. They work along with the patch posted here, but could be modified to output things a bit differently if needed.

Richard says:
http://lists.linuxtogo.org/pipermail/openembedded-core/2011-May/001940.html

Yocto Project - User contributions [en]

Layer Tooling