Layer Tooling

Tasks

Extracted from the brainstorming info below.

1.1 tasks

• Show a parse error if a bbappend matches no existing bb [Dexuan]
• bitbake-layers: stop it displaying the warning message about not being run from the wrapper [Dexuan]
• bitbake-layers: extend it to show overlayed recipes [Paul]
• Update banner info to show layers being used and revisions/branches for each one if available [Dexuan]
• Dependency handling [Paul]
  • Add metadata to layers - can go in layer.conf
    • Layer name
    • Layer version
    • Layer dependencies (as a list of layer names + versions)
  • Add dependency/version check
• Consider integrating Jeremy Puhlman's remote layers patch (see comments below) [Paul]
• Tool to manage combination repos (e.g. the poky repo) - see below [Paul/Ke]
• Tool to merge (flatten) several layers into one [Paul]
  • Take current layer config and apply all .bbappend / remove overridden .bb files and output to a separate directory

Combination repo tool

• Separate tool - task specific and most people won't need to use it
• Assume current dir is within the combination repo
• Define repo components in a separate config file (conf/combolayer.conf ?). For each component:
  • Source repo
  • Destination subdir within combination repo tree
  • Local repo workdir
  • Pinned revision (optional)
  • Last revision merged
• Operations:
  • Create
    1. Check git initialised and working tree is clean
    2. Clone component repositories
  • Update: merge changes in from upstream components
    1. Ensure working tree is clean
    2. Update combination repo
    3. Update component repos
    4. Apply patches from component repo to the combination repo (should this be done in a branch which we can throw away in case of error?)
       1. Get list of revisions since last merged revision, then iterate:
       2. Append upstream revision to commit message
       3. Optionally append signed-off-by (-s)?
       4. Apply revision to combination repo
  • Splitpatch: split a patch up over the components
    1. Use the config file to understand which subdirs belong to which component
    2. Output split patches to component-named subdirs of some destination folder (taken as an argument to the command)

Future tasks

Nice to have - may be added in 1.1 time allowing but not mandatory:

• Break out / make accessible functionality from OE's newcollection.bbclass to allow splitting recipes out to another layer
• Able to visualise dependencies between layers
• bitbake -e on steroids to highlight differences due to layer changes? (long term goal)
• Recipe maintenance tools - this is long term

Brainstorming submissions

Richard Purdie

• There is no dependency information between layers (layer X depends on layer Y and Z)
• There is no version information present (layer X needs version A of layer Y)
• There is no layer "ABI" version number present in layers
• Layer specific sanity checks? (other types of sanity checks on Yocto 1.1 roadmap)
• Ability to fetch layers from other repositories
  • Multi SCM?
  • Use bitbake fetcher?
• Ability to generate repositories representing composited layers (Poky, RP has hacky scripts)
• Tool to seperate patch into one for different composited parts
• Ability to include partial components of repositories (e.g. only beagleboard from meta-ti)
• Tool to merge (flatten) several layers into one
• Able to visualise dependencies between layers
• When bitbake prints banner of branch/commit, need to cover all layer components present
• bitbake -e on steroids to highlight differences due to layer changes? (long term goal)

Implementation thoughts:

• This layer tooling probably belongs at a higher level on the stack than bitbake itself
• Maybe need to split into "bootstrap" steps (e.g where pseudo is established, layers downloaded etc)
• Need some kind of config file defining layer's properties or at least enhance layer.conf significantly
• git submodules and repo should be looked at but due to specific needs and ability to control tool evolution, likely need own tool.
  • Paul says: am guessing a blocker for submodules (other than the fact that they look painful to deal with) is that they don't help us with the "part of another repo" thing; as well as being read-only.

Martin Jansa

• Parsing error when ie foo_1.0.bbappend is found, but foo_1.0.bb in upper layer was upgraded to foo_1.1.bb

Paul Eggleton / Chris Larson

• Show easily which recipes are being used and which are overridden. We have bitbake-layers, but this clearly needs some extension. I think long term we would want to be able to analyse overrides across a set of layers to figure out where common stuff in non-core layers needs pushing up to oe-core.

• Allow layer maintainers to easily pull a version of a recipe into their own layer if it's about to be removed from oe-core - with some complicated recipes that could be painful/annoying to pick out all the necessary files by hand. We have newcollection.bbclass in OE which does this, may be useful to break it out to a separate script somehow.

• Recipe maintenance tools that take advantage of bitbake's parsing logic to aid blanket updates (e.g. variable renames). This should help mitigate the increased difficulty of having recipes spread out over multiple repositories when doing these kinds of updates.
  • Chris says: This is non-trivial, but doable given zecke's ast implementation. I expect we need an API function in the parse package to parse a string/file and hand back the AST nodes, which we could then manipulate, and write code to generate the file back out of the ast.

Jeremy Puhlman

Patch to add remote layers

Paul says: a good start but naturally we want to avoid the bits that hard-code variable values. I wonder about situations where people want their own versions of layers or to share remote layers (e.g. between an oe-core setup and a poky setup); however people can just use local layers for this.

  Jeremy says: In reality they are setting "reasonable" defaults. Hardcoding implies something that is unchangeable. All the values are "set if unset", so they can be changed in the bblayers.conf
  file. Prior to the introduction of layers, all the values were hardcoded, just in a bitbake.conf file, so you always had working fetchers. With out that the fetchers are basically broken until  
  after you have loaded up the layers. Given the fact that nearly every bitbake.conf file contains the same settings for defining fetchcmd and a like, simply having a reasonable default setting for 
  the things the fetchers need is not really that terrible an idea irrespective of the remote layering.

  As for the second question. I have some content tools that I am cleaning up that basically provide methods for defining layers and groups of layers together for mirroring and sharing. That might   
  address what your talking about there. They work along with the patch posted here, but could be modified to output things a bit differently if needed.

Richard says: http://lists.linuxtogo.org/pipermail/openembedded-core/2011-May/001940.html

(discussion moved back to mailing list)