Tinfoil2

Introduction

Tinfoil is an API that allows you to write simple utilities that call into bitbake code to read variables, parse recipes, etc. In order to address several limitations of the current tinfoil implementation (including it not working with bitbake's memory-resident mode) we are currently in the process of a major rewrite of tinfoil and associated code labelled "tinfoil2", being tracked under bug 5470. The WIP is on paule/tinfoil2 branch on poky-contrib.

Challenges

All calls from tinfoil-using scripts (a list of which appears below) into internal code need to be examined, and any that access internal data structures probably need to be fixed to be able to go through the IPC mechanism via a function in the tinfoil class. "Fake" objects replacing the datastore and recipecache have been added to tinfoil to avoid changing too many lines, and remote datastore capability has been added so that client code continue to use and manipulate datastore objects as previously, but some adaptation is still required. We should be taking this opportunity to add reasonable API functions to access various useful functionality where it makes sense.

Using the IPC mechanism presents some challenges. Not only do we need to add commands to lib/bb/commands.py to do the things we want, we also have to ensure what we pass to them can be serialised - but it gets more difficult when you use memory resident mode which uses XMLRPC since that has a very limited set of data types that it can serialise (e.g. no nested dicts). You can start working on the tinfoil2 code in non-memres mode and get it working there, but when you switch to memres (which you should) you'll probably find the code you just added needs additional fixes.

What works

• Normal bitbake builds
• oe-selftest tests:
  • tinfoil (added by tinfoil2 patchset)
  • bblayers
  • devtool (apart from two failures)
  • recipetool (random issues noted below)

Issues / todo list

• Spew of taskhash mismatches when running devtool extract
• Still three failures when running oe-selftest --run-tests devtool
• If client exits in the middle of an async command, server is left in a broken state (e.g. tinfoil.functiondoesntexist() after tinfoil.runCommand('buildFile'...)
• Random hangs and timeouts when running recipetool oe-selftest tests - and killing the hung bitbake instance (usually a bitbake -e) leaves the server thinking that the client is still connected - something to do with the client token still being recorded?

• Errors on server side during a command cause devtool to basically lock up - in fact even an error on the client side will do that :( (e.g. tinfoil.functiondoesntexist()). Haven't checked to see where the code is stuck, possibly waiting for an event that will never come.

• conf/devtool.conf written by something?! (ctrl+c'd / failed tests?)

• Need to have API for properly resolving provides (which get_recipe_file() should use)

• No way to receive events sent during parsing?
• Still getting configParsed event even though filtered out?!
• set_event_mask() doesn't exclude any pre-generated events?
• set_event_mask() still apparently not working

• Can't Ctrl+C out when waiting for bitbake.lock
• If you shutdown() before the current command finishes, you get stuck in a bitbake.lock wait
• Connecting to bitbake server in memres mode fails -> not very well handled error

• Can't ctrl+c out of event loop? e.g. comment out commandfailed handling in _extract_source

• Add missing API functions e.g. for executing tasks - currently devtool extract is using run_command(), there should really be a dedicated function for that
• Run more tests with memres
• Check/improve performance

Memres issues

• Cache is reloaded on every execution with memres
• bitbake -e shows no history with memres (bug 10730)
• --observe-only doesn't seem to work (bug 10731)
• File checksum changes don't rerun tasks (bug 10732) - breaks oe-selftest --run-tests devtool.DevtoolTests.test_devtool_modify
• bitbake-cookerdaemon.log seems to be triggering config change notifications !!!

Annoyances

• check if all recipes parsed if appends=true when parsing a recipe, and if getting pkg_pn
• setEventMask makes notes printed in tasks come through to the UI
• bitbake -m doesn't print anything
• Would be nice if bitbake -m would cd back to directory ready to start oe-init-build-env-memres
• Have a restart server command?
• oe-init-build-env-memres still puts you in build dir if there is an invalid layer in conf/bblayers.conf

Tests

• Test provider functions
• Test for errant debug messages on normal tinfoil operation

Scripts to check this doesn't break

• bitbake-diffsigs
• bitbake-layers
• devtool
• recipetool
• list-packageconfig-flags.py
• verify-homepage.py
• oe-pkgdata-util
• layer index update.py
• bb ? (likely to be heavily broken since it uses more internal API)