Tinfoil2

Introduction

Tinfoil is an API that allows you to write simple utilities that call into bitbake code to read variables, parse recipes, etc. In order to address several limitations of the current tinfoil implementation (including it not working with bitbake's memory-resident mode) we are currently in the process of a major rewrite of tinfoil and associated code labelled "tinfoil2", being tracked under bug 5470. The WIP is on paule/tinfoil2 branch on poky-contrib.

Challenges

All calls from tinfoil-using scripts (a list of which appears below) into internal code need to be examined, and any that access internal data structures probably need to be fixed to be able to go through the IPC mechanism via a function in the tinfoil class. "Fake" objects replacing the datastore and recipecache have been added to tinfoil to avoid changing too many lines, and remote datastore capability has been added so that client code can continue to use and manipulate datastore objects, but some adaptation is still required. We should be taking this opportunity to add reasonable API functions to access various useful functionality where it makes sense.

Using the IPC mechanism presents some challenges. Not only do we need to add commands to lib/bb/commands.py to do the things we want, we also have to ensure what we pass to them can be serialised - but it gets more difficult when you use memory resident mode which uses XMLRPC since that has a very limited set of data types that it can serialise (e.g. no nested dicts). You can start working on the tinfoil2 code in non-memres mode and get it working there, but when you switch to memres (which you should) you'll probably find the code you just added needs additional fixes.

What works

• Normal bitbake builds
• oe-selftest tests:
  • tinfoil (added by tinfoil2 patchset)
  • bblayers
  • devtool (apart from two failures)
  • recipetool (random issues noted below)

Issues / todo list

Pre-merge

• Spew of hash mismatches when running devtool extract. RP suggests not using real cache in buildFile() which may help, but then we still seem to trip the mismatch detection elsewhere
• If client exits in the middle of an async command, server is left in a broken state (e.g. tinfoil.functiondoesntexist() after tinfoil.runCommand('buildFile'...)
• Method of running task with different datastore is still awful (global setVariable)
• Can't Ctrl+C out when waiting for bitbake.lock
• Can't Ctrl+C out of event loop? e.g. comment out commandfailed handling in _extract_source
• Errors on server side during a command cause devtool to basically lock up - in fact even an error on the client side will do that :( (e.g. tinfoil.functiondoesntexist()). Haven't checked to see where the code is stuck, possibly waiting for an event that will never come.
• conf/devtool.conf written by something?! (ctrl+c'd / failed tests?)
• If you shutdown() before the current command finishes, you get stuck in a bitbake.lock wait
• Tidy up commits
• Run full oe-selftest in normal mode

Later

• Need to have API for properly resolving provides (which get_recipe_file() should use)
• No way to receive events sent during parsing?
• Still getting configParsed event even though filtered out?!
• set_event_mask() doesn't exclude any pre-generated events?
• set_event_mask() still apparently not working
• Still two failures when running oe-selftest --run-tests devtool; one is due to hash mismatch error messages, the other file checksum issues with memres (see "Memres issues" below)
• Add missing API functions?
• Some means of cleaning up accumulated datastore objects on the server side?
• Run more tests with memres
• Check/improve performance

Memres issues

• Cache is reloaded on every execution with memres
• bitbake -e shows no history with memres (bug 10730)
• --observe-only doesn't seem to work (bug 10731)
• File checksum changes don't rerun tasks (bug 10732) - breaks oe-selftest --run-tests devtool.DevtoolTests.test_devtool_modify
• bitbake-cookerdaemon.log seems to be triggering config change notifications !!!
• If you try to run two non-observer clients at once you should be presented with a reasonable message, but at the moment you get:

WARNING: Could not connect to server at 127.0.0.1:46777 (<ProtocolError for 127.0.0.1:46777/: 503 Service Unavailable>)
ERROR: Could not connect to server 127.0.0.1:46777
: <ProtocolError for 127.0.0.1:46777/: 503 Service Unavailable>

• Connecting to bitbake server in memres mode fails -> not very well handled error
• Random hangs and timeouts when running recipetool oe-selftest tests - and killing the hung bitbake instance (usually a bitbake -e) leaves the server thinking that the client is still connected - something to do with the client token still being recorded?

Annoyances

• check if all recipes parsed if appends=true when parsing a recipe, and if getting pkg_pn
• setEventMask makes notes printed in tasks come through to the UI
• bitbake -m doesn't print anything
• Would be nice if bitbake -m would cd back to directory ready to start oe-init-build-env-memres
• Have a restart server command?
• oe-init-build-env-memres still puts you in build dir if there is an invalid layer in conf/bblayers.conf

Tests

• Test provider functions
• Test for errant debug messages on normal tinfoil operation

Scripts to check this doesn't break

• bitbake-diffsigs
• bitbake-layers
• devtool
• recipetool
• list-packageconfig-flags.py
• verify-homepage.py
• oe-pkgdata-util
• layer index update.py
• bb ? (likely to be heavily broken since it uses more internal API)