View source for Tinfoil2

== Introduction ==

Tinfoil is an API that allows you to write simple utilities that call into bitbake code to read variables, parse recipes, etc. In order to address several limitations of the current tinfoil implementation (including it not working with bitbake's memory-resident mode) we are currently in the process of a major rewrite of tinfoil and associated code labelled "tinfoil2", being tracked under [https://bugzilla.yoctoproject.org/show_bug.cgi?id=5470 bug 5470]. The WIP is on [http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=paule/tinfoil2 paule/tinfoil2] branch on poky-contrib but this has now been merged to master as of mid-Dec 2016.

=== Challenges ===

All calls from tinfoil-using scripts (a list of which appears below) into internal code need to be examined, and any that access internal data structures probably need to be fixed to be able to go through the IPC mechanism via a function in the tinfoil class. "Fake" objects replacing the datastore and recipecache have been added to tinfoil to avoid changing too many lines, and remote datastore capability has been added so that client code can continue to use and manipulate datastore objects, but some adaptation is still required. We should be taking this opportunity to add reasonable API functions to access various useful functionality where it makes sense.

Using the IPC mechanism presents some challenges. Not only do we need to add commands to lib/bb/commands.py to do the things we want, we also have to ensure what we pass to them can be serialised - but it gets more difficult when you use memory resident mode which uses XMLRPC since that has a very limited set of data types that it can serialise (e.g. no nested dicts). You can start working on the tinfoil2 code in non-memres mode and get it working there, but when you switch to memres (which you should) you'll probably find the code you just added needs additional fixes.

== Issues / todo list ==
* conf/devtool.conf written by something?! (ctrl+c'd / failed tests?)
* If you shutdown() before the current command finishes, you get stuck in a bitbake.lock wait
* Errors on server side during a command cause devtool to basically lock up - in fact even an error on the client side will do that :( (e.g. tinfoil.functiondoesntexist()). Haven't checked to see where the code is stuck, possibly waiting for an event that will never come.
* Can't Ctrl+C out when waiting for bitbake.lock
* If client exits in the middle of an async command, server is left in a broken state (e.g. tinfoil.functiondoesntexist() after tinfoil.runCommand('buildFile'...)
* Need to have API for properly resolving provides (which get_recipe_file() should use)
* No way to receive events sent during parsing?
* Still getting configParsed event even though filtered out?!
* set_event_mask() doesn't exclude any pre-generated events?
* set_event_mask() still apparently not working
* Still two failures when running oe-selftest --run-tests devtool; one is due to hash mismatch error messages, the other file checksum issues with memres (see "Memres issues" below)
* Add missing API functions?
* Some means of cleaning up accumulated datastore objects on the server side?
* Run more tests with memres
* Check/improve performance

=== Memres issues ===
* Cache is reloaded on every execution with memres
* bitbake -e shows no history with memres ([https://bugzilla.yoctoproject.org/show_bug.cgi?id=10730 bug 10730])
* --observe-only doesn't seem to work ([https://bugzilla.yoctoproject.org/show_bug.cgi?id=10731 bug 10731])
* File checksum changes don't rerun tasks ([https://bugzilla.yoctoproject.org/show_bug.cgi?id=10732 bug 10732]) - breaks <code>oe-selftest --run-tests devtool.DevtoolTests.test_devtool_modify</code>
* bitbake-cookerdaemon.log seems to be triggering config change notifications !!!
* If you try to run two non-observer clients at once you should be presented with a reasonable message, but at the moment you get:
 WARNING: Could not connect to server at 127.0.0.1:46777 (<ProtocolError for 127.0.0.1:46777/: 503 Service Unavailable>)
 ERROR: Could not connect to server 127.0.0.1:46777
 : <ProtocolError for 127.0.0.1:46777/: 503 Service Unavailable>
* Connecting to bitbake server in memres mode fails -> not very well handled error
* Random hangs and timeouts when running recipetool oe-selftest tests - and killing the hung bitbake instance (usually a bitbake -e) leaves the server thinking that the client is still connected - something to do with the client token still being recorded?

=== Annoyances ===
* check if all recipes parsed if appends=true when parsing a recipe, and if getting pkg_pn
* setEventMask makes notes printed in tasks come through to the UI
* bitbake -m doesn't print anything
* Would be nice if bitbake -m would cd back to directory ready to start oe-init-build-env-memres
* Have a restart server command?
* oe-init-build-env-memres still puts you in build dir if there is an invalid layer in conf/bblayers.conf

=== Tests ===
* Test provider functions
* Test for errant debug messages on normal tinfoil operation

=== Scripts to check this doesn't break ===
* bitbake-diffsigs
* bitbake-layers
* devtool
* recipetool
* list-packageconfig-flags.py
* verify-homepage.py
* oe-pkgdata-util
* layer index update.py
* bb ? (likely to be heavily broken since it uses more internal API)