Tinfoil2: Difference between revisions

From Yocto Project
Jump to navigationJump to search
No edit summary
No edit summary
 
(5 intermediate revisions by the same user not shown)
Line 1: Line 1:
== Introduction ==
== Introduction ==


Tinfoil is an API that allows you to write simple utilities that call into bitbake code to read variables, parse recipes, etc. In order to address several limitations of the current tinfoil implementation (including it not working with bitbake's memory-resident mode) we are currently in the process of a major rewrite of tinfoil and associated code labelled "tinfoil2", being tracked under [https://bugzilla.yoctoproject.org/show_bug.cgi?id=5470 bug 5470]. The WIP is on [http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=paule/tinfoil2 paule/tinfoil2] branch on poky-contrib.
Tinfoil is an API that allows you to write simple utilities that call into bitbake code to read variables, parse recipes, etc. In order to address several limitations of the current tinfoil implementation (including it not working with bitbake's memory-resident mode) we are currently in the process of a major rewrite of tinfoil and associated code labelled "tinfoil2", being tracked under [https://bugzilla.yoctoproject.org/show_bug.cgi?id=5470 bug 5470]. The WIP is on [http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=paule/tinfoil2 paule/tinfoil2] branch on poky-contrib but this has now been merged to master as of mid-Dec 2016.


=== Challenges ===
=== Challenges ===
Line 8: Line 8:


Using the IPC mechanism presents some challenges. Not only do we need to add commands to lib/bb/commands.py to do the things we want, we also have to ensure what we pass to them can be serialised - but it gets more difficult when you use memory resident mode which uses XMLRPC since that has a very limited set of data types that it can serialise (e.g. no nested dicts). You can start working on the tinfoil2 code in non-memres mode and get it working there, but when you switch to memres (which you should) you'll probably find the code you just added needs additional fixes.
Using the IPC mechanism presents some challenges. Not only do we need to add commands to lib/bb/commands.py to do the things we want, we also have to ensure what we pass to them can be serialised - but it gets more difficult when you use memory resident mode which uses XMLRPC since that has a very limited set of data types that it can serialise (e.g. no nested dicts). You can start working on the tinfoil2 code in non-memres mode and get it working there, but when you switch to memres (which you should) you'll probably find the code you just added needs additional fixes.
== What works ==
* Normal bitbake builds
* oe-selftest tests:
** tinfoil (added by tinfoil2 patchset)
** bblayers
** devtool (apart from two failures)
** recipetool (random issues noted below)


== Issues / todo list ==
== Issues / todo list ==
 
* conf/devtool.conf written by something?! (ctrl+c'd / failed tests?)
* Spew of hash mismatches when running devtool extract. RP suggests not using real cache in buildFile() which may help, but then we still seem to trip the mismatch detection elsewhere
* If you shutdown() before the current command finishes, you get stuck in a bitbake.lock wait
* Still two failures when running oe-selftest --run-tests devtool; one is due to hash mismatch error messages, the other file checksum issues with memres (see "Memres issues" below)
* Errors on server side during a command cause devtool to basically lock up - in fact even an error on the client side will do that :( (e.g. tinfoil.functiondoesntexist()). Haven't checked to see where the code is stuck, possibly waiting for an event that will never come.
* Can't Ctrl+C out when waiting for bitbake.lock
* If client exits in the middle of an async command, server is left in a broken state (e.g. tinfoil.functiondoesntexist() after tinfoil.runCommand('buildFile'...)
* If client exits in the middle of an async command, server is left in a broken state (e.g. tinfoil.functiondoesntexist() after tinfoil.runCommand('buildFile'...)
* Random hangs and timeouts when running recipetool oe-selftest tests - and killing the hung bitbake instance (usually a bitbake -e) leaves the server thinking that the client is still connected - something to do with the client token still being recorded?
* Errors on server side during a command cause devtool to basically lock up - in fact even an error on the client side will do that :( (e.g. tinfoil.functiondoesntexist()). Haven't checked to see where the code is stuck, possibly waiting for an event that will never come.
* conf/devtool.conf written by something?! (ctrl+c'd / failed tests?)
* Need to have API for properly resolving provides (which get_recipe_file() should use)
* Need to have API for properly resolving provides (which get_recipe_file() should use)
* No way to receive events sent during parsing?
* No way to receive events sent during parsing?
* Still getting configParsed event even though filtered out?!
* Still getting configParsed event even though filtered out?!
* set_event_mask() doesn't exclude any pre-generated events?
* set_event_mask() doesn't exclude any pre-generated events?
* set_event_mask() still apparently not working
* set_event_mask() still apparently not working
 
* Still two failures when running oe-selftest --run-tests devtool; one is due to hash mismatch error messages, the other file checksum issues with memres (see "Memres issues" below)
* Can't Ctrl+C out when waiting for bitbake.lock
* Add missing API functions?
* If you shutdown() before the current command finishes, you get stuck in a bitbake.lock wait
* Connecting to bitbake server in memres mode fails -> not very well handled error
 
* Can't ctrl+c out of event loop? e.g. comment out commandfailed handling in _extract_source
 
* Add missing API functions e.g. for executing tasks - currently devtool extract is using run_command(), there should really be a dedicated function for that
* Some means of cleaning up accumulated datastore objects on the server side?
* Some means of cleaning up accumulated datastore objects on the server side?
* Run more tests with memres
* Run more tests with memres
* Check/improve performance
* Check/improve performance
* Tidy up commits


=== Memres issues ===
=== Memres issues ===
Line 57: Line 36:
  ERROR: Could not connect to server 127.0.0.1:46777
  ERROR: Could not connect to server 127.0.0.1:46777
  : <ProtocolError for 127.0.0.1:46777/: 503 Service Unavailable>
  : <ProtocolError for 127.0.0.1:46777/: 503 Service Unavailable>
* Connecting to bitbake server in memres mode fails -> not very well handled error
* Random hangs and timeouts when running recipetool oe-selftest tests - and killing the hung bitbake instance (usually a bitbake -e) leaves the server thinking that the client is still connected - something to do with the client token still being recorded?


=== Annoyances ===
=== Annoyances ===

Latest revision as of 08:56, 9 January 2017

Introduction

Tinfoil is an API that allows you to write simple utilities that call into bitbake code to read variables, parse recipes, etc. In order to address several limitations of the current tinfoil implementation (including it not working with bitbake's memory-resident mode) we are currently in the process of a major rewrite of tinfoil and associated code labelled "tinfoil2", being tracked under bug 5470. The WIP is on paule/tinfoil2 branch on poky-contrib but this has now been merged to master as of mid-Dec 2016.

Challenges

All calls from tinfoil-using scripts (a list of which appears below) into internal code need to be examined, and any that access internal data structures probably need to be fixed to be able to go through the IPC mechanism via a function in the tinfoil class. "Fake" objects replacing the datastore and recipecache have been added to tinfoil to avoid changing too many lines, and remote datastore capability has been added so that client code can continue to use and manipulate datastore objects, but some adaptation is still required. We should be taking this opportunity to add reasonable API functions to access various useful functionality where it makes sense.

Using the IPC mechanism presents some challenges. Not only do we need to add commands to lib/bb/commands.py to do the things we want, we also have to ensure what we pass to them can be serialised - but it gets more difficult when you use memory resident mode which uses XMLRPC since that has a very limited set of data types that it can serialise (e.g. no nested dicts). You can start working on the tinfoil2 code in non-memres mode and get it working there, but when you switch to memres (which you should) you'll probably find the code you just added needs additional fixes.

Issues / todo list

  • conf/devtool.conf written by something?! (ctrl+c'd / failed tests?)
  • If you shutdown() before the current command finishes, you get stuck in a bitbake.lock wait
  • Errors on server side during a command cause devtool to basically lock up - in fact even an error on the client side will do that :( (e.g. tinfoil.functiondoesntexist()). Haven't checked to see where the code is stuck, possibly waiting for an event that will never come.
  • Can't Ctrl+C out when waiting for bitbake.lock
  • If client exits in the middle of an async command, server is left in a broken state (e.g. tinfoil.functiondoesntexist() after tinfoil.runCommand('buildFile'...)
  • Need to have API for properly resolving provides (which get_recipe_file() should use)
  • No way to receive events sent during parsing?
  • Still getting configParsed event even though filtered out?!
  • set_event_mask() doesn't exclude any pre-generated events?
  • set_event_mask() still apparently not working
  • Still two failures when running oe-selftest --run-tests devtool; one is due to hash mismatch error messages, the other file checksum issues with memres (see "Memres issues" below)
  • Add missing API functions?
  • Some means of cleaning up accumulated datastore objects on the server side?
  • Run more tests with memres
  • Check/improve performance

Memres issues

  • Cache is reloaded on every execution with memres
  • bitbake -e shows no history with memres (bug 10730)
  • --observe-only doesn't seem to work (bug 10731)
  • File checksum changes don't rerun tasks (bug 10732) - breaks oe-selftest --run-tests devtool.DevtoolTests.test_devtool_modify
  • bitbake-cookerdaemon.log seems to be triggering config change notifications !!!
  • If you try to run two non-observer clients at once you should be presented with a reasonable message, but at the moment you get:
WARNING: Could not connect to server at 127.0.0.1:46777 (<ProtocolError for 127.0.0.1:46777/: 503 Service Unavailable>)
ERROR: Could not connect to server 127.0.0.1:46777
: <ProtocolError for 127.0.0.1:46777/: 503 Service Unavailable>
  • Connecting to bitbake server in memres mode fails -> not very well handled error
  • Random hangs and timeouts when running recipetool oe-selftest tests - and killing the hung bitbake instance (usually a bitbake -e) leaves the server thinking that the client is still connected - something to do with the client token still being recorded?

Annoyances

  • check if all recipes parsed if appends=true when parsing a recipe, and if getting pkg_pn
  • setEventMask makes notes printed in tasks come through to the UI
  • bitbake -m doesn't print anything
  • Would be nice if bitbake -m would cd back to directory ready to start oe-init-build-env-memres
  • Have a restart server command?
  • oe-init-build-env-memres still puts you in build dir if there is an invalid layer in conf/bblayers.conf

Tests

  • Test provider functions
  • Test for errant debug messages on normal tinfoil operation

Scripts to check this doesn't break

  • bitbake-diffsigs
  • bitbake-layers
  • devtool
  • recipetool
  • list-packageconfig-flags.py
  • verify-homepage.py
  • oe-pkgdata-util
  • layer index update.py
  • bb ? (likely to be heavily broken since it uses more internal API)
Retrieved from "https://wiki.yoctoproject.org/wiki/index.php?title=Tinfoil2&oldid=22581"