Tinfoil2 - Revision history

PaulEggleton at 08:56, 9 January 2017

2017-01-09T08:56:14Z

← Older revision		Revision as of 08:56, 9 January 2017
Line 1:		Line 1:
	== Introduction ==		== Introduction ==

	Tinfoil is an API that allows you to write simple utilities that call into bitbake code to read variables, parse recipes, etc. In order to address several limitations of the current tinfoil implementation (including it not working with bitbake's memory-resident mode) we are currently in the process of a major rewrite of tinfoil and associated code labelled "tinfoil2", being tracked under [https://bugzilla.yoctoproject.org/show_bug.cgi?id=5470 bug 5470]. The WIP is on [http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=paule/tinfoil2 paule/tinfoil2] branch on poky-contrib.		Tinfoil is an API that allows you to write simple utilities that call into bitbake code to read variables, parse recipes, etc. In order to address several limitations of the current tinfoil implementation (including it not working with bitbake's memory-resident mode) we are currently in the process of a major rewrite of tinfoil and associated code labelled "tinfoil2", being tracked under [https://bugzilla.yoctoproject.org/show_bug.cgi?id=5470 bug 5470]. The WIP is on [http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=paule/tinfoil2 paule/tinfoil2] branch on poky-contrib but this has now been merged to master as of mid-Dec 2016.

	=== Challenges ===		=== Challenges ===

PaulEggleton at 06:45, 13 December 2016

2016-12-13T06:45:20Z

← Older revision		Revision as of 06:45, 13 December 2016
Line 8:		Line 8:

	Using the IPC mechanism presents some challenges. Not only do we need to add commands to lib/bb/commands.py to do the things we want, we also have to ensure what we pass to them can be serialised - but it gets more difficult when you use memory resident mode which uses XMLRPC since that has a very limited set of data types that it can serialise (e.g. no nested dicts). You can start working on the tinfoil2 code in non-memres mode and get it working there, but when you switch to memres (which you should) you'll probably find the code you just added needs additional fixes.		Using the IPC mechanism presents some challenges. Not only do we need to add commands to lib/bb/commands.py to do the things we want, we also have to ensure what we pass to them can be serialised - but it gets more difficult when you use memory resident mode which uses XMLRPC since that has a very limited set of data types that it can serialise (e.g. no nested dicts). You can start working on the tinfoil2 code in non-memres mode and get it working there, but when you switch to memres (which you should) you'll probably find the code you just added needs additional fixes.

	~~== What works ==~~
	* Normal bitbake builds
	* oe-selftest tests:
	** tinfoil (added by tinfoil2 patchset)
	** bblayers
	** devtool (apart from two failures)
	** recipetool (random issues noted below)

	== Issues / todo list ==		== Issues / todo list ==
	~~=== Pre-merge ===~~
	* <strike>Spew of hash mismatches when running devtool extract. RP suggests not using real cache in buildFile() which may help, but then we still seem to trip the mismatch detection elsewhere</strike>
	* If client exits in the middle of an async command, server is left in a broken state (e.g. tinfoil.functiondoesntexist() after tinfoil.runCommand('buildFile'...)
	* Method of running task with different datastore is still awful (global setVariable)
	* Can't Ctrl+C out when waiting for bitbake.lock
	* Can't Ctrl+C out of event loop? e.g. comment out commandfailed handling in _extract_source
	* Errors on server side during a command cause devtool to basically lock up - in fact even an error on the client side will do that :( (e.g. tinfoil.functiondoesntexist()). Haven't checked to see where the code is stuck, possibly waiting for an event that will never come.
	* conf/devtool.conf written by something?! (ctrl+c'd / failed tests?)		* conf/devtool.conf written by something?! (ctrl+c'd / failed tests?)
	* If you shutdown() before the current command finishes, you get stuck in a bitbake.lock wait		* If you shutdown() before the current command finishes, you get stuck in a bitbake.lock wait
	* ~~Tidy~~ up ~~commits~~		* Errors on server side during a command cause devtool to basically lock up - in fact even an error on the client side will do that :( (e.g. tinfoil.functiondoesntexist()). Haven't checked to see where the code is stuck, possibly waiting for an event that will never come.
	* Run full oe-~~selftest~~ in ~~normal mode~~		* Can't Ctrl+C out when waiting for bitbake.lock
			* If client exits in the middle of an async command, server is left in a broken state (e.g. tinfoil.functiondoesntexist() after tinfoil.runCommand('buildFile'...)
	~~=== Later ===~~
	* Need to have API for properly resolving provides (which get_recipe_file() should use)		* Need to have API for properly resolving provides (which get_recipe_file() should use)
	* No way to receive events sent during parsing?		* No way to receive events sent during parsing?

PaulEggleton at 08:59, 5 December 2016

2016-12-05T08:59:02Z

← Older revision		Revision as of 08:59, 5 December 2016
Line 18:		Line 18:

	== Issues / todo list ==		== Issues / todo list ==
			=== Pre-merge ===
	* <strike>Spew of hash mismatches when running devtool extract. RP suggests not using real cache in buildFile() which may help, but then we still seem to trip the mismatch detection elsewhere</strike>		* <strike>Spew of hash mismatches when running devtool extract. RP suggests not using real cache in buildFile() which may help, but then we still seem to trip the mismatch detection elsewhere</strike>
			* If client exits in the middle of an async command, server is left in a broken state (e.g. tinfoil.functiondoesntexist() after tinfoil.runCommand('buildFile'...)
	* Method of running task with different datastore is still awful (global setVariable)		* Method of running task with different datastore is still awful (global setVariable)
	* If client exits in the middle of an async command, server is left in a broken state (e.g. tinfoil.functiondoesntexist() after tinfoil.runCommand('buildFile'...)
	* Random hangs and timeouts when running recipetool oe-selftest tests - and killing the hung bitbake instance (usually a bitbake -e) leaves the server thinking that the client is still connected - something to do with the client token still being recorded?
	* Can't Ctrl+C out when waiting for bitbake.lock		* Can't Ctrl+C out when waiting for bitbake.lock
	* Can't Ctrl+C out of event loop? e.g. comment out commandfailed handling in _extract_source		* Can't Ctrl+C out of event loop? e.g. comment out commandfailed handling in _extract_source
	* Errors on server side during a command cause devtool to basically lock up - in fact even an error on the client side will do that :( (e.g. tinfoil.functiondoesntexist()). Haven't checked to see where the code is stuck, possibly waiting for an event that will never come.		* Errors on server side during a command cause devtool to basically lock up - in fact even an error on the client side will do that :( (e.g. tinfoil.functiondoesntexist()). Haven't checked to see where the code is stuck, possibly waiting for an event that will never come.
	* conf/devtool.conf written by something?! (ctrl+c'd / failed tests?)		* conf/devtool.conf written by something?! (ctrl+c'd / failed tests?)
			* If you shutdown() before the current command finishes, you get stuck in a bitbake.lock wait
			* Tidy up commits
			* Run full oe-selftest in normal mode

			=== Later ===
	* Need to have API for properly resolving provides (which get_recipe_file() should use)		* Need to have API for properly resolving provides (which get_recipe_file() should use)
	* No way to receive events sent during parsing?		* No way to receive events sent during parsing?
Line 32:		Line 36:
	* set_event_mask() doesn't exclude any pre-generated events?		* set_event_mask() doesn't exclude any pre-generated events?
	* set_event_mask() still apparently not working		* set_event_mask() still apparently not working
	* If you shutdown() before the current command finishes, you get stuck in a bitbake.lock wait
	* Still two failures when running oe-selftest --run-tests devtool; one is due to hash mismatch error messages, the other file checksum issues with memres (see "Memres issues" below)		* Still two failures when running oe-selftest --run-tests devtool; one is due to hash mismatch error messages, the other file checksum issues with memres (see "Memres issues" below)
	* Add missing API functions?		* Add missing API functions?
Line 38:		Line 41:
	* Run more tests with memres		* Run more tests with memres
	* Check/improve performance		* Check/improve performance
	* Tidy up commits

	=== Memres issues ===		=== Memres issues ===
Line 51:		Line 53:
	: <ProtocolError for 127.0.0.1:46777/: 503 Service Unavailable>		: <ProtocolError for 127.0.0.1:46777/: 503 Service Unavailable>
	* Connecting to bitbake server in memres mode fails -> not very well handled error		* Connecting to bitbake server in memres mode fails -> not very well handled error
			* Random hangs and timeouts when running recipetool oe-selftest tests - and killing the hung bitbake instance (usually a bitbake -e) leaves the server thinking that the client is still connected - something to do with the client token still being recorded?

	=== Annoyances ===		=== Annoyances ===

PaulEggleton at 01:48, 2 December 2016

2016-12-02T01:48:33Z

← Older revision		Revision as of 01:48, 2 December 2016
Line 19:		Line 19:
	== Issues / todo list ==		== Issues / todo list ==

	* Spew of hash mismatches when running devtool extract. RP suggests not using real cache in buildFile() which may help, but then we still seem to trip the mismatch detection elsewhere		* <strike>Spew of hash mismatches when running devtool extract. RP suggests not using real cache in buildFile() which may help, but then we still seem to trip the mismatch detection elsewhere</strike>
			* Method of running task with different datastore is still awful (global setVariable)
	* If client exits in the middle of an async command, server is left in a broken state (e.g. tinfoil.functiondoesntexist() after tinfoil.runCommand('buildFile'...)		* If client exits in the middle of an async command, server is left in a broken state (e.g. tinfoil.functiondoesntexist() after tinfoil.runCommand('buildFile'...)
	* Random hangs and timeouts when running recipetool oe-selftest tests - and killing the hung bitbake instance (usually a bitbake -e) leaves the server thinking that the client is still connected - something to do with the client token still being recorded?		* Random hangs and timeouts when running recipetool oe-selftest tests - and killing the hung bitbake instance (usually a bitbake -e) leaves the server thinking that the client is still connected - something to do with the client token still being recorded?

PaulEggleton at 04:18, 1 December 2016

2016-12-01T04:18:53Z

← Older revision		Revision as of 04:18, 1 December 2016
Line 33:		Line 33:
	* If you shutdown() before the current command finishes, you get stuck in a bitbake.lock wait		* If you shutdown() before the current command finishes, you get stuck in a bitbake.lock wait
	* Still two failures when running oe-selftest --run-tests devtool; one is due to hash mismatch error messages, the other file checksum issues with memres (see "Memres issues" below)		* Still two failures when running oe-selftest --run-tests devtool; one is due to hash mismatch error messages, the other file checksum issues with memres (see "Memres issues" below)
	* Add missing API functions ~~e.g. for executing tasks - currently devtool extract is using run_command(), there should really be a dedicated function for that~~		* Add missing API functions?
	* Some means of cleaning up accumulated datastore objects on the server side?		* Some means of cleaning up accumulated datastore objects on the server side?
	* Run more tests with memres		* Run more tests with memres

PaulEggleton at 03:02, 1 December 2016

2016-12-01T03:02:24Z

← Older revision		Revision as of 03:02, 1 December 2016
Line 20:		Line 20:

	* Spew of hash mismatches when running devtool extract. RP suggests not using real cache in buildFile() which may help, but then we still seem to trip the mismatch detection elsewhere		* Spew of hash mismatches when running devtool extract. RP suggests not using real cache in buildFile() which may help, but then we still seem to trip the mismatch detection elsewhere
	* Still two failures when running oe-selftest --run-tests devtool; one is due to hash mismatch error messages, the other file checksum issues with memres (see "Memres issues" below)
	* If client exits in the middle of an async command, server is left in a broken state (e.g. tinfoil.functiondoesntexist() after tinfoil.runCommand('buildFile'...)		* If client exits in the middle of an async command, server is left in a broken state (e.g. tinfoil.functiondoesntexist() after tinfoil.runCommand('buildFile'...)
	* Random hangs and timeouts when running recipetool oe-selftest tests - and killing the hung bitbake instance (usually a bitbake -e) leaves the server thinking that the client is still connected - something to do with the client token still being recorded?		* Random hangs and timeouts when running recipetool oe-selftest tests - and killing the hung bitbake instance (usually a bitbake -e) leaves the server thinking that the client is still connected - something to do with the client token still being recorded?
			* Can't Ctrl+C out when waiting for bitbake.lock
			* Can't Ctrl+C out of event loop? e.g. comment out commandfailed handling in _extract_source
	* Errors on server side during a command cause devtool to basically lock up - in fact even an error on the client side will do that :( (e.g. tinfoil.functiondoesntexist()). Haven't checked to see where the code is stuck, possibly waiting for an event that will never come.		* Errors on server side during a command cause devtool to basically lock up - in fact even an error on the client side will do that :( (e.g. tinfoil.functiondoesntexist()). Haven't checked to see where the code is stuck, possibly waiting for an event that will never come.

	* conf/devtool.conf written by something?! (ctrl+c'd / failed tests?)		* conf/devtool.conf written by something?! (ctrl+c'd / failed tests?)

	* Need to have API for properly resolving provides (which get_recipe_file() should use)		* Need to have API for properly resolving provides (which get_recipe_file() should use)

	* No way to receive events sent during parsing?		* No way to receive events sent during parsing?
	* Still getting configParsed event even though filtered out?!		* Still getting configParsed event even though filtered out?!
	* set_event_mask() doesn't exclude any pre-generated events?		* set_event_mask() doesn't exclude any pre-generated events?
	* set_event_mask() still apparently not working		* set_event_mask() still apparently not working

	* Can't Ctrl+C out when waiting for bitbake.lock
	* If you shutdown() before the current command finishes, you get stuck in a bitbake.lock wait		* If you shutdown() before the current command finishes, you get stuck in a bitbake.lock wait
	* ~~Connecting~~ to ~~bitbake server in~~ memres ~~mode fails -> not very well handled error~~		* Still two failures when running oe-selftest --run-tests devtool; one is due to hash mismatch error messages, the other file checksum issues with memres (see "Memres issues" below)

	* Can't ctrl+c out of event loop? e.g. comment out commandfailed handling in _extract_source

	* Add missing API functions e.g. for executing tasks - currently devtool extract is using run_command(), there should really be a dedicated function for that		* Add missing API functions e.g. for executing tasks - currently devtool extract is using run_command(), there should really be a dedicated function for that
	* Some means of cleaning up accumulated datastore objects on the server side?		* Some means of cleaning up accumulated datastore objects on the server side?
Line 57:		Line 49:
	ERROR: Could not connect to server 127.0.0.1:46777		ERROR: Could not connect to server 127.0.0.1:46777
	: <ProtocolError for 127.0.0.1:46777/: 503 Service Unavailable>		: <ProtocolError for 127.0.0.1:46777/: 503 Service Unavailable>
			* Connecting to bitbake server in memres mode fails -> not very well handled error

	=== Annoyances ===		=== Annoyances ===

PaulEggleton at 02:03, 30 November 2016

2016-11-30T02:03:01Z

← Older revision		Revision as of 02:03, 30 November 2016
Line 45:		Line 45:
	* Run more tests with memres		* Run more tests with memres
	* Check/improve performance		* Check/improve performance
			* Tidy up commits

	=== Memres issues ===		=== Memres issues ===

PaulEggleton at 22:26, 28 November 2016

2016-11-28T22:26:57Z

← Older revision		Revision as of 22:26, 28 November 2016
Line 48:		Line 48:
	=== Memres issues ===		=== Memres issues ===
	* Cache is reloaded on every execution with memres		* Cache is reloaded on every execution with memres
	* bitbake -e shows no history with memres (bug 10730)		* bitbake -e shows no history with memres ([https://bugzilla.yoctoproject.org/show_bug.cgi?id=10730 bug 10730])
	* --observe-only doesn't seem to work (bug 10731)		* --observe-only doesn't seem to work ([https://bugzilla.yoctoproject.org/show_bug.cgi?id=10731 bug 10731])
	* File checksum changes don't rerun tasks (bug 10732) - breaks oe-selftest --run-tests devtool.DevtoolTests.test_devtool_modify		* File checksum changes don't rerun tasks ([https://bugzilla.yoctoproject.org/show_bug.cgi?id=10732 bug 10732]) - breaks <code>oe-selftest --run-tests devtool.DevtoolTests.test_devtool_modify</code>
	* bitbake-cookerdaemon.log seems to be triggering config change notifications !!!		* bitbake-cookerdaemon.log seems to be triggering config change notifications !!!
	* If you try to run two non-observer clients at once you should be presented with a reasonable message, but at the moment you get:		* If you try to run two non-observer clients at once you should be presented with a reasonable message, but at the moment you get:

PaulEggleton at 22:25, 28 November 2016

2016-11-28T22:25:06Z

← Older revision		Revision as of 22:25, 28 November 2016
Line 19:		Line 19:
	== Issues / todo list ==		== Issues / todo list ==

	* Spew of ~~taskhash~~ mismatches when running devtool extract		* Spew of hash mismatches when running devtool extract. RP suggests not using real cache in buildFile() which may help, but then we still seem to trip the mismatch detection elsewhere
	* Still two failures when running oe-selftest --run-tests devtool; one is due to ~~taskhash~~ mismatch error messages, the other file checksum issues with memres (see "Memres issues" below)		* Still two failures when running oe-selftest --run-tests devtool; one is due to hash mismatch error messages, the other file checksum issues with memres (see "Memres issues" below)
	* If client exits in the middle of an async command, server is left in a broken state (e.g. tinfoil.functiondoesntexist() after tinfoil.runCommand('buildFile'...)		* If client exits in the middle of an async command, server is left in a broken state (e.g. tinfoil.functiondoesntexist() after tinfoil.runCommand('buildFile'...)
	* Random hangs and timeouts when running recipetool oe-selftest tests - and killing the hung bitbake instance (usually a bitbake -e) leaves the server thinking that the client is still connected - something to do with the client token still being recorded?		* Random hangs and timeouts when running recipetool oe-selftest tests - and killing the hung bitbake instance (usually a bitbake -e) leaves the server thinking that the client is still connected - something to do with the client token still being recorded?

PaulEggleton at 22:22, 28 November 2016

2016-11-28T22:22:09Z

← Older revision		Revision as of 22:22, 28 November 2016
Line 52:		Line 52:
	* File checksum changes don't rerun tasks (bug 10732) - breaks oe-selftest --run-tests devtool.DevtoolTests.test_devtool_modify		* File checksum changes don't rerun tasks (bug 10732) - breaks oe-selftest --run-tests devtool.DevtoolTests.test_devtool_modify
	* bitbake-cookerdaemon.log seems to be triggering config change notifications !!!		* bitbake-cookerdaemon.log seems to be triggering config change notifications !!!
			* If you try to run two non-observer clients at once you should be presented with a reasonable message, but at the moment you get:
			WARNING: Could not connect to server at 127.0.0.1:46777 (<ProtocolError for 127.0.0.1:46777/: 503 Service Unavailable>)
			ERROR: Could not connect to server 127.0.0.1:46777
			: <ProtocolError for 127.0.0.1:46777/: 503 Service Unavailable>

	=== Annoyances ===		=== Annoyances ===