Toaster and bitbake communications - Revision history

Michael Wood at 15:47, 30 March 2016

2016-03-30T15:47:12Z

← Older revision		Revision as of 15:47, 30 March 2016
Line 1:		Line 1:
			[[Category:Toaster]]

	This is a write up of some investigation I did into how toaster asks bitbake to do stuff, and how it hears about what bitbake is doing. It's not definitive and not guaranteed to be correct, but it might provide some pointers if you are working on toaster. If anyone knows better, please feel free to correct this.		This is a write up of some investigation I did into how toaster asks bitbake to do stuff, and how it hears about what bitbake is doing. It's not definitive and not guaranteed to be correct, but it might provide some pointers if you are working on toaster. If anyone knows better, please feel free to correct this.

Elliot Smith: /* How do toasterui + buildinfohelper manage events? */

2015-11-19T14:55:48Z

How do toasterui + buildinfohelper manage events?

Elliot Smith: /* How do events get on BBUIEventQueue? */

2015-11-19T14:55:09Z

How do events get on BBUIEventQueue?

@@ Line 174: / Line 174: @@
 Events are constructed from xmlrpc messages coming from the bitbake server (see runqueue.py, runQueuePipe.read()).
 == Conclusion ==
 At this point, I felt I understood enough about how events are processed for my purpose, so I didn't dig any further. I knew where I could amend an event to add/remove properties on it, which is what I was after.

Michael Wood: /* runbuilds */

2015-09-17T10:50:14Z

runbuilds

← Older revision		Revision as of 10:50, 17 September 2015
Line 13:		Line 13:
	The conversation with bitbake is handled via the runbuilds command (bitbake/lib/toaster/bldcontrol/management/commands/runbuilds.py). This command is run in a loop once every second from the toaster start script (bitbake/bin/toaster).		The conversation with bitbake is handled via the runbuilds command (bitbake/lib/toaster/bldcontrol/management/commands/runbuilds.py). This command is run in a loop once every second from the toaster start script (bitbake/bin/toaster).

	The runbuilds.schedule() function looks for BuildRequests in the database. A BuildRequest is created each time you press a "Build" button in the toaster web interface. Those build requests are ~~then used~~ to ~~invoke~~ the ~~"real" builds~~ in ~~bitbake~~.		The runbuilds.schedule() function looks for BuildRequests in the database. A '''BuildRequest''' is created each time you press a "Build" button in the toaster web interface. Those build requests are created in orm/model.py schedule_build on the project object. When creating this BuildRequest also created are BRLayer for each of the layers in your project, BRTarget for the target to be built and BRBitbake for the version of bitbake to use, these are all defined in bldcontrol/models.py, and contain copies of the information from the project.

			The localhostbecontroller takes this BuildRequest

	* schedule() gets a localhostbecontroller (be = build environment) instance (there is a becontroller for remote bitbakes, but I don't think the implementation is complete) and assigns it to a variable bec.		* schedule() gets a localhostbecontroller (be = build environment) instance (there is a becontroller for remote bitbakes, but I don't think the implementation is complete) and assigns it to a variable bec.

Michael Wood: /* localhostbecontroller */

2015-09-16T17:51:31Z

localhostbecontroller

← Older revision		Revision as of 17:51, 16 September 2015
Line 23:		Line 23:
	=== localhostbecontroller ===		=== localhostbecontroller ===

	This is the build environment controller, which handles instances of build environments.		This is the build environment controller, which handles instances of build environments. It is also responsible for cloning any layers that are in the build request.

	It is a subclass of BuildEnvironmentController.		It is a subclass of BuildEnvironmentController.

Michael Wood: /* BuildInfoHelper */

2015-09-16T17:32:54Z

BuildInfoHelper

← Older revision		Revision as of 17:32, 16 September 2015
Line 105:		Line 105:
	It does this by asking the "bc" BuildController (which can only be localhostbecontroller.py at the moment as this is the only controller which contains the implementation) to return the git checkout path via 'getGitCloneDirectory' this returns what it believes was the git checkout base location, in our example hopefully something like "/home/yocto/cloned_layers/_mylayer_master.toaster_cloned" then the buildinfohelper adds the directory name (brl.dirpath) from the information in the build request layer, e.g. "mydir" after all this reconstruction we hope that the recipe path we got from the event starts with the reconstructed path. e.g. Does "/home/yocto/cloned_layers/_mylayer_master.toaster_cloned/mydir/example.bb" start with "/home/yocto/cloned_layers/_mylayer_master.toaster_cloned/mydir/" if yes then we have a layer in toaster to which we can associate the information from the build to.		It does this by asking the "bc" BuildController (which can only be localhostbecontroller.py at the moment as this is the only controller which contains the implementation) to return the git checkout path via 'getGitCloneDirectory' this returns what it believes was the git checkout base location, in our example hopefully something like "/home/yocto/cloned_layers/_mylayer_master.toaster_cloned" then the buildinfohelper adds the directory name (brl.dirpath) from the information in the build request layer, e.g. "mydir" after all this reconstruction we hope that the recipe path we got from the event starts with the reconstructed path. e.g. Does "/home/yocto/cloned_layers/_mylayer_master.toaster_cloned/mydir/example.bb" start with "/home/yocto/cloned_layers/_mylayer_master.toaster_cloned/mydir/" if yes then we have a layer in toaster to which we can associate the information from the build to.

	One problem here is that the getGitCloneDirectory function a) doesn't exist in the other hostcontrollers and b) doesn't return the correct value		One problem here is that the getGitCloneDirectory function a) doesn't exist in the other hostcontrollers and b) doesn't always return the correct value due to some special case

	=== toasterui.py ===		=== toasterui.py ===

Michael Wood: /* BuildInfoHelper */

2015-09-16T16:54:53Z

BuildInfoHelper

← Older revision		Revision as of 16:54, 16 September 2015
Line 95:		Line 95:
	=== BuildInfoHelper ===		=== BuildInfoHelper ===

	This is passed build events in the toasterui.py ~~script~~.		This is passed build events from the toasterui.py which routes them off to the buildinfohelper based on the event types.

	BuildInfoHelper is responsible for constructing toaster ORM objects from events. The BuildInfoHelper is constructed with a server (instance of BitBakeXMLRPCServerConnection in the case of toaster), so it can also interrogate the bitbake server for extra environmental data via getVariable().		BuildInfoHelper is responsible for constructing toaster ORM objects from events. The BuildInfoHelper is constructed with a server (instance of BitBakeXMLRPCServerConnection in the case of toaster), so it can also interrogate the bitbake server for extra environmental data via getVariable().

			BuildInfoHelper also tries to match the data from the build recipe events from toasterui.py to data in toaster's database, so that toaster can "learn" from the build, this means that we get recipe and package information from the build which otherwise we are unable to determine.

			The task of matching the recipes and associated data is done when we get the recipe information from the store task event, this contains a path of the recipe e.g. "/home/yocto/cloned_layers/_mylayer_master.toaster_cloned/mydir/example.bb... the buildinfohelper then tries to find a layer in toaster that has a checkout directory location that starts with the one provided by the event.

			It does this by asking the "bc" BuildController (which can only be localhostbecontroller.py at the moment as this is the only controller which contains the implementation) to return the git checkout path via 'getGitCloneDirectory' this returns what it believes was the git checkout base location, in our example hopefully something like "/home/yocto/cloned_layers/_mylayer_master.toaster_cloned" then the buildinfohelper adds the directory name (brl.dirpath) from the information in the build request layer, e.g. "mydir" after all this reconstruction we hope that the recipe path we got from the event starts with the reconstructed path. e.g. Does "/home/yocto/cloned_layers/_mylayer_master.toaster_cloned/mydir/example.bb" start with "/home/yocto/cloned_layers/_mylayer_master.toaster_cloned/mydir/" if yes then we have a layer in toaster to which we can associate the information from the build to.

			One problem here is that the getGitCloneDirectory function a) doesn't exist in the other hostcontrollers and b) doesn't return the correct value

	=== toasterui.py ===		=== toasterui.py ===

Elliot Smith: /* main.py (bitbake/lib/bb/main.py) */

2015-09-11T14:53:16Z

main.py (bitbake/lib/bb/main.py)

← Older revision		Revision as of 14:53, 11 September 2015
Line 113:		Line 113:
	It's passed two parameters:		It's passed two parameters:

	1. server_connection.connection		# server_connection.connection
	2. server_connection.events		# server_connection.events

	The second of these is the eventHandler which is set up in a loop in toasterui.py.		The second of these is the eventHandler which is set up in a loop in toasterui.py.
Line 132:		Line 132:
	So toasterui.main() gets:		So toasterui.main() gets:

	1. server_connection.connection => BitBakeXMLRPCServerConnection.connection		# server_connection.connection => BitBakeXMLRPCServerConnection.connection
	2. server_connection.events => BitBakeXMLRPCServerConnection.events		# server_connection.events => BitBakeXMLRPCServerConnection.events

	The events object is an instance of uievent.BBUIEventQueue in our case, as we're using toasterui.		The events object is an instance of uievent.BBUIEventQueue in our case, as we're using toasterui.

Elliot Smith: Created page with "This is a write up of some investigation I did into how toaster asks bitbake to do stuff, and how it hears about what bitbake is doing. It's not definitive and not guaranteed ..."

2015-09-11T14:51:14Z

Created page with "This is a write up of some investigation I did into how toaster asks bitbake to do stuff, and how it hears about what bitbake is doing. It's not definitive and not guaranteed ..."

New page

This is a write up of some investigation I did into how toaster asks bitbake to do stuff, and how it hears about what bitbake is doing. It's not definitive and not guaranteed to be correct, but it might provide some pointers if you are working on toaster. If anyone knows better, please feel free to correct this.

Any paths given below are relative to the root of the poky/poky-contrib source tree.

== toaster asking bitbake to do stuff ==

First off I wanted to figure out how clicking on the "Build" button in the toaster interface triggers a bitbake build.

toaster only asks bitbake to perform builds when in managed mode. This is the point I started from when doing this investigation. I think analysis mode is very similar, except toaster doesn't ask bitbake to do anything, it just listens to stuff which is already happening.

=== runbuilds ===

The conversation with bitbake is handled via the runbuilds command (bitbake/lib/toaster/bldcontrol/management/commands/runbuilds.py). This command is run in a loop once every second from the toaster start script (bitbake/bin/toaster).

The runbuilds.schedule() function looks for BuildRequests in the database. A BuildRequest is created each time you press a "Build" button in the toaster web interface. Those build requests are then used to invoke the "real" builds in bitbake.

* schedule() gets a localhostbecontroller (be = build environment) instance (there is a becontroller for remote bitbakes, but I don't think the implementation is complete) and assigns it to a variable bec.

* schedule() then calls bec.triggerBuild() for a build request which is in the BuildRequest.REQ_QUEUED state. Only one build request gets picked up each time the runbuilds script runs: the one with the largest ID.

* schedule() also creates a build identification variable for each build request, combining the primary key of the build request with the primary key of the build environment controller's build environment (!)).

=== localhostbecontroller ===

This is the build environment controller, which handles instances of build environments.

It is a subclass of BuildEnvironmentController.

* triggerBuild() calls getBBController(), which returns a BitBakeController instance (bbctrl). getBBController() actually instantiates the controller if it isn't already instantiated, passing it a "server" object. This server object is an instance of bb.server.xmlrpc.BitBakeXMLRPCClient().

* triggerBuild() then calls the BitBakeController build() method: bbctrl.build()

=== BitBakeController (bbctrl) ===

This is constructed with the server object, which is an XMLRPC connection to a BitBake server.

* build() invokes the "buildTargets" command on the connection using runCommand().

=== BitBakeXMLRPCClient (bitbake/lib/bb/server/xmlrpc.py) ===

This is a subclass of BitBakeServer.

It has a "connection" member, which has runCommand() invoked on it.

When constructed by getBBController(), initServer() is called as part of its construction. This is a method from BitBakeServer which sets the interface address (default: (localhost,0)) and creates an XMLRPCServer() using this interface address.

Once initServer() is done, establishConnection() is invoked, which creates the "real" socket.

The established connection (which runCommand() is invoked on) is a BitBakeXMLRPCServerConnection.

=== BitBakeServer ===

BitbakeServer is a subclass of BitBakeBaseServer (declared in bitbake/lib/bb/server/__init__.py); this actually has the addcooker() method which associates a bitbake cooker instance with the connection.

BitBakeBaseServer acts like a decorator around a server implementation; in our case, it's a BitBakeXMLRPCServerConnection.

Calling addcooker() on the BitBakeBaseServer also calls it on any server implementation (serverImpl) which is wrapped by BitBakeServer.

=== BitBakeXMLRPCServerConnection ===

This is a subclass of BitBakeBaseServerConnection.

When runCommand() is invoked on this, it's passed on to the cooker object associated with it, i.e. it actually invokes self.cooker.command.runCommand().

The cooker object is associated with the BitBakeXMLRPCServerConnection when the bitbake/lib/bb/main.py script runs.

=== main.py ===

The main script which starts a bitbake server and ui. This is what runs when you use "bitbake" from the command line.

toaster starts the bitbake server with the --server-only switch, which calls bitbake_main() in this file; this in turn calls the main() function.

This instantiates a bb.cooker.BBCooker and adds it to the server implementation via addcooker(). The cooker is what actually enables commands to be sent to the bitbake server.

=== BBCooker (bitbake/lib/bb/cooker.py) ===

This has a "command" property which is an instance of BBCommand; this is what runCommand() finally gets invoked on.

=== BBCommand ===

runCommand() calls a method from an instance of CommandSync (all the synchronous commands bitbake understands) or CommandAsync (the asynchronous ones), depending on the type of command passed to runCommand(). Both the CommandSync and CommandAsync instances are added to the BBCooker when it is created.

For example, runCommand("getVariable", ...) is invoked via CommandsSync.getVariable().

The commands which go through BBCommand.runCommand() now make their way to the bitbake server over XMLRPC.

== toaster listening to what bitbake is doing ==

At this point, I realised I probably understood enough to see how bitbake was being invoked from toaster: asking toaster to start a build sends a "buildTargets" command to the bitbake XMLRPC server, via a rather indirect series of objects and method calls.

What I wanted to know now was how toaster listens to the result of that command. At a high level, I understood that it gathered events from the XMLRPC connection and converted them into database objects. However, I didn't really get the code path.

I started from BuildInfoHelper, which is where bitbake events are converted into toaster db objects.

=== BuildInfoHelper ===

This is passed build events in the toasterui.py script.

BuildInfoHelper is responsible for constructing toaster ORM objects from events. The BuildInfoHelper is constructed with a server (instance of BitBakeXMLRPCServerConnection in the case of toaster), so it can also interrogate the bitbake server for extra environmental data via getVariable().

=== toasterui.py ===

This has a main() function which sets up an event listener loop.

main() is passed a server, eventHandler and params. The eventHandler is the object which listens to bitbake events.

toasterui is called from main.py (see below).

=== main.py (bitbake/lib/bb/main.py) ===

The toasterui.py main() method is invoked from main.py.

It's passed two parameters:

1. server_connection.connection
2. server_connection.events

The second of these is the eventHandler which is set up in a loop in toasterui.py.

server_connection comes from a call to establishConnection() in main.py. The server is constructed via a servermodule, which is dynamically chosen in main.py according to the parameters used to invoke it (-t). It will either be "process" or "xmlrpc".

The UI type is set in the main.py script via the -u option.

toaster invokes bitbake with: -t xmlrpc -u toasterui

which means that we get the toasterui.main() called, and we get an xmlrpc bitbake server.

Back to server_connection.events...

This refers to an xmlrpc bitbake server connection's events object.

So toasterui.main() gets:

1. server_connection.connection => BitBakeXMLRPCServerConnection.connection
2. server_connection.events => BitBakeXMLRPCServerConnection.events

The events object is an instance of uievent.BBUIEventQueue in our case, as we're using toasterui.

=== uievent.BBUIEventQueue (bitbake/lib/bb/ui/uievent.py) ===

This is our event handler ("events") in toasterui.main(), which is receiving events on the XMLRPC connection (see later).

toasterui.main() calls events.waitEvent(0.25), which looks for events on the queue every 0.25s. If the queue has events, one is popped off.

Each event popped from the queue is passed to uihelper.BBUIHelper.eventHandler(), which adds build tracking information (how many packages built, tasks completed etc.)

The event then goes to the BuildInfoHelper, where it gets stored in toaster's database.

==== How do events get on BBUIEventQueue? ====

BBUIEventQueue is instantiated on the BitBakeXMLRPCServerConnection object.

It is passed a BBServer, which is an xmlrpclib.ServerProxy (from the Python standard library).

The ServerProxy has a method corresponding to each method on the server it is proxying for; in this case, the proxied server is a BBServer, so the proxy has the same methods as BBServer.

Event handling is set up by calling self.BBServer.registerEventHandler() from BBUIEventQueue.

registerEventHandler() is defined in bitbake/lib/bb/server/xmlrpc.py, BitBakeServerCommands. This in turn calls bb.event.register_UIHandler, defined in bitbake/lib/bb/event.py.

When an event is fired, each handler registered for it is invoked with that event (in event.py).

The main function for firing events in event.py is fire_from_worker(), which is called from bitbake/lib/bb/runqueue.py.

Events are constructed from xmlrpc messages coming from the bitbake server (see runqueue.py, runQueuePipe.read()).

== Conclusion ==

At this point, I felt I understood enough about how events are processed for my purpose, so I didn't dig any further. I knew where I could amend an event to add/remove properties on it, which is what I was after.

← Older revision		Revision as of 14:55, 19 November 2015
Line 185:		Line 185:
	The best example of this comes from bitbake's Task* events. We receive two events for a task in buildinfohelper:		The best example of this comes from bitbake's Task* events. We receive two events for a task in buildinfohelper:

	1. The event notifying that the task started, e.g. TaskStarted, runQueueTaskStarted, sceneQueueTaskStarted		# The event notifying that the task started, e.g. TaskStarted, runQueueTaskStarted, sceneQueueTaskStarted
	2. The end state of the task, e.g. TaskFailedSilent, TaskCompleted, runQueueTaskFailed, sceneQueueTaskFailed, runQueueTaskCompleted, sceneQueueTaskCompleted, runQueueTaskSkipped		# The end state of the task, e.g. TaskFailedSilent, TaskCompleted, runQueueTaskFailed, sceneQueueTaskFailed, runQueueTaskCompleted, sceneQueueTaskCompleted, runQueueTaskSkipped

	Because Toaster represents a Task as a single entity, with an outcome state, we can't add a complete Task record when a task starts: we add a partial Task record, then update that when the task end event is received.		Because Toaster represents a Task as a single entity, with an outcome state, we can't add a complete Task record when a task starts: we add a partial Task record, then update that when the task end event is received.
Line 211:		Line 211:
	Here's an example of the workflow to make it clearer:		Here's an example of the workflow to make it clearer:

	1. toasterui receives a BuildStarted event and passes it to buildinfohelper.		# toasterui receives a BuildStarted event and passes it to buildinfohelper.
	2. buildinfohelper creates a database record B for the new build, and stores it as a property on itself.		# buildinfohelper creates a database record B for the new build, and stores it as a property on itself.
	3. toasterui receives a TaskStarted event with name "foo" and file "/bar/bar/humbug"; it passes the event to buildinfohelper.		# toasterui receives a TaskStarted event with name "foo" and file "/bar/bar/humbug"; it passes the event to buildinfohelper.
	4. buildinfohelper stores the partial task in the database as T, associating it with the build record B; it also stores T under the key "/bar/bar/humbug:foo" in taskdata.		# buildinfohelper stores the partial task in the database as T, associating it with the build record B; it also stores T under the key "/bar/bar/humbug:foo" in taskdata.
	5. toasterui receives a TaskCompleted event with name "foo" and file "/bar/bar/humbug"; it passes the event to buildinfohelper.		# toasterui receives a TaskCompleted event with name "foo" and file "/bar/bar/humbug"; it passes the event to buildinfohelper.
	6. buildinfohelper marries up the new TaskCompleted event with the partial record T, by matching the new event's key "/bar/bar/humbug:foo" with the key for the TaskStarted event which is already in taskdata (see 4); T is updated in the database.		# buildinfohelper marries up the new TaskCompleted event with the partial record T, by matching the new event's key "/bar/bar/humbug:foo" with the key for the TaskStarted event which is already in taskdata (see 4); T is updated in the database.
	7. toasterui receives a BuildCompleted event and passes it to buildinfohelper.		# toasterui receives a BuildCompleted event and passes it to buildinfohelper.
	8. buildinfohelper assumes that the BuildCompleted event applies to the build B already stored in its internal state. It updates the database record for B with data about when the build ended, build artifacts etc.		# buildinfohelper assumes that the BuildCompleted event applies to the build B already stored in its internal state. It updates the database record for B with data about when the build ended, build artifacts etc.
	9. toasterui creates a new buildinfohelper, ready to deal with the next build.		# toasterui creates a new buildinfohelper, ready to deal with the next build.

	backlog is a list of log events which haven't yet been attached to a build; if a log event occurs before the build has been saved, it is added to the list; then, once a BuildStarted event has occurred, the list of events in the backlog is added to the database and associated with that build.		backlog is a list of log events which haven't yet been attached to a build; if a log event occurs before the build has been saved, it is added to the list; then, once a BuildStarted event has occurred, the list of events in the backlog is added to the database and associated with that build.