Toaster and bitbake communications

This is a write up of some investigation I did into how toaster asks bitbake to do stuff, and how it hears about what bitbake is doing. It's not definitive and not guaranteed to be correct, but it might provide some pointers if you are working on toaster. If anyone knows better, please feel free to correct this.

Any paths given below are relative to the root of the poky/poky-contrib source tree.

toaster asking bitbake to do stuff

First off I wanted to figure out how clicking on the "Build" button in the toaster interface triggers a bitbake build.

toaster only asks bitbake to perform builds when in managed mode. This is the point I started from when doing this investigation. I think analysis mode is very similar, except toaster doesn't ask bitbake to do anything, it just listens to stuff which is already happening.

runbuilds

The conversation with bitbake is handled via the runbuilds command (bitbake/lib/toaster/bldcontrol/management/commands/runbuilds.py). This command is run in a loop once every second from the toaster start script (bitbake/bin/toaster).

The runbuilds.schedule() function looks for BuildRequests in the database. A BuildRequest is created each time you press a "Build" button in the toaster web interface. Those build requests are then used to invoke the "real" builds in bitbake.

• schedule() gets a localhostbecontroller (be = build environment) instance (there is a becontroller for remote bitbakes, but I don't think the implementation is complete) and assigns it to a variable bec.

• schedule() then calls bec.triggerBuild() for a build request which is in the BuildRequest.REQ_QUEUED state. Only one build request gets picked up each time the runbuilds script runs: the one with the largest ID.

• schedule() also creates a build identification variable for each build request, combining the primary key of the build request with the primary key of the build environment controller's build environment (!)).

localhostbecontroller

This is the build environment controller, which handles instances of build environments.

It is a subclass of BuildEnvironmentController.

• triggerBuild() calls getBBController(), which returns a BitBakeController instance (bbctrl). getBBController() actually instantiates the controller if it isn't already instantiated, passing it a "server" object. This server object is an instance of bb.server.xmlrpc.BitBakeXMLRPCClient().

• triggerBuild() then calls the BitBakeController build() method: bbctrl.build()

BitBakeController (bbctrl)

This is constructed with the server object, which is an XMLRPC connection to a BitBake server.

• build() invokes the "buildTargets" command on the connection using runCommand().

BitBakeXMLRPCClient (bitbake/lib/bb/server/xmlrpc.py)

This is a subclass of BitBakeServer.

It has a "connection" member, which has runCommand() invoked on it.

When constructed by getBBController(), initServer() is called as part of its construction. This is a method from BitBakeServer which sets the interface address (default: (localhost,0)) and creates an XMLRPCServer() using this interface address.

Once initServer() is done, establishConnection() is invoked, which creates the "real" socket.

The established connection (which runCommand() is invoked on) is a BitBakeXMLRPCServerConnection.

BitBakeServer

BitbakeServer is a subclass of BitBakeBaseServer (declared in bitbake/lib/bb/server/__init__.py); this actually has the addcooker() method which associates a bitbake cooker instance with the connection.

BitBakeBaseServer acts like a decorator around a server implementation; in our case, it's a BitBakeXMLRPCServerConnection.

Calling addcooker() on the BitBakeBaseServer also calls it on any server implementation (serverImpl) which is wrapped by BitBakeServer.

BitBakeXMLRPCServerConnection

This is a subclass of BitBakeBaseServerConnection.

When runCommand() is invoked on this, it's passed on to the cooker object associated with it, i.e. it actually invokes self.cooker.command.runCommand().

The cooker object is associated with the BitBakeXMLRPCServerConnection when the bitbake/lib/bb/main.py script runs.

main.py

The main script which starts a bitbake server and ui. This is what runs when you use "bitbake" from the command line.

toaster starts the bitbake server with the --server-only switch, which calls bitbake_main() in this file; this in turn calls the main() function.

This instantiates a bb.cooker.BBCooker and adds it to the server implementation via addcooker(). The cooker is what actually enables commands to be sent to the bitbake server.

BBCooker (bitbake/lib/bb/cooker.py)

This has a "command" property which is an instance of BBCommand; this is what runCommand() finally gets invoked on.

BBCommand

runCommand() calls a method from an instance of CommandSync (all the synchronous commands bitbake understands) or CommandAsync (the asynchronous ones), depending on the type of command passed to runCommand(). Both the CommandSync and CommandAsync instances are added to the BBCooker when it is created.

For example, runCommand("getVariable", ...) is invoked via CommandsSync.getVariable().

The commands which go through BBCommand.runCommand() now make their way to the bitbake server over XMLRPC.

toaster listening to what bitbake is doing

At this point, I realised I probably understood enough to see how bitbake was being invoked from toaster: asking toaster to start a build sends a "buildTargets" command to the bitbake XMLRPC server, via a rather indirect series of objects and method calls.

What I wanted to know now was how toaster listens to the result of that command. At a high level, I understood that it gathered events from the XMLRPC connection and converted them into database objects. However, I didn't really get the code path.

I started from BuildInfoHelper, which is where bitbake events are converted into toaster db objects.

BuildInfoHelper

This is passed build events from the toasterui.py which routes them off to the buildinfohelper based on the event types.

BuildInfoHelper is responsible for constructing toaster ORM objects from events. The BuildInfoHelper is constructed with a server (instance of BitBakeXMLRPCServerConnection in the case of toaster), so it can also interrogate the bitbake server for extra environmental data via getVariable().

BuildInfoHelper also tries to match the data from the build recipe events from toasterui.py to data in toaster's database, so that toaster can "learn" from the build, this means that we get recipe and package information from the build which otherwise we are unable to determine.

The task of matching the recipes and associated data is done when we get the recipe information from the store task event, this contains a path of the recipe e.g. "/home/yocto/cloned_layers/_mylayer_master.toaster_cloned/mydir/example.bb... the buildinfohelper then tries to find a layer in toaster that has a checkout directory location that starts with the one provided by the event.

It does this by asking the "bc" BuildController (which can only be localhostbecontroller.py at the moment as this is the only controller which contains the implementation) to return the git checkout path via 'getGitCloneDirectory' this returns what it believes was the git checkout base location, in our example hopefully something like "/home/yocto/cloned_layers/_mylayer_master.toaster_cloned" then the buildinfohelper adds the directory name (brl.dirpath) from the information in the build request layer, e.g. "mydir" after all this reconstruction we hope that the recipe path we got from the event starts with the reconstructed path. e.g. Does "/home/yocto/cloned_layers/_mylayer_master.toaster_cloned/mydir/example.bb" start with "/home/yocto/cloned_layers/_mylayer_master.toaster_cloned/mydir/" if yes then we have a layer in toaster to which we can associate the information from the build to.

One problem here is that the getGitCloneDirectory function a) doesn't exist in the other hostcontrollers and b) doesn't return the correct value

toasterui.py

This has a main() function which sets up an event listener loop.

main() is passed a server, eventHandler and params. The eventHandler is the object which listens to bitbake events.

toasterui is called from main.py (see below).

main.py (bitbake/lib/bb/main.py)

The toasterui.py main() method is invoked from main.py.

It's passed two parameters:

1. server_connection.connection
2. server_connection.events

The second of these is the eventHandler which is set up in a loop in toasterui.py.

server_connection comes from a call to establishConnection() in main.py. The server is constructed via a servermodule, which is dynamically chosen in main.py according to the parameters used to invoke it (-t). It will either be "process" or "xmlrpc".

The UI type is set in the main.py script via the -u option.

toaster invokes bitbake with: -t xmlrpc -u toasterui

which means that we get the toasterui.main() called, and we get an xmlrpc bitbake server.

Back to server_connection.events...

This refers to an xmlrpc bitbake server connection's events object.

So toasterui.main() gets:

1. server_connection.connection => BitBakeXMLRPCServerConnection.connection
2. server_connection.events => BitBakeXMLRPCServerConnection.events

The events object is an instance of uievent.BBUIEventQueue in our case, as we're using toasterui.

uievent.BBUIEventQueue (bitbake/lib/bb/ui/uievent.py)

This is our event handler ("events") in toasterui.main(), which is receiving events on the XMLRPC connection (see later).

toasterui.main() calls events.waitEvent(0.25), which looks for events on the queue every 0.25s. If the queue has events, one is popped off.

Each event popped from the queue is passed to uihelper.BBUIHelper.eventHandler(), which adds build tracking information (how many packages built, tasks completed etc.)

The event then goes to the BuildInfoHelper, where it gets stored in toaster's database.

How do events get on BBUIEventQueue?

BBUIEventQueue is instantiated on the BitBakeXMLRPCServerConnection object.

It is passed a BBServer, which is an xmlrpclib.ServerProxy (from the Python standard library).

The ServerProxy has a method corresponding to each method on the server it is proxying for; in this case, the proxied server is a BBServer, so the proxy has the same methods as BBServer.

Event handling is set up by calling self.BBServer.registerEventHandler() from BBUIEventQueue.

registerEventHandler() is defined in bitbake/lib/bb/server/xmlrpc.py, BitBakeServerCommands. This in turn calls bb.event.register_UIHandler, defined in bitbake/lib/bb/event.py.

When an event is fired, each handler registered for it is invoked with that event (in event.py).

The main function for firing events in event.py is fire_from_worker(), which is called from bitbake/lib/bb/runqueue.py.

Events are constructed from xmlrpc messages coming from the bitbake server (see runqueue.py, runQueuePipe.read()).

Conclusion

At this point, I felt I understood enough about how events are processed for my purpose, so I didn't dig any further. I knew where I could amend an event to add/remove properties on it, which is what I was after.