Toaster 1.7 design

A couple of videos to kick off things

The following videos illustrate some of the features we are thinking of adding to Toaster during the 1.7 release cycle. When watching them, keep in mind that this is all very rough: more like sketches to help us start a conversation about scope than proper designs. Chances are the final designs will be completely different.

About projects - media:Building-1.7.mov
Starting a build - media:How-to-build.mov

Feature list

Ability to create new projects

This is the ability to use the Toaster interface to create new project directories, manage the content, and initiate the builds of selectable target images.

Discussion

[DAVE] What is a typical workflow that gets to the build project part? I see the configure screens but not the actualy build launch.

[DAVE] Does clicking 'new build' fire off a build? If so, which configuration is built?

[BELEN] Heh, I completely forgot to show that in the first video. Here is part 2 showing how to start a build and answering (I hope) both questions.

[DAVE] Can projects be configured, primed for builds, but not built?

[BELEN] Yes.

[DAVE] Will the user be able to configure several builds, then queue them all to build?

[BELEN] Users might create more than one project then start a build for each of them by clicking the "build" button in each project page, but nothing more sophisticated than that for the time being. Here is where things can get tricky. Where does Toaster stop and something like Buildbot or Jenkins start? My take is that we want to work alongside and enhance build automation systems, not replace them.

[DAVE] What is the size of the build variable list? It's hard to figure from the conf/local.conf file. If it is a small list, like 10, should we provide better and variable specific UI widgets rather than just edit boxes, depending on the variable's "type" (eg browse button with any path variable like SSTATE_DIR)?

[BELEN] This is one of the questions we should start working on straight away. In informal chats we have spoken about a split between "build variables" (the ones that affect the build and therefore you can edit via Toaster) and "infrastructure variables" (to call them something, that affect the build infrastructure and you cannot edit via Toaster. Things like PARALLEL_MAKE for example). We should start getting into details about that split as soon as possible, and determine which variables we are going to expose via Toaster to be edited. Once again, Paul's feedback here would be good (RP's probably as well).

Regarding variable "types", I had a brief conversation with RP recently about this issue. It is unlikely that the meta data will provide us with information about the type of input expected by a variable. If and how the Toaster interface should fill that gap is up for discussion. From my perspective, we should do as much as we can to help users assign valid values to their variables.

[DAVE] I don't recall a user entry on your videos that specify the project directory for the build. This must be part of the initial release. I don't recommend making it an edited data file option, eg not via in settings.py.

[BELEN] When you run Toaster locally you can configure the location of your build directory as you wish, but when Toaster is running remotely, do we want to let people select the location of their build directory?

[DAVE] Can we make any selection on any page 'stick' with cookies or 'unstick' with a 'reset to defaults' button, in addition to propagating laborious configuration setups inputs via 'clone configuration'?

[BELEN] We will need to overcome my personal hate of 'reset to defaults' options :) They are very dangerous if we don't provide an 'undo' option as well: they might wipe your configuration and revert it to the rather useless default one, losing a bunch of work you did beforehand. Any configuration changes you make are saved: in that sense, they are 'sticky'. If I create project A, then I add 2 layers to the project configuration, select a machine and 3 targets, all those options are saved to the project configuration. You don't need to add the 2 layers, select the machine and the 3 targets every time you want to run a build.

Hide builds from a project

This option simply removes or hides selected builds from the Toaster display, for example builds quickly halted because the owner realized that they missed a setting. The database records and the project on the disk is untouched. This un-clutters the display, yet requires little programing (and development time).

Discussion

[BELEN] I have no strong feelings against this one, but we need to bear in mind that:

• It will probably be hard to allow hiding in bulk, since the reasons why someone might want to hide a build could be wildly different. So this will have to be done on a build by build basis.

• This also involves providing a way to reverse the hiding action, i.e. see hidden builds and show them again.

Delete builds from a project

This option is where a selected build and all its related database records are removed from the database. This removes the database overhead for this unwanted builds. The project on the disk is untouched. Since the database should be normalized, this operation should be clean and not interfere with other builds. I believe we have a command line version of this operation with Toaster-1.6.

Discussion

[Alex] Technically, this is straightforward. In terms of policy access, we need to define one; I propose that only the project owner is allowed to delete builds. Impersonation is trivial right now, but once we implement authentication, bricks will fall in the right place in terms of access control.

Ability to add / remove layers

Add and delete from the project layer directories. These layers may be within repositories or in plain directories on the disk.

Discussion

[FARRELL] How would I upload my specialized device driver/library/system call handler/etc that necessarily needs to be part of the build? Would this be done by importing a layer?

[BELEN] Yes. For the 1.7 release, all customisation must be done via layers.

[DAVE] Are you constraining 'import layer' to a git repo path? Why not just a local directory that is not a git repo? In that case should we have a browse button as well as an edit box?

[BELEN] This is a really good question. I need the technical crowd to establish what's possible / what we want to do. We can then design accordingly.

Ability to identify / set project owner / user

A build farm manager (and their shared Toaster instance) may accept build sets from clients, in which case it is important to know who initiates and owns each specific build, for completion notifications and project space recycling.

Discussion

[FARRELL] You presented the notion of a user - in the movie it was 'Belen'. I think it may be necessary to formalize this idea with a login screen and at least rudimentary permission and protection. Here is a use case: Imagine a vendor deploys Toaster on a server which serves all of its customers. The customers are very diverse and are often competitors of one another and do not want their configuration/information/ideas shared with other vendor customers.

[BELEN] Very true: access control is a big deal and very important. However, we are not sure we can tackle this in the 1.7 release. That's why in the video I speak of "owners" instead of "users". Our "owner" concept derives from the need to identify who created a certain project, but they are not really users since, for example, they don't involve authentication (there is no password).

Here is a possible description of these "owners". Each client accessing a Toaster instance might have an owner linked to it. You don't explicitly create these owners: you just set them in the process of creating your first project. Then, we remember them. Owners involve only 3 pieces of information: a name, an optional email address and the client they are associated with. You can change the details at any time. So "owners" don't involve access control: they just tell me who created (and therefore owns) a certain project. You can have the same owner details in several clients: I might have a laptop and a builds machine, I access the same Toaster instance from both, and I can use the same owner name and email address in both.

The above opens some questions:

• can others edit the configuration of a project I own?
• can others start builds against a project I own?
• can others download artifacts of a project I own?
• can others even see the projects I own?

The simplest thing would be either:

• 'yes' to all of the above, but that means that people creating Toaster instances that can be accessed by more than one person have to be mindful of the fact that everything in them is open.

• 'no' to all of the above, and channel all collaboration through the export / import projects functionality, which is safer but also stifles co-operation.

Just to clarify, ownership as above is established per project, not per build.

[FARRELL] As much as I hate to suggest this...It may be necessary to introduce the notion of group access - Owner/Group/Others ala unix/linux. I think it may be necessary for users to be able to share their results while excluding others. I can imagine a number of use cases. Here is one:

       Two customers/users, Fu and Bar, work for the same company C. Each wants to
       be able to have personal 'experimental' builds but may need to share a 'production'
       build - react to errors and download results, etc. Of course, company C doesn't want
       company B to see their results, configurations, custom drivers, etc. But, Fu needs to
       see and have access to some of Bar's results. These problems can be solved with the
       introduction of the group access.

I realize this adds a layer of complexity which may be necessary. An alternative would be access control lists (acl). These can be even more flexible but a lot more complicated to implement, manage and maintain.

I think we should decide on an access control model soon and work out an implementation. Even with the current implementation, retrofitting will be difficult and somewhat tedious which will become even more so as we add more features.

[DAVID] The intermediate near term solution is to consider the 1.7 as working with only trusted users, in that they are all already within the same intranet. Under this limitation, if user Foo needs their work fully separate from user Bar, they could just have separate instances and databases of Toaster running, and use existing Linux security to accomplish this.

That said, we could provide for the default build/project view to default to the current "owner", where they would have to go out of their way to other people's work.

We could also consider an "audit log" that captures project creation (and especially project deletion), so that at least you know if someone stomped on your stuff :-)

[BELEN] I am with David on this one: I think we can find temporary alternatives. It's not that I disagree with Farrell, mind: he is completely right that we will need this. Problem is: we need to be mindful of what we can deliver with the resources we have.

Ability to identify build host and build directory

To support distributed build hosts sharing a common Toaster database, there needs to be a way in the database to identify which host (IP address and or host name) and local directory the project was built in.

Ability to download files (YP #6096)

While developers would presumably have access to all build files within the intranet for the build logs and generated target image files, there are cases where that is not so. For example, the owner may be using a Windows host or a portable device to access Toaster, and may wish to download and view the error logs to determine their next course of action. Also, it may be the case that the specific build host is accessible only to the build farm manager and not the individual project owner, and having Toaster serve the file(s) would resolve this barrier.

Discussion

[FARRELL] The ability to download the result of the build should be somewhere - maybe a column on the All Builds page?

[DAVID] As for things like file downloads, between Ravi and myself I think we will find time to get that one done since we both really want it!

[BELEN] Absolutely: in fact, this is something I might start designing straight away so that it can be implemented in 1.7M1. However, in order to do that, we need to reach agreement as to what a build artifact is:

• Is it just the rootfs files? Or do they include
• the cooker and task logs?
• shared state objects used?
• recipe files / patches?
• [yours here]

Share projects via configuration (export / import)

This is the capture and sharing of the minimal project configuration data, such that a remote developer can replicate the same project against their own Yocto Project installation.

Discussion

[DAVE] The video ended at the export project discussion. Is export just sending email, the conf file (or files) to someone, or is there be more to it?

[BELEN] I am not sure what the answer is: we need input from Paul and Alex here. In my head, the project configuration (not the builds) is exported to some kind of file(s) that I can then share with others in whichever way I see fit (email, file sharing, etc).

Toaster System Architectures

This is an iteration of the possible supported system architectures of a Toaster implementation. We should validate how the proposed design would work with each of these options.

1) Local host, local project, database in project, toaster and web server started from project

This is the default implementation, using the "source toaster [start|stop]" support.

2) Local host, Multiple local projects, shared local database, toaster and web server started separately

This is currently supported by (a) edits to the local.conf file to point to the shared database location, and (b) using special commands to start the shared Toaster and webserver instance.

3) Multiple hosts, Multiple local projects per host, shared network database, toaster and web server started with network database instance

Each host would either have its own installation of YP, or have a centralized copy shared over NFS. This implementation is currently not yet documented nor supported.

Open Bugzilla features / defects

This is what's open in Bugzilla right now. Apart from what's already assigned to 1.6.1, there are 2 issues that I am keen on getting fixed:

5187 - Same cooker log file name used for several builds
6095 - Retrieve full build information independently of task execution

Feel free to add to this list.

Future features

A list of features that have come up in discussion, but not for the 1.7 release.

Delete projects

This option deletes the project all the way down to the project's instance on the disk. This should be a safe operation unless a separate project is directly sharing its sstate cache. That would be an recommended usage, where a proper usage would a standalone shared sstate cache. There may be artifacts in that shared sstate cache that are specific and unique to the deleted project, but it is out of scope for this option to try to delete those as well.

Ability to add/remove recipes

This is the ability to add or remove recipes from the project's configuration.

Ability to import source (tarballs, repos)

This is the ability to bring external sources into the project and include them with wrapper recipes.

Ability to add/remove packages from image

This is the ability to add or remove specific binary packages from the respective image(s) of a project.

Share projects via Copy (all but "./tmp")

Project directories (all except for their local "tmp" directory) should be portable between hosts, where the receiving developer needs only to adjust the local.conf file for the local installation and sstate directories.

Ability to get notification of build status

Since builds can take a long time, it would be very useful to get for example email notification when a build stops, either with success or with failure, rather than needing to continually and manually poll the Toaster interface.

Discussion

[DAVE] I agree, but can we generalize this post-build processing to enable post-build script execution triggered by either success, failure, or either? We could provide a few parameterized scripts to, for example:

• send email to a user or list with build status,
• (on success) start an emulation session with regression tests on a build,
• (on success) download the built file system or SDK to a user's host or target,
• provide hooks for the user to substitute their alternative post-build triggers.

Ability to access Toaster from portable device

Since builds generally take a long time, the user may be away from their desk (or even work hours) and wish to track the status of their pending builds. Supporting a basic Toaster interface for such limited devices would be advantageous.

Snapshots of builds for long term archiving (project dir lifecycle)

A build manager may wish to keep the pertinent facts of builds for an extended period of time. This could be for a few days to allow time for the build's owner to reconcile the results, or if maybe over a year for tracking long-term trends and regressions. In each case, there should be a way to capture a minimal set of information from the build directory (e.g. the conf files and any build failure logs) and preserve it so that together with the database there is enough information to study that project even when the full build directory is removed and recycled.