Toaster 1.7 design

The design of the Toaster user interface takes place in iterations, and most recent designs supersede older ones.

The latest iteration is always available at: http://www.yoctoproject.org/toaster

Detailed design documentation is attached to Bugzilla features. Features with design documentation attached have "GUI design available" written in the Whiteboard field, so it is easy to search for them.

Bugzilla feature list

We have finished planning the Toaster scope for the Yocto Project 1.7 release. The outcome is a set of Bugzilla features

The rest of this page will give you an idea of the scope definition process and discussions.

A couple of videos to kick off things

The following videos illustrate some of the features we are thinking of adding to Toaster during the 1.7 release cycle. When watching them, keep in mind that this is all very rough: more like sketches to help us start a conversation about scope than proper designs. Chances are the final designs will be completely different.

About projects - media:Building-1.7.mov
Starting a build - media:How-to-build.mov

Feature list

Ability to create new projects

This is the ability to use the Toaster interface to create new project directories, manage the content, and initiate the builds of selectable target images.

Summary of discussion

• A project is a set of layers and configuration variables values that describes how to do a build.
• Creating a project means writing up values for the above-mentioned set, and store these values in a persistent manner.
• A build for a project means effecting (instantiating) the build process for the current set of values.
• Build directory location (host/machine) is not relevant to Toaster.
• Saving the build artifacts and providing access to the build artifacts is critical for Toaster.

Discussion

[DAVE] What is a typical workflow that gets to the build project part? I see the configure screens but not the actually build launch.

[DAVE] Does clicking 'new build' fire off a build? If so, which configuration is built?

[BELEN] Heh, I completely forgot to show that in the first video. Here is part 2 showing how to start a build and answering (I hope) both questions.

[DAVE] Can projects be configured, primed for builds, but not built?

[BELEN] Yes.

[DAVE] Will the user be able to configure several builds, then queue them all to build?

[BELEN] Users might create more than one project then start a build for each of them by clicking the "build" button in each project page, but nothing more sophisticated than that for the time being. Here is where things can get tricky. Where does Toaster stop and something like Buildbot or Jenkins start? My take is that we want to work alongside and enhance build automation systems, not replace them.

[DAVE] What is the size of the build variable list? It's hard to figure from the conf/local.conf file. If it is a small list, like 10, should we provide better and variable specific UI widgets rather than just edit boxes, depending on the variable's "type" (eg browse button with any path variable like SSTATE_DIR)?

[BELEN] This is one of the questions we should start working on straight away. In informal chats we have spoken about a split between "build variables" (the ones that affect the build and therefore you can edit via Toaster) and "infrastructure variables" (to call them something, that affect the build infrastructure and you cannot edit via Toaster. Things like PARALLEL_MAKE for example). We should start getting into details about that split as soon as possible, and determine which variables we are going to expose via Toaster to be edited. Once again, Paul's feedback here would be good (RP's probably as well).

Regarding variable "types", I had a brief conversation with RP recently about this issue. It is unlikely that the meta data will provide us with information about the type of input expected by a variable. If and how the Toaster interface should fill that gap is up for discussion. From my perspective, we should do as much as we can to help users assign valid values to their variables.

[DAVE] I don't recall a user entry on your videos that specify the project directory for the build. This must be part of the initial release. I don't recommend making it an edited data file option, eg not via in settings.py.

[BELEN] When you run Toaster locally you can configure the location of your build directory as you wish, but when Toaster is running remotely, do we want to let people select the location of their build directory?

[DAVE] Can we make any selection on any page 'stick' with cookies or 'unstick' with a 'reset to defaults' button, in addition to propagating laborious configuration setups inputs via 'clone configuration'?

[BELEN] We will need to overcome my personal hate of 'reset to defaults' options :) They are very dangerous if we don't provide an 'undo' option as well: they might wipe your configuration and revert it to the rather useless default one, losing a bunch of work you did beforehand. Any configuration changes you make are saved: in that sense, they are 'sticky'. If I create project A, then I add 2 layers to the project configuration, select a machine and 3 targets, all those options are saved to the project configuration. You don't need to add the 2 layers, select the machine and the 3 targets every time you want to run a build.

[RAVI] I am wondering about the purpose of a project. In the design, project has a machine and owner attributes. If we define a project as a collection of builds for a machine, then we should not allow to change the machine after a project is created. Then, if a project is just a collection of builds for a particular machine, then we can modify the existing all builds view to group builds by machine. Based on this argument, I do not see a need for project and build hierarchy.

[DAVID] My group understands that a "project" is a specific build directory. A project will most likely to have multiple builds, and things like the (a) machine, (b) layers, (c) conf variables, and so forth would be presumably set before the first target is selected and executed. It is also the case that the developer may decide to change any of those settings, including even the machine type, and then carry on with more builds. The use case would be for example is arch coverage testing of a particular set of kernel or user space packages.

A host can then have multiple project directories (I most often work that way), for example when a developer truly wants to keep their development and/or debugging from not polluting other projects.

Also, separate users should do their work in separate project directories, now that we are extending the model to cover that.

There can then be multiple hosts, with all of their multiple projects, all channeling their Toaster data into a shared database. And with that you have a build farm.

[RAVI] I see that the uniqueness of a project is the <hostname>://<build directory path>. The contents of the build directory can be changed and multiple builds can be kicked off from the same project with different configurations. If this the case, the project creation should also setup a build directory on the host.

Looking at this use case, we should have toaster running on the host machine with a shared database. Or if we run a single toaster instance serving multiple projects (hosts and build directories), we should have a mechanism for the toaster machine to talk to the project machines. Am I missing something?

[PAUL] Yes, but the project need not be tied to a specific host or location on disk. To my mind, the project is simply a configuration set - if you had a build farm the actual build could be run on any server within that farm and that side of it would be largely hidden from the user (until it needed to be visible, e.g. when you need to diagnose a host contamination issue.) This then means that the management of build directories can be independent, i.e. old build directories can be automatically deleted after a time period or when space runs low, but the project remains in the database for later use when needed.

[DAVID] I am completely willing to accept the above definition of a "Toaster Project". The part that convinces me is that (a) while under active build and development it does map to a physical location, but (b) it can hold the data and presumably the "snapshot" artifacts of a physical build directory long after that actual directory has been deleted. That last part is a highly desired feature from the Wind River team, and was also brought up as a specific question at my Toaster presentation at the Yocto Project Summit.

This implies that given the abstract nature of a Toaster "Project", it makes sense that not only can the developer share that "project" with another developer and Toaster instance (as per Belén design), but it makes sense that the original developer can make their own clones of that project when they for example want use as a reference basis for new projects with additional adjustments and/or for binding and running on a different host/directory.

Here is what I would propose is the workflow.

1) When a Toaster Project is created,

• The "user" is the preset to the last "user", else if this is the first time it is defaulted to $USER. This is a required field.

• The "email" is preset to the last email that matches "user", else it is empty. QUESTION: Is this a required field?

• The host:dir values are preset just as "poky/oe-init-build-env" would preset these values, in other words to the current machine and to "poky/../build" (unless redirected with the appropriate environment variables). If that directory (a) already exists and has a "conf/local.conf" file and (b) that directory exists for an existing Project, then that is an error. This would allow us to re-connect to a deleted or unregistered YP build directories.

• We either have an explicit "Project Create" button, or we allow for the first build target command to insure that for new projects the project directory is created and the content preset by "poky/oe-init-build-env" before continuing.

2) For external hosts mounted via NFS:

• If an external host is mounted via a YP compatible NFS system, then that would be covered by the normal workflow above, using the local host and the respective NFS path. QUESTION: do we wish to identify the actual remove HOST ID in these circumstances, or just let the NFS path speak for itself?

3) For fully external hosts

This is the situation where a host remote to the Toaster, but is intended to be managed by this Toaster and use this Toaster's database.

• This implies using some sort of rpc, probably via SSH, to execute the creation and build targets.

• There would presumably be an installation of YP on that remote host, and that location would need to be known to Toaster. The installation could be as simple as an NFS mount that points to the local YP installation, which would provide a single source at the cost of NFS traffic.

Hide builds from a project

This option simply removes or hides selected builds from the Toaster display, for example builds quickly halted because the owner realized that they missed a setting. The database records and the project on the disk is untouched. This un-clutters the display, yet requires little programing (and development time).

Discussion

[BELEN] I have no strong feelings against this one, but we need to bear in mind that:

• It will probably be hard to allow hiding in bulk, since the reasons why someone might want to hide a build could be wildly different. So this will have to be done on a build by build basis.

• This also involves providing a way to reverse the hiding action, i.e. see hidden builds and show them again.

[DAVID] In the Wind River version of Wiki, we have a "Manage" button at the bottom. This brings up a page of all of the entries, with the ability to change the entry's hidden and description attributes. This does allow a bulk update easily.

I propose that we do the same, where a "Manage" button brings up all of the builds and/or projects, each highlighted according to if it is currently hidden, also each with a text entry for the description (see the next design proposal) and a check box to toggle the hidden state.

We then need to decide if this meta date is stored in the database or in the page's cookie.

Ability to add a description to a build or project

User's may not recall the difference and purpose of a given build or project, so it would be good for the Toaster interface to allow the user to add this and update this as needed.

[DAVID] See the discussion above about "hidden" builds for a proposed implementation.

Delete builds from a project

This option is where a selected build and all its related database records are removed from the database. This removes the database overhead for this unwanted builds. The project on the disk is untouched. Since the database should be normalized, this operation should be clean and not interfere with other builds. I believe we have a command line version of this operation with Toaster-1.6.

Discussion

[Alex] Technically, this is straightforward. In terms of policy access, we need to define one; I propose that only the project owner is allowed to delete builds. Impersonation is trivial right now, but once we implement authentication, bricks will fall in the right place in terms of access control.

Ability to add / remove layers

Add and delete from the project layer directories. These layers may be within repositories or in plain directories on the disk.

Discussion

[FARRELL] How would I upload my specialized device driver/library/system call handler/etc that necessarily needs to be part of the build? Would this be done by importing a layer?

[BELEN] Yes. For the 1.7 release, all customisation must be done via layers.

[DAVE] Are you constraining 'import layer' to a git repo path? Why not just a local directory that is not a git repo? In that case should we have a browse button as well as an edit box?

[BELEN] This is a really good question. I need the technical crowd to establish what's possible / what we want to do. We can then design accordingly.

[DAVID] It is the case that we recommend all of our customers to add their content via layers, and specifically layer directories since they are most likely to be under continual and direct change and would not be placed into a formal repo, or more likely they would be managed by their SCM system as is and certainly not packed into a base git clone.

Ability to identify / set project owner / user

A build farm manager (and their shared Toaster instance) may accept build sets from clients, in which case it is important to know who initiates and owns each specific build, for completion notifications and project space recycling.

Discussion

[FARRELL] You presented the notion of a user - in the movie it was 'Belen'. I think it may be necessary to formalize this idea with a login screen and at least rudimentary permission and protection. Here is a use case: Imagine a vendor deploys Toaster on a server which serves all of its customers. The customers are very diverse and are often competitors of one another and do not want their configuration/information/ideas shared with other vendor customers.

[BELEN] Very true: access control is a big deal and very important. However, we are not sure we can tackle this in the 1.7 release. That's why in the video I speak of "owners" instead of "users". Our "owner" concept derives from the need to identify who created a certain project, but they are not really users since, for example, they don't involve authentication (there is no password).

Here is a possible description of these "owners". Each client accessing a Toaster instance might have an owner linked to it. You don't explicitly create these owners: you just set them in the process of creating your first project. Then, we remember them. Owners involve only 3 pieces of information: a name, an optional email address and the client they are associated with. You can change the details at any time. So "owners" don't involve access control: they just tell me who created (and therefore owns) a certain project. You can have the same owner details in several clients: I might have a laptop and a builds machine, I access the same Toaster instance from both, and I can use the same owner name and email address in both.

The above opens some questions:

• can others edit the configuration of a project I own?
• can others start builds against a project I own?
• can others download artifacts of a project I own?
• can others even see the projects I own?

The simplest thing would be either:

• 'yes' to all of the above, but that means that people creating Toaster instances that can be accessed by more than one person have to be mindful of the fact that everything in them is open.

• 'no' to all of the above, and channel all collaboration through the export / import projects functionality, which is safer but also stifles co-operation.

Just to clarify, ownership as above is established per project, not per build.

[FARRELL] As much as I hate to suggest this...It may be necessary to introduce the notion of group access - Owner/Group/Others ala unix/linux. I think it may be necessary for users to be able to share their results while excluding others. I can imagine a number of use cases. Here is one:

       Two customers/users, Fu and Bar, work for the same company C. Each wants to
       be able to have personal 'experimental' builds but may need to share a 'production'
       build - react to errors and download results, etc. Of course, company C doesn't want
       company B to see their results, configurations, custom drivers, etc. But, Fu needs to
       see and have access to some of Bar's results. These problems can be solved with the
       introduction of the group access.

I realize this adds a layer of complexity which may be necessary. An alternative would be access control lists (acl). These can be even more flexible but a lot more complicated to implement, manage and maintain.

I think we should decide on an access control model soon and work out an implementation. Even with the current implementation, retrofitting will be difficult and somewhat tedious which will become even more so as we add more features.

[DAVID] The intermediate near term solution is to consider the 1.7 as working with only trusted users, in that they are all already within the same intranet. Under this limitation, if user Foo needs their work fully separate from user Bar, they could just have separate instances and databases of Toaster running, and use existing Linux security to accomplish this.

That said, we could provide for the default build/project view to default to the current "owner", where they would have to go out of their way to other people's work.

We could also consider an "audit log" that captures project creation (and especially project deletion), so that at least you know if someone stomped on your stuff :-)

[BELEN] I am with David on this one: I think we can find temporary alternatives. It's not that I disagree with Farrell, mind: he is completely right that we will need this. Problem is: we need to be mindful of what we can deliver with the resources we have.

Ability to identify build host and build directory

To support distributed build hosts sharing a common Toaster database, there needs to be a way in the database to identify which host (IP address and or host name) and local directory the project was built in.

Discussion

[ALEX] I am not sure I follow the use case / user story here. Can we get more details on why this is important ?

Ability to download files (YP #6096)

While developers would presumably have access to all build files within the intranet for the build logs and generated target image files, there are cases where that is not so. For example, the owner may be using a Windows host or a portable device to access Toaster, and may wish to download and view the error logs to determine their next course of action. Also, it may be the case that the specific build host is accessible only to the build farm manager and not the individual project owner, and having Toaster serve the file(s) would resolve this barrier.

Discussion

[FARRELL] The ability to download the result of the build should be somewhere - maybe a column on the All Builds page?

[DAVID] As for things like file downloads, between Ravi and myself I think we will find time to get that one done since we both really want it!

[BELEN] Absolutely: in fact, this is something I might start designing straight away so that it can be implemented in 1.7M1. However, in order to do that, we need to reach agreement as to what a build artifact is:

• Is it just the rootfs files? Or do they include
• the cooker and task logs?
• shared state objects used?
• recipe files / patches?
• [yours here]

Share projects via configuration (export / import)

This is the capture and sharing of the minimal project configuration data, such that a remote developer can replicate the same project against their own Yocto Project installation.

Discussion

[DAVE] The video ended at the export project discussion. Is export just sending email, the conf file (or files) to someone, or is there be more to it?

[BELEN] I am not sure what the answer is: we need input from Paul and Alex here. In my head, the project configuration (not the builds) is exported to some kind of file(s) that I can then share with others in whichever way I see fit (email, file sharing, etc).

[ALEX] I'm thinking that exporting the project will produce a file that can be imported on another Toaster instance and replicate the original settings of the project that was exported. The concrete means of achieving this may be as simple as exporting a JSON snapshot of the relevant database entries.

Definition of a Project

The proposed design adds the new concept of the "Project".

Discussion

[PAUL] To my mind, the project is simply a configuration set - if you had a build farm the actual build could be run on any server within that farm and that side of it would be largely hidden from the user (until it needed to be visible, e.g. when you need to diagnose a host contamination issue.) This then means that the management of build directories can be independent, i.e. old build directories can be automatically deleted after a time period or when space runs low, but the project remains in the database for later use when needed.

[DAVID] I am completely willing to accept your definition of a "Toaster Project". The part that convinces me is that (a) while under active build and development it does map to a physical location, but (b) it can hold the data and presumably the "snapshot" artifacts of a physical build directory long after that actual directory has been deleted. That last part is a highly desired feature from the Wind River team, and was also brought up as a specific question at my Toaster presentation at the Yocto Project Summit.

This implies that given the abstract nature of a Toaster "Project", it makes sense that not only can the developer share that "project" with another developer and Toaster instance (as per Belén design), but it makes sense that the original developer can make their own clones of that project when they for example want use as a reference basis for new projects with additional adjustments and/or for binding and running on a different host/directory.

Workflow around creating a Project

[DAVID] Here is what I would propose as a workflow.

1) When a Toaster Project is created,

• The "user" is the preset to the last "user", else if this is the first time it is defaulted to $USER. This is a required field.

• The "email" is preset to the last email that matches "user", else it is empty. QUESTION: Is this a required field?

• The host:dir values are preset just as "poky/oe-init-build-env" would preset these values, in other words to the current machine and to "poky/../build" (unless redirected with the appropriate environment variables). If that directory (a) already exists and has a "conf/local.conf" file and (b) that directory is already registered with an existing Project, then that is an error. This would provide some sanity check, but would also allow us to re-connect to an unregistered YP build directory (and/or ones previous registered with a subsequently deleted project).

• We either have an explicit "Project Create" button, or we allow for the first build target command to insure that for new projects the project directory is created and the content preset by "poky/oe-init-build-env" before continuing.

2) For external hosts mounted via NFS:

• If an external host is mounted via a YP compatible NFS system, then that would be covered by the normal workflow above, using the local host and the respective NFS path. QUESTION: do we wish to identify the actual remove HOST ID in these circumstances, or just let the NFS path speak for itsel

3) For fully external hosts

This is the situation where a host remote to the Toaster, but is intended to be managed by this Toaster and use this Toaster's database.

• This implies using some sort of rpc, probably via SSH, to execute the creation and build targets.

• There would presumably be an installation of YP on that remote host, and that location would need to be known to Toaster. The installation could be as simple as an NFS mount that points to the local YP installation, which would provide a single source at the cost of NFS traffic.

Toaster System Architectures

This is an iteration of the possible supported system architectures of a Toaster implementation. We should validate how the proposed design would work with each of these options.

Minimal Local Project Implementation

Configuration: Local host, local project, database in project, toaster and web server started from project

This is the default implementation, using the "source toaster [start|stop]" support.

Multiple Local Project Implementation

Configuration: Local host, Multiple local projects, shared local database, toaster and web server started separately

This is currently supported by (a) edits to the local.conf file to point to the shared database location, and (b) using special commands to start the shared Toaster and webserver instance.

Multiple Distributed Host Implementation

Configuration: Multiple hosts, Multiple local projects per host, shared network database, toaster and web server started with network database instance

Each host would either have its own installation of YP, or have a centralized copy shared over NFS. This implementation is currently not yet documented nor supported.

Open Bugzilla features / defects

This is what's open in Bugzilla right now. Apart from what's already assigned to 1.6.1, there are 2 issues that I am keen on getting fixed:

5187 - Same cooker log file name used for several builds
6095 - Retrieve full build information independently of task execution

Feel free to add to this list.

Future features

A list of features that have come up in discussion, but not for the 1.7 release.

Delete projects

This option deletes the project all the way down to the project's instance on the disk. This should be a safe operation unless a separate project is directly sharing its sstate cache. That would be an recommended usage, where a proper usage would a standalone shared sstate cache. There may be artifacts in that shared sstate cache that are specific and unique to the deleted project, but it is out of scope for this option to try to delete those as well.

Ability to add/remove recipes

This is the ability to add or remove recipes from the project's configuration.

Ability to import source (tarballs, repos)

This is the ability to bring external sources into the project and include them with wrapper recipes.

Ability to add/remove packages from image

This is the ability to add or remove specific binary packages from the respective image(s) of a project.

Share projects via Copy (all but "./tmp")

Project directories (all except for their local "tmp" directory) should be portable between hosts, where the receiving developer needs only to adjust the local.conf file for the local installation and sstate directories.

Ability to get notification of build status

Since builds can take a long time, it would be very useful to get for example email notification when a build stops, either with success or with failure, rather than needing to continually and manually poll the Toaster interface.

Discussion

[DAVE] I agree, but can we generalize this post-build processing to enable post-build script execution triggered by either success, failure, or either? We could provide a few parameterized scripts to, for example:

• send email to a user or list with build status,
• (on success) start an emulation session with regression tests on a build,
• (on success) download the built file system or SDK to a user's host or target,
• provide hooks for the user to substitute their alternative post-build triggers.

Ability to access Toaster from portable device

Since builds generally take a long time, the user may be away from their desk (or even work hours) and wish to track the status of their pending builds. Supporting a basic Toaster interface for such limited devices would be advantageous.

Ability to compare projects/builds

A user may not recall the difference between one build and another, or one project and another. The data is in the database, so it would not be hard to generate a report itemizing the observed differences. Here are some the potential changes that can be extracted:

• Changed variable definitions (less base path differences). This would also cover changes in things like the machine, layers, extra added or excluded recipes, and multi-lib support.
• Changes in the included packages list
• Perhaps changes in the build/caching statistics

Snapshots of builds for long term archiving (project dir lifecycle)

A build manager may wish to keep the pertinent facts of builds for an extended period of time. This could be for a few days to allow time for the build's owner to reconcile the results, or if maybe over a year for tracking long-term trends and regressions. In each case, there should be a way to capture a minimal set of information from the build directory (e.g. the conf files and any build failure logs) and preserve it so that together with the database there is enough information to study that project even when the full build directory is removed and recycled.