Toaster 1.7 design

A couple of videos to kick off things

media:Building-1.7.mov

Feature list

Ability to create new projects

This is the ability to use the Toaster interface to create new project directories, manage the content, and initiate the builds of selectable target images.

Discussion

[DAVE] What is a typical workflow that gets to the build project part? I see the configure screens but not the actualy build launch.

[DAVE] Does clicking 'new build' fire off a build? If so, which configuration is built?

[BELEN] Heh, I completely forgot to show that in the first video. Here is a part 2 showing how to start a build and answering (I hope) both questions.

[DAVE] Can projects be configured, primed for builds, but not built?

[BELEN] Yes.

[DAVE] Will the user be able to configure several builds, then queue them all to build?

[BELEN] Users might create more than one project then start a build for each of them by clicking the "build" button in each project page, but nothing more sophisticated than that for the time being. Here is where things can get tricky. Where does Toaster stop and something like Buildbot or Jenkins start? My take is that we want to work alongside and enhance build automation systems, not replace them.

[DAVE] What is the size of the build variable list? It's hard to figure from the conf/local.conf file. If it is a small list, like 10, should we provide better and variable specific UI widgets rather than just edit boxes, depending on the variable's "type" (eg browse button with any path variable like SSTATE_DIR)?

[BELEN] This is one of the questions we should start working on straight away. In informal chats we have spoken about a split between "build variables" (the ones that affect the build and therefore you can edit via Toaster) and "infrastructure variables" (to call them something, that affect the build infrastructure and you cannot edit via Toaster. Things like PARALLEL_MAKE for example). We should start getting into details about that split as soon as possible, and determine which variables we are going to expose via Toaster to be edited. Once again, Paul's feedback here will be needed (RP's probably as well).

Hide builds from a project

This option simply removes or hides selected builds from the Toaster display, for example builds quickly halted because the owner realized that they missed a setting. The database records and the project on the disk is untouched. This un-clutters the display, yet requires little programing (and development time).

Discussion

[BELEN] I have no strong feelings against this one, but we need to bear in mind that:

• It will probably be hard to allow hiding in bulk, since the reasons why someone might want to hide a build could be wildly different. So this will have to be done on a build by build basis.

• This also involves providing a way to reverse the hiding action, i.e. see hidden builds and show them again.

Delete builds from a project

This option is where a selected build and all its related database records are removed from the database. This removes the database overhead for this unwanted builds. The project on the disk is untouched. Since the database should be normalized, this operation should be clean and not interfere with other builds. I believe we have a command line version of this operation with Toaster-1.6.

Ability to add / remove layers

Add and delete from the project layer directories. These layers may be within repositories or in plain directories on the disk.

Discussion

[FARRELL] How would I upload my specialized device driver/library/system call handler/etc that necessarily needs to be part of the build? Would this be done by importing a layer?

[BELEN] Yes. For the 1.7 release, all customisation must be done via layers.

[DAVE] Are you constraining 'import layer' to a git repo path? Why not just a local directory that is not a git repo? In that case should we have a browse button as well as an edit box?

[BELEN] This is a really good question. I need the technical crowd to establish what's possible / what we want to do. We can then design accordingly.

Ability to identify / set project owner / user

A build farm manager (and their shared Toaster instance) may accept build sets from clients, in which case it is important to know who initiates and owns each specific build, for completion notifications and project space recycling.

Discussion

[FARRELL] You presented the notion of a user - in the movie it was 'Belen'. I think it may be necessary to formalize this idea with a login screen and at least rudimentary permission and protection. Here is a use case: Imagine a vendor deploys Toaster on a server which serves all of its customers. The customers are very diverse and are often competitors of one another and do not want their configuration/information/ideas shared with other vendor customers.

[BELEN] Very true: access control is a big deal and very important. However, we don't think we can tackle this in the 1.7 release. That's why in the video I speak of "owners" instead of "users". Our "owner" concept derives from the need to identify who created a certain project, but they are not really users since, for example, they don't involve authentication (there is no password).

Here is a possible description of these "owners". Each client accessing a Toaster instance might have an owner linked to it. You don't explicitly create these owners: you just set them in the process of creating your first project. Then, we remember them. Owners involve only 3 pieces of information: a name, an optional email address and the client they are associated with. You can change the details at any time. So "owners" don't involve access control: they just tell me who created (and therefore owns) a certain project. You can have the same owner details in several clients: I might have a laptop and a builds machine, I access Toaster from both, and I can use the same owner name and email address in both.

The above opens some questions:

• can others edit the configuration of a project I own?
• can others start builds against a project I own?
• can others download artifacts of a project I own?
• can others even see the projects I own?

The simplest thing would be either:

• 'yes' to all of the above, but that means that people creating Toaster instances that can be accessed by more than one person have to be mindful of the fact that everything in them is open.

• 'no' to all of the above, and channel all collaboration through the export / import projects functionality, which is safer but also stifles co-operation.

Ability to identify build host and build directory

To support distributed build hosts sharing a common Toaster database, there needs to be a way in the database to identify which host (IP address and or host name) and local directory the project was built in.

Ability to download files

While developers would presumably have access to all build files within the intranet for the build logs and generated target image files, there are cases where that is not so. For example, the owner may be using a Windows host or a portable device to access Toaster, and may wish to download and view the error logs to determine their next course of action. Also, it may be the case that the specific build host is accessible only to the build farm manager and not the individual project owner, and having Toaster serve the file(s) would resolve this barrier.

Discussion

[FARRELL] The ability to download the result of the build should be somewhere - maybe a column on the All Builds page?

[DAVID] As for things like file downloads, between Ravi and myself I think we will find time to get that one done since we both really want it!

[BELEN] Absolutely: in fact, this is something I might start designing straight away so that it can be implemented in 1.7M1. However, in order to do that, we need to reach agreement as to what a build artifact is:

• Is it just the rootfs files? Or do they include
• the cooker and task logs?
• shared state objects used?
• recipe files / patches?
• [yours here]

Share projects via configuration (export / import)

This is the capture and sharing of the minimal project configuration data, such that a remote developer can replicate the same project against their own Yocto Project installation.

Discussion

[DAVE] The video ended at the export project discussion. Is export just sending email, the conf file (or files) to someone, or is there be more to it?

[BELEN] I am not sure what the answer is: we need input from Paul and Alex here. In my head, the project configuration (not the builds) is exported to some kind of file(s) that I can then share with others in whichever way I see fit (email, file sharing, etc).