Toaster

Toaster is a Web-based interface to the Bitbake build system and the Poky distribution inside the Yocto Project. This project was formely known as Web Hob / Webhob / webhob, and you may still find references to the old name in the documentation.

General discussion about Toaster happens on a dedicated mailing list: https://lists.yoctoproject.org/listinfo/toaster

Project planning is available below, and the design documents we follow.

User interface

The design of the user interface takes place in iterations, and most recent designs supersede older ones.

The latest iteration is always available at: http://www.yoctoproject.org/toaster

The document File:Toaster 1.6 content structure.pdf shows the content structure of Toaster in the 1.6 release.

Other design documentation

• Visualisations - an inventory of data visualisations we aim to include in the first release of Toaster
• File:WH roadmap.pdf - the roadmap for Toaster development produced by the London-based agency Tobias&Tobias
• File:Multiuser support in Web Hob.pdf - different design approaches to handle multi-user scenarios
• File:Web Hob content structure.pdf - Toaster information architecture, as it came out from the design project with Tobias&Tobias

Old design documents

• Web_Hob_design_project_with_T&T - The outcome of the design project with the external agency Tobias&Tobias
• A preliminary design project by Jim Kosem

Architecture and component design

We are planning the capabilities of Toaster based on evolutive stages of implementation, using community feedback on each stage to plan, design and implement a set of capabilities for the next stage. In the first stage, synchronized with the planning of Yocto Project 1.5 release, we implemented the back end of a build analysis / image inspection module. As part of the Yocto Project 1.6 release, we are building a web-based graphical user interface for the build analysis / image inspection module.

Toaster is designed as a collection of components that will run independently performing isolated functions. The interfaces between components are documented on this wiki as to ease interoperability with newer components. From design phase, we've taken care to account for further expansion needs, and account for scalability problems.

• Data store investigation results and choices

• Django models

• REST API Contracts

• Bugzilla feature list

Installation and Running

Before you start, make sure you install the packages required by the Yocto Project

Using the Yocto Project 1.6 "Daisy" release or the master branch

You need this stuff ready

• Django 1.5. The easiest way is to install system wide with pip

   $ sudo pip install django==1.5

• South 0.8.4

  $ sudo pip install South==0.8.4

• Make sure that port 8000 and 8200 are free, i.e. no servers on them

To get it up and running

• clone the master tree: http://git.yoctoproject.org/git/poky
• set up a build as normal: source poky/oe-init-build-env
• at this point edit local.conf, or layers, etc.
• start Toaster system: source toaster start
• run builds normally: bitbake core-image-minimal
• to see the web interface: xdg-open http://localhost:8000/
• to stop Toaster: source toaster stop

Using the Dora + Toaster release (part of Yocto Project 1.5)

You need this stuff ready

• Django 1.4.5; easiest way is to install system wide with pip install django=="1.4.5"
• Make sure that port 8000 and 8200 are free, i.e. no servers on them

To get it up and running

• checkout the Dora + Toaster release tree: http://git.yoctoproject.org/cgit/cgit.cgi/poky/log/?h=dora-toaster
• set up a build as normal: source poky/oe-init-build-env
• at this point edit local.conf, or layers, etc.
• start Toaster system: source toaster start
• run builds normally: bitbake core-image-minimal
• to see the web interface: xdg-open http://localhost:8000/
• to stop Toaster: source toaster stop

Bitbake extra options

This is an overview of extra options needed for Bitbake. These should be automatically added by Toaster in a conf/toaster.conf file that is enabled when the system starts through the toaster command.

The gritty details are here:

Toaster needs the toaster bbclass enabled in Bitbake in order to record target image package information. INHERIT += "toaster"

This enables the toaster bbclass in Bitbake, needed to record target image package information.

INHERIT += "buildhistory"
BUILDHISTORY_COMMIT = "1"

This enables buildhistory in Bitbake, needed to record target image package information. This last command is not strictly needed, yet without it buildhistory will not work right in it's own right.

A bunch of files are created and used under the build/ directory:

• toaster.sqlite - the database file
• toaster_web.log - the log file of the web server
• toaster_ui.log - the log file of the ui component
• .toastermain.pid - contains the pid of the web server
• .toasterui.pid - contains the pid of the DSI data bridge
• bitbake-cookerdaemon.log - the log file of the bitbake server

Deleting a Build from the Toaster Database

By default, Toaster stores all build data in a toaster.sqlite database file located in the Build Directory.

You can delete builds from the Toaster database using two bespoke commands (buildslist and builddelete) added to

/poky/bitbake/lib/toaster/manage.py

Follow these steps to delete a build:

1. Identify the build: Run the following command from your Build Directory to display a user-readable, tabbed list of build IDs and identification information:

    $ python ../bitbake/lib/toaster/manage.py buildslist

2. Delete a specific build: Using the build ID displayed in the previous step, run the following command to delete a specific build:

    $ python ../bitbake/lib/toaster/manage.py builddelete <build_id>

The build and all related information is deleted from the Toaster database. The deletion operation traverses the ForeignKey relationships recursively.

When using the buildslist and builddelete commands, you need to consider the following:

• For default configurations, your working directory must be the Build Directory. Using these command from the Build Directory allows Toaster to find the toaster.sqlite file, which is located in the Build Directory.

• For other database configurations, it is possible that you can use these commands from a directory other than the Build directory. To do so, the toastermain/settings.py file must be configured to point to the correct database backend.

It is recommended to always call manage.py from the Build Directory.

Setting up a Toaster Instance on a Remote Host

Under normal circumstances, starting Toaster causes three things happen:

• A BitBake server starts

• The Toaster UI starts, which connects to the BitBake server on one side and to the SQL database on the other side

• The web server starts, which reads the database and displays the web user interface

Situations exist, however, where you might want to have multiple instances of Toaster running on various remote machines. You can create this situation by basically modifying how Toaster starts and where the common SQL database resides. You are able to do this because it is not required that Toaster starts the above set of components in order to run. Minimally, an instance of Toaster requires just one of the components to run. Consequently, you are free to manually start as many or few of the components as you need rather than using the Toaster script to cause all three things to happen.

The concepts for setting up multiple instances of Toaster revolve around maintaining a common SQL database and Web server that show data from that common database and then setting up separate instances of BitBake servers and Toaster user interfaces for each separate BitBake build directory. Note that the common SQL database and the Web server show data from all the various BitBake builds. Setting the SQL database outside of any BitBake build directories maintains a separation layer between the various builds.

The database is persistent because the logging database is set up external to the database server (e.g. MySQL). It is not even necessary to run the BitBake servers, the SQL server, and the Web server on the same machine. Each component can be run on its own machine.

Here are the steps to get set up:

1. Set up the SQL Logging Server and the Web Server

You can use any SQL server out of the box (e.g. apt-get install mysgl-server works for an Ubuntu system). If you are concerned about performance, you might want to hand-tune the server. You must set up proper username and password access for the server. You need administration rights for the mysql root account. Realize that this is not the same thing as root access on the machine.

Clone a separate, local Git repository of the Toaster master branch to use for running the Web server. You do not perform builds on this tree. You need to create this local repository away from any build areas.

In the separately cloned tree for the Web server, edit the bitbake/lib/toaster/toastermain/settings.py file so that the DATABASES value points to the previously created database server. Use the username and password you established earlier.

Run the database sync scripts to create the needed tables as follows:

    $ python bitbake/lib/toaster/manage.py syncdb
    $ python bitbake/lib/toaster/manage.py migrate orm

Start the Web server using the following command:

    $ python bitbake/lib/toaster/manage.py runserver

2. Enable build logging to the common SQL server for each build directory you are using

Edit _build local_ bitbake/lib/toaster/toastermain/settings.py to alter the DATABASES value to point to the common SQL logging server.

Start the BitBake server using the following command:

    $ bitbake --postread conf/toaster.conf --server-only -t xmlrpc -B localhost:0 && export BBSERVER=localhost:-1

Start the logging user interface using the following command:

    $ nohup bitbake --observer-only -u toasterui &

NOTE: No hard-coded ports are used as there is enough code to run autodiscovery for ports to prevent collisions.

At this point, you are ready to run your builds using commands such as:

    $ bitbake core-image-minimal

When you are finished, you need to kill the BitBake server for that particular build area:

    $ bitbake -m

3. Verify that it all works

You should examine the logs and be sure that the logging worked, that data is persistent, and that data from multiple builds from different areas was supported.

Testing

Documentation and results related to Toaster Quality Assurance.

• Toaster testing plan

Contributing to Toaster

Toaster is a community effort and welcomes your contribution.

• Contribute to Toaster

Old page content

This page is about the Web Hob project. Web Hob is a web-based interface to the Yocto Project.

General discussion about Web Hob happens on a dedicated mailing list: https://lists.yoctoproject.org/listinfo/webhob

There have been 2 main pieces of work related to Web Hob so far:

• A design project with the London-based agency Tobias & Tobias
• A preliminary design project by Jim Kosem

Web Hob information architecture

This document represents the content structure of the Web Hob application.

File:Web Hob content structure.pdf

Different approaches to multi-user workflows

This document outlines the different approaches we have uncovered so far to facilitate multi-user and team work with Web Hob.

• File:Multiuser support in Web Hob.pdf

Visualisations index

The build analysis functionality in Web Hob will include several graphical presentations of build data. The first step to design them is listing them all.