Toaster database

ORM diagram

Toaster Uses Django and it's ORM for database interaction.

The database table/model definitions can be found in the orm/models.py file. We also have additional models defined for the build controller, these are generally the 'behind the scenes' way in which we translate toaster information into the build controller process. This page is a high level look at the database design and how it is used.

Categories of data in Toaster

Build data

Build data is Layers, Layer_Versions, Recipes, Machines, Packages, Dependencies and all other objects that are output by running a build of a recipe. Once it is created it is never edited or updated. Unless deleted by the user.

Configuration data

Configuration data is data which is provided by Toaster to configure an openembedded based project in Toaster, like Build data this is stored as Layers, Layer_Versions, Recipes and Machines etc. The configuration data can be produced by fetching it from the Layer Index (the toaster start up script attempts to do this for you at first run) from importing information from the toasterconf.json file, importing Layer definitions into Projects via the Import layer page and also from running a build.

Project data

This is the container for Configuration data and Build data. Additionally it contains project specific data such as user imported Layers and CustomImageRecipe.

Top level models

There are many intermediate tables in Toaster's database, see diagram. Below are the most top level models on which everything else depends. These models map to the metadata concepts that make up openembedded/Bitbake.

Layer and Layer_Version

A Layer_Version is the model in which most other models relate to in Toaster. Every Layer_Version object belongs to a Layer object which contains the common metadata for each 'version' or revision of the Layer.

An example would be that we have a Layer called 'meta-openembedded' which has 3 Layer_Versions; one Layer_Version which is set to the release 'dizzy' one on 'fido' and one on 'master' (in most cases these correspond to git branch names)

Layer 1 - N Layer_Version 1 - N Recipe 1 - N Packages

Recipe

Recipes contain the information about a collection or particular piece of software. This can be the name, version and description and any distribution packages are created by the Recipe.

An example would be a recipe called myrecipe version 1.2 which produces 3 Packages myrecipe-dev, myrecipe-dbg and myrecipe. Recipes are the objects which get built by BitBake.

Recipe N - 1 Layer_Version

Recipe 1 - N Package

CustomImageRecipe

This is a special kind of Recipe which has three additional fields; base_recipe, last_updated and project. base_recipe is the Recipe which the Custom recipe has been based on and is the recipe which will be copied in when the Custom recipe is generated. The last_updated field is used internally to keep track of the last time the package list was last updated from a build.

Package

A Package is the representation of distribution packages produced by a Recipe when it has been built. Package data is almost always Build data since we don't know about packages until they have been produced. The exception is when the package data is used as configuration data to Customise a recipe by adding and removing known packages.

Package N - 1 Recipe

CustomImagePackage

This is a subclass of Package which contains 3 extra fields. recipe_includes are for keeping track of the packages which are included as part of the base recipe when it is built. e.g. the base_recipe has package busybox. recipe_excludes are packages which are normally included as defined by the base_recipe which have been explicitly asked to be excluded by the user. recipe_appends are the packages which are being appended to this recipe (think IMAGE_INSTALL +=)

These objects are configuration data and are generated by running builds. When a build runs we create or update CustomImagePackages meaning that each build helps to create a pool of packages which are available to the user to add or remove based on the available recipes when creating a CustomImageRecipe

CustomImagePackage N - N CustomImageRecipe

Build

The Build model contains the metadata for reporting information about a Build. It is the top level object for many of the Build related models which contain output from a Build. The Build model also includes when the Build started and stopped, and the status of the build. Any data in toaster that has a build object related to it is considered Build data.

Build 1 - 1 Target 1 - N Target_File

Build 1 - N Task

Build 1 - N BuildArtifact

Build 1 - N LogMessage 1 - 1 Task

Build 1 - N Variable

Build 1 - N Layer_Version

Target

A target is the instruction on what to build that is sent to Bitbake. When a build is triggered a Target object is created which records the one or more Recipe names to be built and the Task that you're asking BitBake to do (no task supplied executes the default which is build the recipe).

Target 1 - 1 Build

Project

The project model is the container for the configuration data of your current project, this includes the release, the Layer_Versions that you're using in your project and any project wide variables that are set when you trigger a build. Builds that you trigger on your project are also associated with the project.

Project 1 - N Build N - N Layer_Version 1 - N Recipe 1 - N Package

Project 1 - N ProjectLayers 1 - 1 Layer_Version

Project 1 - N Layer_Versions (ImportedLayer)

Project 1 - N CustomImageRecipe

What running a Build affects

When you've configured your project in Toaster (using configuration data) and hit build on a Recipe two main things happen.

1. Update or create the Recipe and Layer_Version in Toaster with any new information that we gathered by building it

When a recipe is built we can update the configuration information in Toaster to be more accurate. e.g. if we have newer information than provided by the Layer index or if we're building a recipe in an imported layer for the first time we can update that Layer_Version to say what Recipes are available in the layer or other metadata items which we have discovered by building (Summary, Description, Version, Licence, etc) we also create and update the packages which are provided by the recipe. These are created as CustomImagePackage objects as their use is for customising the packages installed in an image recipe (CustomImageRecipe)

2. Make a copy of the Recipe(s) built and the Layer_Version(s) in the project at build time

Copying creates a snapshot of that configuration data for the purposes of the build history. It also allows us to compartmentalise the build specific data e.g. what version the recipe was on, what the git checkout was at the time of build, the Machine set which may affect the recipes and packages etc

The build property for these copied Layer_Versions is set to the current Build object. When this is set the objects down the relational chain (Layer_Version 1 - N Recipe 1 - N Packages) are considered part of the "build data" and should never be changed or edited after this point.

What running lsupdates affects

When lsupdates (layer source updates) is run all the LayerSources run their update function. Layer sources provide the configuration data for Toaster from pre-generated configuration data. This allows us to to bootstrap Toaster to a state where you can select Recipes, Machines and Layers for your project without having had the cost of downloading all of the Layers in openembedded and then parsing or building them. Toaster by default uses the Layer index as a LayerSource to download this pre-generated configuration data.

Layer index layer source

The Layer index source has a ResT API which is called by the LayerIndex LayerSource which creates or updates (if they exist) Layers, Layer_Versions, Recipes and Machines.