'''*******This is a WIP*******'''

== Introduction to ToasterTable ==

ToasterTable is a set of classes in the toastergui module of the Toaster Django application. It is used to present a set of database records in a style and with functionality consistent with the rest of Toaster.

The functionality provided by ToasterTable includes:
* Sorting
* Column hiding/showing
* Filtering
* Paging
* Search

When a ToasterTable is updated, a new set of records matching the filter criteria, search string, sort order and page number is fetched from the back-end using Ajax requests. The ToasterTable is then updated in place from the JSON in the response, using JavaScript to redraw the table rows.

As a developer, you don't need to worry (necessarily) about how this works: you just implement a subclass of ToasterTable, specify a template for rendering it, and supply a URL mapping.

The following sections explain how to use ToasterTable. The filenames in these sections refer to files in the toastergui/ directory.

Note that to be able to follow the example, you will need to set up Toaster for development work. This is described in [http://www.yoctoproject.org/docs/2.0.1/toaster-manual/toaster-manual.html the Toaster manual].

== How to create a ToasterTable ==

Most of the tables needed for Toaster are already implemented. Many of the key tables in Toaster already use ToasterTable, but some don't. A full list of unconverted tables is in https://bugzilla.yoctoproject.org/show_bug.cgi?id=8363.

This means that you will typically be porting an existing table to ToasterTable, rather than adding a new table from scratch. However, to keep things simple, this section explains how to create a ToasterTable from scratch. Knowing how to do this should make it easier to port an existing table to ToasterTable in future.

To provide a worked example, I'll create a new table which is a variant of the existing "all builds" table, using a reduced number of columns and filters.

=== Create the table class ===

A quick overview of where things are and where they go when creating a table:

* widgets.py contains the ToasterTable class. This is the base class for all ToasterTables.
* New tables are added to tables.py.
* A table in tables.py maps to a URL within the Toaster application; the mapping is defined in urls.py.
* Templates for the table are added to templates/.
* If you need [https://docs.djangoproject.com/en/1.9/howto/custom-template-tags/ custom template tags], these go in templatetags/.

To create a new table, first add the class definition to tables.py:

<pre>
# import any models you need for the table
from orm.models import Build

class MiniBuildsTable(ToasterTable):
def __init__(self, *args, **kwargs):
super(MiniBuildsTable, self).__init__(*args, **kwargs)
self.default_orderby = '-completed_on'
self.title = 'Mini Builds Table'

def setup_queryset(self, *args, **kwargs):
self.queryset = Build.objects.all()

def setup_columns(self, *args, **kwargs):
self.add_column(title='Completed on',
help_text='The date and time when the build finished',
hideable=False,
orderable=True,
static_data_name='completed_on',
static_data_template='{{data.completed_on | date:"d/m/y H:i"}}')
</pre>

The key to creating a ToasterTable is to decide which data is going to be shown in the table and define this in the ''setup_queryset()'' method. This will typically be a queryset derived from one model in the Toaster application. To see the available models, check the orm/models.py file.

Each row in the table corresponds to a record in the queryset. The ''setup_queryset()'' method specifies how to get this queryset for the table: it must set the self.queryset property to the Django queryset containing the data. In the MiniBuildsTable, we show all of the builds by default (using the standard Django model API).

Each column in the ToasterTable corresponds to a field in the queryset. Inside a table row, a cell corresponds to one or more fields from each record in the queryset. A cell can show various aspects of a record: a formatted version of a field's value, an amalgam of multiple fields, a computation based on a group of related records etc. For example, in a row of the MiniBuildsTable, a cell might contain the outcome of the build, the number of tasks which failed during the build, or the time since the build completed.

The ''setup_columns()'' method defines which columns should be shown in the table. In this initial version of MiniBuildsTable, only the "completed_on" column is shown. This column has the following properties:
* It cannot be hidden (hideable=False).
* It can be used to order the table (orderable=True).
* static_data_name sets the name of the column so that it can be married up with the sort order specified in the querystring, and with the default_orderby for the ToasterTable (see below).
* static_data_template specifies how to render a value from a row into the HTML for the page. The data.completed_on reference in the template demonstrates how to access field values from the record for the row: the object representing the current row can be reference via the 'data' object in the static_data_template string. The whole of Django's template machinery is available when specifying how to render properties of the record for a row: you can include other templates and use filters and template tags (your own or Django's). In this case, the date filter is used to format the date into a human-readable form.

The self.default_orderby member set in __init__() specifies the default column to use for ordering the table. In this case, the completed_on field is used to sort the builds in the table; the "-" specifies reverse order, so the newest build appears at the top of the table. Note that the value for self.default_orderby should match the name of a field in the model, and the name of a column added to the table.

See the [???adding more columns] section for full details of the column options available.

=== Add the template ===

This should go into templates/, as this is where Django looks for view templates.

In all cases, you will need to include the toastertable.html template from your template. You will also need to add links for the jQuery UI CSS and JS files. (In an ideal world, this would be in toastertable.html, but for historical reasons, it isn't yet.)

Finally, you should define an xhr_table_url variable in your template. This is used to request new data for the table when filters or search terms are applied, or if a new page is requested. ToasterTable will send a request for the table JSON to this URL, then automatically refresh the table display with the new data when it's received.

As an example, here's a minimal template for the Mini Builds page, which would go in templates/minibuilds.html:

<pre>
{% extends 'base.html' %}
{% load static %}

{% block extraheadcontent %}
<link rel="stylesheet" href="{% static 'css/jquery-ui.min.css' %}" type='text/css'>
<link rel="stylesheet" href="{% static 'css/jquery-ui.structure.min.css' %}" type='text/css'>
<link rel="stylesheet" href="{% static 'css/jquery-ui.theme.min.css' %}" type='text/css'>
<script src="{% static 'js/jquery-ui.min.js' %}"></script>
{% endblock %}

{% block title %}{{title}}{% endblock %}

{% block pagecontent %}
<h1 class="page-header top-air">{{title}}</h1>
{% url 'minibuilds' as xhr_table_url %}
{% include 'toastertable.html' %}
{% endblock %}
</pre>

Depending on which side menus or other wrapping you need around your table, you may need to create a template by adapting an existing one (particularly if you are porting a table to ToasterTable).

Note that the template above extends base.html, which provides the header/footer for Toaster itself but doesn't have a side menu. For an example of a more complex page which does have a side menu, see generic-toastertable-page.html and its parent template, baseprojectpage.html.

The important points to remember, though, are shown in the example template above; in particular, the need to import the jQuery UI assets, set xhr_table_url, and include the toastertable.html template.

=== Add a URL mapping for the table ===

To be able to view the Mini Builds page, it needs a mapping in the urls.py file:

<pre>
urlpatterns = patterns('toastergui.views',
// ... other mappings ...

url(r'^minibuilds/$',
tables.MiniBuildsTable.as_view(template_name='minibuilds.html'),
name='minibuilds')
)
</pre>

This is standard Django template mapping code. The only wrinkle is that because a ToasterTable is a Django TemplateView subclass, we call Django's as_view() method on our table class to render it, passing the name of the template file. (Usually, a view mapping would specify a function in the views.py file.)

Now, with Toaster running locally, you should be able to visit

http://localhost:8000/toastergui/minibuilds/

in a browser and see your Mini Builds table (note that it will be empty unless you've run some builds).

[[File:MiniBuildsTable.png|750px|Adding a project name column to MiniBuildsTable]]

=== Aside: turning off ToasterTable caching ===

By default, ToasterTable caches data to prevent the same data being fetched multiple times. However, during development, this may mean that you see stale data in the table (for example, I changed the setup_columns() method in MiniBuildsTable to remove a column, but when I refreshed the page, the column was still present).

To get around this, pass the nocache option in the querystring in your browser, e.g.

http://localhost:8000/toastergui/minibuilds?nocache=true

This ensures that ToasterTable doesn't cache data between page refreshes.

== Further ToasterTable configuration ==

The following sections flesh out the options for adding columns to a ToasterTable, adding more non-table data to the page, and using the table filter APIs.

=== Column options ===

As stated previously, columns are added to the table inside the setup_columns() method of your ToasterTable subclass. The following options can be set when adding a column to a table:

* '''title''' (string): The heading for the column.
* '''help_text''' (string; optional): Text which describes the column content of the column; this is shown in a popover when the question mark next to the column heading is hovered over. If not set, the column doesn't have a help icon.
* '''hideable''' (boolean; default=True): True if the user can hide the column (using the "Edit columns" drop-down).
* '''hidden''' (boolean; default=False): True if the column is hidden by default; the user can show the column using the "Edit columns" drop-down.
* '''field_name''' (string; optional): Name of the property to render into the field. Note that this can be a property or method on the model being rendered (e.g. for a Build object, completed_on) ; it could also be a property of a related object (e.g. for a Build, project__name).
* '''static_data_name''' (string; optional): This should ''not'' be set if field_name is set, but ''must'' be set if static_data_template is set.
* '''static_data_template''' (string; optional): The template to render for each row; the data for the row is interpolated into the template. This is more flexible than field_name, as you can add links, conditional statements and additional formatting each time the cell for this column is rendered for a record.
* '''orderable''' (boolean; default=False): True if the table can be ordered by this column. Note that if this is True, either field_name or static_data_name should match a name of one of the fields in the queryset for the ToasterTable. If not, Toaster will not be able to sort the table by that column.
* '''filter_name''' (string; optional): Name of the TableFilter associated with this column, which adds filtering behaviour for this column; see the [???Filter API] section for full details.

As an example, I'll add two columns to the Mini Builds table.

First, a column to show the project name for the build:

<pre>
class MiniBuildsTable(ToasterTable):
# ... other code ...

def setup_columns(self, *args, **kwargs):
self.add_column(title='Completed on',
help_text='The date and time when the build finished',
hideable=False,
orderable=True,
static_data_name='completed_on',
static_data_template='{{data.completed_on | date:"d/m/y H:i"}}')

self.add_column(title='Project',
help_text='The project associated with this build',
hideable=True,
orderable=True,
field_name='project__name')
</pre>

The rest of the MiniBuildsTable remains the same: only setup_columns() changes. In this case, I'm showing the project name for the build using the field_name property for the column. This is simple, but doesn't allow me to modify how the value is formatted or add any HTML elements around it.

Try http://localhost:8000/toastergui/minibuilds/ again and you should see the new column. Note that this column can be hidden in the "Edit columns" drop-down, and can also be used to order the table.

[[File:MiniBuildsTable-projectname.png|750px|Adding a project name column to MiniBuildsTable]]

Next, a column to show the outcome for the build, with a link to that build's dashboard:

<pre>
class MiniBuildsTable(ToasterTable):
# ... other code ...

def setup_columns(self, *args, **kwargs):
outcome_template = '''
<a href="{% url 'builddashboard' data.id %}">
{% if data.outcome == 0 %}
succeeded
{% elif data.outcome == 1 %}
failed
{% endif %}
</a>
'''

self.add_column(title='Completed on',
help_text='The date and time when the build finished',
hideable=False,
orderable=True,
static_data_name='completed_on',
static_data_template='{{data.completed_on | date:"d/m/y H:i"}}')

self.add_column(title='Project',
help_text='The project associated with this build',
hideable=True,
orderable=True,
field_name='project__name')

self.add_column(title='Outcome',
help_text='The outcome of the build',
hideable=True,
orderable=True,
static_data_name='outcome',
static_data_template=outcome_template)
</pre>

In this case, static_data_template is used as we don't just want the value of the outcome field for the row: we also want to create a link. The template is slightly more complex than previous ones, so it's in a variable to make the code cleaner. Note the use of the built-in Django ''url'' template tag to get the URL for the build dashboard.

[[File:MiniBuildsTable-outcome.png|750px|Adding an outcome column to MiniBuildsTable]]

There is one issue with outcome_template: it compares the outcome field's value with integer values (0 for succeeded, 1 for failed) to determine what to show in the cell. This is a bit opaque for any developer coming to the project later, and also fragile if the integer values representing build outcomes change at a later date. It would be better to use the constants for Build outcomes (Build.SUCCEEDED, Build.FAILED) instead. However, the Build object is not accessible to the template, as it isn't passed in with the template context. The next section explains how to deal with this and similar missing context.

=== Additional context data ===

Sometimes a page will require data which is not in the queryset. This data may be shown on the page, or may influence how other data is rendered. For example, in the previous section, the outcome was displayed according to whether the build was a success or had some other outcome; but, rather than using hard-coded integer values, it would be better to use the Build.SUCCEEDED and Build.FAILED constants for this.

As another example, the Mini Builds page could show the time of the most recent build. While this could be done with some work in the template, it would be much cleaner to add a variable for "most_recent_build" to the template context instead.

In both cases, the required data must be added to the extra context data for the page. This can be done in one of two ways, depending on how you want to use it:

# If the data needs to be available to column templates, it should be added to the ''static_context_extra'' dictionary for the ToasterTable. For example, because we need to access constants on the Build class in the column templates, we should add Build to static_context_extra.
# If the data needs to be available to other templates (e.g. minibuilds.html in the case of Mini Builds), it should be set in the ''get_context_data()'' method.

==== Data for column templates ====

Data can be made available to column templates for a ToasterTable by adding it to the static_context_extra dictionary inside the __init__() method. This makes the data available for use in column templates (set using static_data_template). However, this data is ''only'' accessible from column templates, and is ''not'' available to the other templates used to render the page.

For example, to add the Build class to static_context_extra so that it can be used in a column template:

<pre>
class MiniBuildsTable(ToasterTable):
def __init__(self, *args, **kwargs):
super(MiniBuildsTable, self).__init__(*args, **kwargs)
self.default_orderby = '-completed_on'
self.title = 'Mini Builds Table'

# add the Build class to the static context
self.static_context_extra['Build'] = Build

def setup_queryset(self, *args, **kwargs):
self.queryset = Build.objects.all()

def setup_columns(self, *args, **kwargs):
# reference constants on Build from a column template
outcome_template = '''
<a href="{% url 'builddashboard' data.id %}">
{% if data.outcome == extra.Build.SUCCEEDED %}
succeeded
{% elif data.outcome == extra.Build.FAILED %}
failed
{% endif %}
</a>
'''

# ... the rest of setup_columns is the same ...
</pre>

To refer to data from static_context_extra in a column template, use the ''extra'' keyword (similar to how the ''data'' keyword works). In this case, I referenced the constants on the Build class with ''extra.Build.SUCCEEDED'' and ''extra.Build.FAILED''.

The page renders the same as before, but we no longer have hard-coded integer values in the column templates.

==== Data for other templates ====

Adding other data to the context for use in the page template is the same as for other Django TemplateView objects. See [https://docs.djangoproject.com/en/1.9/topics/class-based-views/generic-display/#adding-extra-context the Django documentation] for full details.

As a simple worked example, here's how we could show the most recent Toaster build in the Mini Builds page. First, we need to implement get_context_data() and add our own data to it; in this case, a "most_recent_build" property containing a Build object:

<pre>
class MiniBuildsTable(ToasterTable):
# ... other methods remain as they were ...

def get_context_data(self, **kwargs):
# invoke the super class' method, to include data like the page
# title in the context
context = super(MiniBuildsTable, self).get_context_data(**kwargs)

# add the most recent build to the context
all_builds = Build.objects.all().order_by('-completed_on')
context['most_recent_build'] = all_builds.first()

return context
</pre>

All of the properties on the context object returned by get_context_data() are available in the page template, as per the context for a standard Django template. To show the most recent build in the minibuilds.html template, we now use the most_recent_build property we added in get_context_data(), we modify the pagecontent block:

<pre>
{% block pagecontent %}
<h1 class="page-header top-air">{{title}}</h1>


{% if most_recent_build %}
<p>Most recent build completed: {{most_recent_build.completed_on}}</p>
{% endif %}

{% url 'minibuilds' as xhr_table_url %}
{% include 'toastertable.html' %}
{% endblock %}
</pre>

The result looks like this:

[[File:MiniBuildsTable-mostrecentbuild.png|750px|Showing the most recent build in the MiniBuildsTable]]

==== Dynamic data for other templates ====

If you need different data in the context depending on the URL (for example, you want an object retrieved using the ID in the URL), the pattern is similar to the above. The main difference is that you make use of the kwargs parameter passed to get_context_data(), which contains parameters derived from the page's URL.

As an example, the project builds table shows all of the builds for a project in a ToasterTable. It makes sense to add the project to the context so that its name can be shown at the top of the page. For a URL like http://localhost:8000/toastergui/project/X/builds, Toaster assigns the "X" in the URL to a parameter called ''pid''; this can be retrieved in get_context_data() and used to fetch the project for which we are showing builds:

<pre>
class ProjectBuildsTable(ToasterTable):
# ... other methods ...

def get_context_data(self, **kwargs):
context = super(ProjectBuildsTable, self).get_context_data(**kwargs)

# use the pid parameter extracted from the URL
context['project'] = Project.objects.get(pk=kwargs['pid'])

return context
</pre>

This can then be referenced the usual way in the template (e.g. <code>{{project}}</code>).

=== Search ===

Search is automatically enabled on a ToasterTable. However, you may find that you are unable to search on the fields you would like to. This section explains how to fix that.

By default, ToasterTable searches against the queryset's model, using an OR query with icontains (case insensitive, contains string) matching. The fields used for the search are defined by the search_allowed_fields property of the model.

For MiniBuildsTable, the queryset's model is orm.models.Build; search_allowed_fields (at the time of writing) for Build is set to:

<pre>
['machine', 'cooker_log_path', 'target__target', 'target__target_image_file__file_name']
</pre>

Doing a search for a string like "test" would therefore construct a query OR clause like the following (pseudo-SQL):

<pre>
... WHERE machine LIKE '%test%' OR cooker_log_path LIKE '%test%' OR target.target LIKE '%test%'
OR target.target_image_file.file_name LIKE '%test%'
</pre>

(The actual SQL clause is far more complicated, as Django would have to use various JOIN statements to combine rows from multiple database tables.)

Note that the search_allowed_fields don't have to be fields of the model: they can be fields in related models (here, the Target and TargetImageFile models related to a Build).

This means that it's not possible to search by project name in the MiniBuildsTable. To make this possible, we could modify search_allowed_fields for the Build model to:

<pre>
['machine', 'cooker_log_path', 'target__target', 'target__target_image_file__file_name', 'project__name']
</pre>

This would now allow the Mini Builds table to be searched by project name as well.

== Filter API ==

The classes for creating filters are defined in tablefilter.py.

A filter applies to a single column in the table. Any actions on a filter should filter records according to criteria which make sense with respect to that column. For example, you wouldn't add a filter to the "Completed on" column of the "All Builds" table which filters builds according to whether they failed or succeeded; but you would add a filter which filters the builds based on the time when the build was completed.

A filter adds an option to the column heading to filter in/out particular records or to remove filtering altogether.

Each filter has one or more actions associated with it. A filter also automatically gets a default action which turns off the filter and shows all the records again.

Each action changes the records shown in its associated ToasterTable, by applying a set of criteria to the query used to fetch records (e.g. show records for a single date, for a date range, or matching/not matching some other criteria related to the column).

To add a filter to a column, do the following:

# Create a TableFilter object for the column.
# Assign the filter to the column in the ToasterTable.
# Attach TableFilterAction objects to the TableFilter.

These steps are explained in detail below.

=== Add a table filter ===

The TableFilter class acts as a container for a group of TableFilterAction objects.

To create a TableFilter, instantiate it with a unique identifier (unique to the ToasterTable) and a text string which is displayed in the popup for the filter. The filters should be defined in the setup_filters() method for the ToasterTable.

Once the filter is defined, add it to the ToasterTable using the add_filter() method.

Here's an example of a TableFilter for the outcome column of MiniBuildsTable:

<pre>
# already imported in tables.py, but shown here for completeness
from toastergui.tablefilter import TableFilter

class MiniBuildsTable(ToasterTable):
# ... other methods ...

def setup_filters(self, *args, **kwargs):
# filter by outcome (succeeded or failed)
outcome_filter = TableFilter(
'outcome_filter',
'Filter builds by outcome'
)

# add the filter to the ToasterTable
self.add_filter(outcome_filter)
</pre>

Internally, add_filter() uses a TableFilterMap in the ToasterTable, which maps from column names to TableFilters.

Note that the filter has no actions, so won't actually filter the table yet.

=== Associate a filter with a column ===

To associate the filter with a column, pass the name of the filter in the filter_name argument for the column. For example, to associate the outcome_filter defined above with the outcome column, change setup_columns() like this:

<pre>
class MiniBuildsTable(ToasterTable):
# ... other methods ...

def setup_columns(self, *args, **kwargs):
# ... other column definitions ...

self.add_column(title='Outcome',
help_text='The outcome of the build',
hideable=True,
orderable=True,
static_data_name='outcome',
static_data_template=outcome_template,
filter_name='outcome_filter')
</pre>

(Note the additional filter_name keyword passed to add_column().)

Now if you visit http://localhost:8000/toastergui/minibuilds, the outcome column in the table should show the filter icon; clicking on this opens the popup which allows a user to select the filter to apply. Because we have no actions on the filter, the only option is the "All" one:

[[File:MiniBuildsTable-outcomefilter.png|750px|Adding an outcome filter to the MiniBuildsTable]]

outcome_filter is the string used to refer to the filter in the querystring; see the section [How filters are applied] for more details about how filters are applied.

=== Add table filter actions ===

The next step is to add actions to the filter.

TableFilterAction is the base class for an action associated with a filter. The following types of filter action (subclasses of TableFilterAction) are already implemented:

* TableFilterActionToggle: filter the records shown in the table by some arbitrary criteria; the action is either on or off.
* TableFilterActionDay: filter the records shown by day (yesterday or today).
* TableFilterActionDateRange: filter the records shown by a from/to date range.

In the case of TableFilterActionDay and TableFilterActionDateRange, you specify the field which is used for the filter action. In the case of TableFilterActionToggle, you can use arbitrary criteria for the action.

To add an action to a filter, create an instance of the desired action class and use the TableFilter.add_action() method to associate it with a filter. For example, here's how to add two actions to the outcome_filter defined earlier; one to show successful builds, and the other to show failed builds:

<pre>
# already imported in tables.py, but shown here for completeness
from toastergui.tablefilter import TableFilter
from toastergui.tablefilter import TableFilterActionToggle

class MiniBuildsTable(ToasterTable):
# ... other methods ...

def setup_filters(self, *args, **kwargs):
# filter by outcome (succeeded or failed)
outcome_filter = TableFilter(
'outcome_filter',
'Filter builds by outcome'
)

successful_builds_action = TableFilterActionToggle(
'successful_builds',
'Successful builds',
Q(outcome=Build.SUCCEEDED)
)

failed_builds_action = TableFilterActionToggle(
'failed_builds',
'Failed builds',
Q(outcome=Build.FAILED)
)

outcome_filter.add_action(successful_builds_action)
outcome_filter.add_action(failed_builds_action)

# add the filter to the ToasterTable
self.add_filter(outcome_filter)
</pre>

Here's how this will render:

[[File:MiniBuildsTable-outcomefilteractions.png|750px|Adding outcome filter actions to the MiniBuildsTable]]

I'll break this down further to give some detail about how the filter action is created. Here's the code which creates the filter action to show successful builds only:

<pre>
successful_builds_action = TableFilterActionToggle(
'successful_builds',
'Successful builds',
Q(outcome=Build.SUCCEEDED)
)
</pre>

The arguments to TableFilterActionToggle have the following meanings:
* 'successful_builds' is the name of the action. This is used to map from the filter parameter in the querystring.
* 'Successful builds' is the label shown next to the radio button which activates this action in the popup.
* Q(outcome=Build.SUCCEEDED) shows the criteria used to filter the records in the table's queryset when the filter action is applied.

The next section explains how we go from clicking on a radio button in the filter popup to filtering the records shown in the table.

=== How filters are applied ===

The short version:

When a ToasterTable is rendered, a filter icon is shown on any column which has a filter_name defined for it. Clicking on this icon populates the filter dialog, then opens it so the user can select a filter to apply. When the user clicks on the "Apply" button in the dialog, the page is requested with the filter name, filter action, and filter value in the querystring. These parameters are then used to filter the records shown in the table.

The long version:

* Each filter icon has a filter name associated with it.
* When a filter icon is clicked, the Toaster UI makes an Ajax request to the Toaster back-end for data about that filter name.
* When the filter data is received (in JSON format), the filter dialog is populated with radio buttons (one per action), labels (one per action) and any additional fields (e.g. date range fields for TableFilterActionDateRange actions). The count of records which will be returned by that action is part of the data returned by the server; if this is 0, the label and radio button are disabled. See static/js/table.js for the code which populates the filter dialog.
* The dialog is opened so the user can choose a filter to apply. The user clicks radio buttons, fills in fields etc.
* When the user clicks on the "Apply" button, the URL for the page is modified in place to reflect the filter criteria. The filter is represented in the URL by two querystring parameters:
** filter=<filter name>:<filter action> : the filter name maps to the name used when creating the TableFilter object; and the filter action corresponds to one of the names of a TableFilterAction object added to the TableFilter. For example, filter=outcome_filter:successful_builds will map to the outcome_filter TableFilter, and its successful_builds TableFilterAction.
** filter_value=<filter value string> : for a TableFilterActionToggle filter, this is always "on", to show that the filter is applied; for a TableFilterActionDay this is either "today" or "yesterday"; for a TableFilterActionDateRange, this is a "from,to" date range in the format "2015-12-09,2015-12-11".
* The table data is fetched via Ajax, using the filter and filter_value parameters as part of the URL. These set the filter name, action and value to use for filtering.
* The back-end applies the requested filter to the queryset (as well as any existing search string) and returns it as JSON.
* The Toaster UI code redraws the table with the filtered queryset.

File:MiniBuildsTable-outcomefilteractions.png

2016-04-05T14:06:04Z

Elliot Smith:

ToasterTable

2016-04-05T14:04:10Z

Elliot Smith:

'''*******This is a WIP*******'''

== Introduction to ToasterTable ==

ToasterTable is a set of classes in the toastergui module of the Toaster Django application. It is used to present a set of database records in a style and with functionality consistent with the rest of Toaster.

The functionality provided by ToasterTable includes:
* Sorting
* Column hiding/showing
* Filtering
* Paging
* Search

When a ToasterTable is updated, a new set of records matching the filter criteria, search string, sort order and page number is fetched from the back-end using Ajax requests. The ToasterTable is then updated in place from the JSON in the response, using JavaScript to redraw the table rows.

As a developer, you don't need to worry (necessarily) about how this works: you just implement a subclass of ToasterTable, specify a template for rendering it, and supply a URL mapping.

The following sections explain how to use ToasterTable. The filenames in these sections refer to files in the toastergui/ directory.

Note that to be able to follow the example, you will need to set up Toaster for development work. This is described in [http://www.yoctoproject.org/docs/2.0.1/toaster-manual/toaster-manual.html the Toaster manual].

== How to create a ToasterTable ==

Most of the tables needed for Toaster are already implemented. Many of the key tables in Toaster already use ToasterTable, but some don't. A full list of unconverted tables is in https://bugzilla.yoctoproject.org/show_bug.cgi?id=8363.

This means that you will typically be porting an existing table to ToasterTable, rather than adding a new table from scratch. However, to keep things simple, this section explains how to create a ToasterTable from scratch. Knowing how to do this should make it easier to port an existing table to ToasterTable in future.

To provide a worked example, I'll create a new table which is a variant of the existing "all builds" table, using a reduced number of columns and filters.

=== Create the table class ===

A quick overview of where things are and where they go when creating a table:

* widgets.py contains the ToasterTable class. This is the base class for all ToasterTables.
* New tables are added to tables.py.
* A table in tables.py maps to a URL within the Toaster application; the mapping is defined in urls.py.
* Templates for the table are added to templates/.
* If you need [https://docs.djangoproject.com/en/1.9/howto/custom-template-tags/ custom template tags], these go in templatetags/.

To create a new table, first add the class definition to tables.py:

<pre>
# import any models you need for the table
from orm.models import Build

class MiniBuildsTable(ToasterTable):
def __init__(self, *args, **kwargs):
super(MiniBuildsTable, self).__init__(*args, **kwargs)
self.default_orderby = '-completed_on'
self.title = 'Mini Builds Table'

def setup_queryset(self, *args, **kwargs):
self.queryset = Build.objects.all()

def setup_columns(self, *args, **kwargs):
self.add_column(title='Completed on',
help_text='The date and time when the build finished',
hideable=False,
orderable=True,
static_data_name='completed_on',
static_data_template='{{data.completed_on | date:"d/m/y H:i"}}')
</pre>

The key to creating a ToasterTable is to decide which data is going to be shown in the table and define this in the ''setup_queryset()'' method. This will typically be a queryset derived from one model in the Toaster application. To see the available models, check the orm/models.py file.

Each row in the table corresponds to a record in the queryset. The ''setup_queryset()'' method specifies how to get this queryset for the table: it must set the self.queryset property to the Django queryset containing the data. In the MiniBuildsTable, we show all of the builds by default (using the standard Django model API).

Each column in the ToasterTable corresponds to a field in the queryset. Inside a table row, a cell corresponds to one or more fields from each record in the queryset. A cell can show various aspects of a record: a formatted version of a field's value, an amalgam of multiple fields, a computation based on a group of related records etc. For example, in a row of the MiniBuildsTable, a cell might contain the outcome of the build, the number of tasks which failed during the build, or the time since the build completed.

The ''setup_columns()'' method defines which columns should be shown in the table. In this initial version of MiniBuildsTable, only the "completed_on" column is shown. This column has the following properties:
* It cannot be hidden (hideable=False).
* It can be used to order the table (orderable=True).
* static_data_name sets the name of the column so that it can be married up with the sort order specified in the querystring, and with the default_orderby for the ToasterTable (see below).
* static_data_template specifies how to render a value from a row into the HTML for the page. The data.completed_on reference in the template demonstrates how to access field values from the record for the row: the object representing the current row can be reference via the 'data' object in the static_data_template string. The whole of Django's template machinery is available when specifying how to render properties of the record for a row: you can include other templates and use filters and template tags (your own or Django's). In this case, the date filter is used to format the date into a human-readable form.

The self.default_orderby member set in __init__() specifies the default column to use for ordering the table. In this case, the completed_on field is used to sort the builds in the table; the "-" specifies reverse order, so the newest build appears at the top of the table. Note that the value for self.default_orderby should match the name of a field in the model, and the name of a column added to the table.

See the [???adding more columns] section for full details of the column options available.

=== Add the template ===

This should go into templates/, as this is where Django looks for view templates.

In all cases, you will need to include the toastertable.html template from your template. You will also need to add links for the jQuery UI CSS and JS files. (In an ideal world, this would be in toastertable.html, but for historical reasons, it isn't yet.)

Finally, you should define an xhr_table_url variable in your template. This is used to request new data for the table when filters or search terms are applied, or if a new page is requested. ToasterTable will send a request for the table JSON to this URL, then automatically refresh the table display with the new data when it's received.

As an example, here's a minimal template for the Mini Builds page, which would go in templates/minibuilds.html:

<pre>
{% extends 'base.html' %}
{% load static %}

{% block extraheadcontent %}
<link rel="stylesheet" href="{% static 'css/jquery-ui.min.css' %}" type='text/css'>
<link rel="stylesheet" href="{% static 'css/jquery-ui.structure.min.css' %}" type='text/css'>
<link rel="stylesheet" href="{% static 'css/jquery-ui.theme.min.css' %}" type='text/css'>
<script src="{% static 'js/jquery-ui.min.js' %}"></script>
{% endblock %}

{% block title %}{{title}}{% endblock %}

{% block pagecontent %}
<h1 class="page-header top-air">{{title}}</h1>
{% url 'minibuilds' as xhr_table_url %}
{% include 'toastertable.html' %}
{% endblock %}
</pre>

Depending on which side menus or other wrapping you need around your table, you may need to create a template by adapting an existing one (particularly if you are porting a table to ToasterTable).

Note that the template above extends base.html, which provides the header/footer for Toaster itself but doesn't have a side menu. For an example of a more complex page which does have a side menu, see generic-toastertable-page.html and its parent template, baseprojectpage.html.

The important points to remember, though, are shown in the example template above; in particular, the need to import the jQuery UI assets, set xhr_table_url, and include the toastertable.html template.

=== Add a URL mapping for the table ===

To be able to view the Mini Builds page, it needs a mapping in the urls.py file:

<pre>
urlpatterns = patterns('toastergui.views',
// ... other mappings ...

url(r'^minibuilds/$',
tables.MiniBuildsTable.as_view(template_name='minibuilds.html'),
name='minibuilds')
)
</pre>

This is standard Django template mapping code. The only wrinkle is that because a ToasterTable is a Django TemplateView subclass, we call Django's as_view() method on our table class to render it, passing the name of the template file. (Usually, a view mapping would specify a function in the views.py file.)

Now, with Toaster running locally, you should be able to visit

http://localhost:8000/toastergui/minibuilds/

in a browser and see your Mini Builds table (note that it will be empty unless you've run some builds).

[[File:MiniBuildsTable.png|750px|Adding a project name column to MiniBuildsTable]]

=== Aside: turning off ToasterTable caching ===

By default, ToasterTable caches data to prevent the same data being fetched multiple times. However, during development, this may mean that you see stale data in the table (for example, I changed the setup_columns() method in MiniBuildsTable to remove a column, but when I refreshed the page, the column was still present).

To get around this, pass the nocache option in the querystring in your browser, e.g.

http://localhost:8000/toastergui/minibuilds?nocache=true

This ensures that ToasterTable doesn't cache data between page refreshes.

== Further ToasterTable configuration ==

The following sections flesh out the options for adding columns to a ToasterTable, adding more non-table data to the page, and using the table filter APIs.

=== Column options ===

As stated previously, columns are added to the table inside the setup_columns() method of your ToasterTable subclass. The following options can be set when adding a column to a table:

* '''title''' (string): The heading for the column.
* '''help_text''' (string; optional): Text which describes the column content of the column; this is shown in a popover when the question mark next to the column heading is hovered over. If not set, the column doesn't have a help icon.
* '''hideable''' (boolean; default=True): True if the user can hide the column (using the "Edit columns" drop-down).
* '''hidden''' (boolean; default=False): True if the column is hidden by default; the user can show the column using the "Edit columns" drop-down.
* '''field_name''' (string; optional): Name of the property to render into the field. Note that this can be a property or method on the model being rendered (e.g. for a Build object, completed_on) ; it could also be a property of a related object (e.g. for a Build, project__name).
* '''static_data_name''' (string; optional): This should ''not'' be set if field_name is set, but ''must'' be set if static_data_template is set.
* '''static_data_template''' (string; optional): The template to render for each row; the data for the row is interpolated into the template. This is more flexible than field_name, as you can add links, conditional statements and additional formatting each time the cell for this column is rendered for a record.
* '''orderable''' (boolean; default=False): True if the table can be ordered by this column. Note that if this is True, either field_name or static_data_name should match a name of one of the fields in the queryset for the ToasterTable. If not, Toaster will not be able to sort the table by that column.
* '''filter_name''' (string; optional): Name of the TableFilter associated with this column, which adds filtering behaviour for this column; see the [???Filter API] section for full details.

As an example, I'll add two columns to the Mini Builds table.

First, a column to show the project name for the build:

<pre>
class MiniBuildsTable(ToasterTable):
# ... other code ...

def setup_columns(self, *args, **kwargs):
self.add_column(title='Completed on',
help_text='The date and time when the build finished',
hideable=False,
orderable=True,
static_data_name='completed_on',
static_data_template='{{data.completed_on | date:"d/m/y H:i"}}')

self.add_column(title='Project',
help_text='The project associated with this build',
hideable=True,
orderable=True,
field_name='project__name')
</pre>

The rest of the MiniBuildsTable remains the same: only setup_columns() changes. In this case, I'm showing the project name for the build using the field_name property for the column. This is simple, but doesn't allow me to modify how the value is formatted or add any HTML elements around it.

Try http://localhost:8000/toastergui/minibuilds/ again and you should see the new column. Note that this column can be hidden in the "Edit columns" drop-down, and can also be used to order the table.

[[File:MiniBuildsTable-projectname.png|750px|Adding a project name column to MiniBuildsTable]]

Next, a column to show the outcome for the build, with a link to that build's dashboard:

<pre>
class MiniBuildsTable(ToasterTable):
# ... other code ...

def setup_columns(self, *args, **kwargs):
outcome_template = '''
<a href="{% url 'builddashboard' data.id %}">
{% if data.outcome == 0 %}
succeeded
{% elif data.outcome == 1 %}
failed
{% endif %}
</a>
'''

self.add_column(title='Completed on',
help_text='The date and time when the build finished',
hideable=False,
orderable=True,
static_data_name='completed_on',
static_data_template='{{data.completed_on | date:"d/m/y H:i"}}')

self.add_column(title='Project',
help_text='The project associated with this build',
hideable=True,
orderable=True,
field_name='project__name')

self.add_column(title='Outcome',
help_text='The outcome of the build',
hideable=True,
orderable=True,
static_data_name='outcome',
static_data_template=outcome_template)
</pre>

In this case, static_data_template is used as we don't just want the value of the outcome field for the row: we also want to create a link. The template is slightly more complex than previous ones, so it's in a variable to make the code cleaner. Note the use of the built-in Django ''url'' template tag to get the URL for the build dashboard.

[[File:MiniBuildsTable-outcome.png|750px|Adding an outcome column to MiniBuildsTable]]

There is one issue with outcome_template: it compares the outcome field's value with integer values (0 for succeeded, 1 for failed) to determine what to show in the cell. This is a bit opaque for any developer coming to the project later, and also fragile if the integer values representing build outcomes change at a later date. It would be better to use the constants for Build outcomes (Build.SUCCEEDED, Build.FAILED) instead. However, the Build object is not accessible to the template, as it isn't passed in with the template context. The next section explains how to deal with this and similar missing context.

=== Additional context data ===

Sometimes a page will require data which is not in the queryset. This data may be shown on the page, or may influence how other data is rendered. For example, in the previous section, the outcome was displayed according to whether the build was a success or had some other outcome; but, rather than using hard-coded integer values, it would be better to use the Build.SUCCEEDED and Build.FAILED constants for this.

As another example, the Mini Builds page could show the time of the most recent build. While this could be done with some work in the template, it would be much cleaner to add a variable for "most_recent_build" to the template context instead.

In both cases, the required data must be added to the extra context data for the page. This can be done in one of two ways, depending on how you want to use it:

# If the data needs to be available to column templates, it should be added to the ''static_context_extra'' dictionary for the ToasterTable. For example, because we need to access constants on the Build class in the column templates, we should add Build to static_context_extra.
# If the data needs to be available to other templates (e.g. minibuilds.html in the case of Mini Builds), it should be set in the ''get_context_data()'' method.

==== Data for column templates ====

Data can be made available to column templates for a ToasterTable by adding it to the static_context_extra dictionary inside the __init__() method. This makes the data available for use in column templates (set using static_data_template). However, this data is ''only'' accessible from column templates, and is ''not'' available to the other templates used to render the page.

For example, to add the Build class to static_context_extra so that it can be used in a column template:

<pre>
class MiniBuildsTable(ToasterTable):
def __init__(self, *args, **kwargs):
super(MiniBuildsTable, self).__init__(*args, **kwargs)
self.default_orderby = '-completed_on'
self.title = 'Mini Builds Table'

# add the Build class to the static context
self.static_context_extra['Build'] = Build

def setup_queryset(self, *args, **kwargs):
self.queryset = Build.objects.all()

def setup_columns(self, *args, **kwargs):
# reference constants on Build from a column template
outcome_template = '''
<a href="{% url 'builddashboard' data.id %}">
{% if data.outcome == extra.Build.SUCCEEDED %}
succeeded
{% elif data.outcome == extra.Build.FAILED %}
failed
{% endif %}
</a>
'''

# ... the rest of setup_columns is the same ...
</pre>

To refer to data from static_context_extra in a column template, use the ''extra'' keyword (similar to how the ''data'' keyword works). In this case, I referenced the constants on the Build class with ''extra.Build.SUCCEEDED'' and ''extra.Build.FAILED''.

The page renders the same as before, but we no longer have hard-coded integer values in the column templates.

==== Data for other templates ====

Adding other data to the context for use in the page template is the same as for other Django TemplateView objects. See [https://docs.djangoproject.com/en/1.9/topics/class-based-views/generic-display/#adding-extra-context the Django documentation] for full details.

As a simple worked example, here's how we could show the most recent Toaster build in the Mini Builds page. First, we need to implement get_context_data() and add our own data to it; in this case, a "most_recent_build" property containing a Build object:

<pre>
class MiniBuildsTable(ToasterTable):
# ... other methods remain as they were ...

def get_context_data(self, **kwargs):
# invoke the super class' method, to include data like the page
# title in the context
context = super(MiniBuildsTable, self).get_context_data(**kwargs)

# add the most recent build to the context
all_builds = Build.objects.all().order_by('-completed_on')
context['most_recent_build'] = all_builds.first()

return context
</pre>

All of the properties on the context object returned by get_context_data() are available in the page template, as per the context for a standard Django template. To show the most recent build in the minibuilds.html template, we now use the most_recent_build property we added in get_context_data(), we modify the pagecontent block:

<pre>
{% block pagecontent %}
<h1 class="page-header top-air">{{title}}</h1>


{% if most_recent_build %}
<p>Most recent build completed: {{most_recent_build.completed_on}}</p>
{% endif %}

{% url 'minibuilds' as xhr_table_url %}
{% include 'toastertable.html' %}
{% endblock %}
</pre>

The result looks like this:

[[File:MiniBuildsTable-mostrecentbuild.png|750px|Showing the most recent build in the MiniBuildsTable]]

==== Dynamic data for other templates ====

If you need different data in the context depending on the URL (for example, you want an object retrieved using the ID in the URL), the pattern is similar to the above. The main difference is that you make use of the kwargs parameter passed to get_context_data(), which contains parameters derived from the page's URL.

As an example, the project builds table shows all of the builds for a project in a ToasterTable. It makes sense to add the project to the context so that its name can be shown at the top of the page. For a URL like http://localhost:8000/toastergui/project/X/builds, Toaster assigns the "X" in the URL to a parameter called ''pid''; this can be retrieved in get_context_data() and used to fetch the project for which we are showing builds:

<pre>
class ProjectBuildsTable(ToasterTable):
# ... other methods ...

def get_context_data(self, **kwargs):
context = super(ProjectBuildsTable, self).get_context_data(**kwargs)

# use the pid parameter extracted from the URL
context['project'] = Project.objects.get(pk=kwargs['pid'])

return context
</pre>

This can then be referenced the usual way in the template (e.g. <code>{{project}}</code>).

=== Search ===

Search is automatically enabled on a ToasterTable. However, you may find that you are unable to search on the fields you would like to. This section explains how to fix that.

By default, ToasterTable searches against the queryset's model, using an OR query with icontains (case insensitive, contains string) matching. The fields used for the search are defined by the search_allowed_fields property of the model.

For MiniBuildsTable, the queryset's model is orm.models.Build; search_allowed_fields (at the time of writing) for Build is set to:

<pre>
['machine', 'cooker_log_path', 'target__target', 'target__target_image_file__file_name']
</pre>

Doing a search for a string like "test" would therefore construct a query OR clause like the following (pseudo-SQL):

<pre>
... WHERE machine LIKE '%test%' OR cooker_log_path LIKE '%test%' OR target.target LIKE '%test%'
OR target.target_image_file.file_name LIKE '%test%'
</pre>

(The actual SQL clause is far more complicated, as Django would have to use various JOIN statements to combine rows from multiple database tables.)

Note that the search_allowed_fields don't have to be fields of the model: they can be fields in related models (here, the Target and TargetImageFile models related to a Build).

This means that it's not possible to search by project name in the MiniBuildsTable. To make this possible, we could modify search_allowed_fields for the Build model to:

<pre>
['machine', 'cooker_log_path', 'target__target', 'target__target_image_file__file_name', 'project__name']
</pre>

This would now allow the Mini Builds table to be searched by project name as well.

== Filter API ==

The classes for creating filters are defined in tablefilter.py.

A filter applies to a single column in the table. Any actions on a filter should filter records according to criteria which make sense with respect to that column. For example, you wouldn't add a filter to the "Completed on" column of the "All Builds" table which filters builds according to whether they failed or succeeded; but you would add a filter which filters the builds based on the time when the build was completed.

A filter adds an option to the column heading to filter in/out particular records or to remove filtering altogether.

Each filter has one or more actions associated with it. A filter also automatically gets a default action which turns off the filter and shows all the records again.

Each action changes the records shown in its associated ToasterTable, by applying a set of criteria to the query used to fetch records (e.g. show records for a single date, for a date range, or matching/not matching some other criteria related to the column).

To add a filter to a column, do the following:

# Create a TableFilter object for the column.
# Assign the filter to the column in the ToasterTable.
# Attach TableFilterAction objects to the TableFilter.

These steps are explained in detail below.

=== Add a table filter ===

The TableFilter class acts as a container for a group of TableFilterAction objects.

To create a TableFilter, instantiate it with a unique identifier (unique to the ToasterTable) and a text string which is displayed in the popup for the filter. The filters should be defined in the setup_filters() method for the ToasterTable.

Once the filter is defined, add it to the ToasterTable using the add_filter() method.

Here's an example of a TableFilter for the outcome column of MiniBuildsTable:

<pre>
# already imported in tables.py, but shown here for completeness
from toastergui.tablefilter import TableFilter

class MiniBuildsTable(ToasterTable):
# ... other methods ...

def setup_filters(self, *args, **kwargs):
# filter by outcome (succeeded or failed)
outcome_filter = TableFilter(
'outcome_filter',
'Filter builds by outcome'
)

# add the filter to the ToasterTable
self.add_filter(outcome_filter)
</pre>

Internally, add_filter() uses a TableFilterMap in the ToasterTable, which maps from column names to TableFilters.

Note that the filter has no actions, so won't actually filter the table yet.

=== Associate a filter with a column ===

To associate the filter with a column, pass the name of the filter in the filter_name argument for the column. For example, to associate the outcome_filter defined above with the outcome column, change setup_columns() like this:

<pre>
class MiniBuildsTable(ToasterTable):
# ... other methods ...

def setup_columns(self, *args, **kwargs):
# ... other column definitions ...

self.add_column(title='Outcome',
help_text='The outcome of the build',
hideable=True,
orderable=True,
static_data_name='outcome',
static_data_template=outcome_template,
filter_name='outcome_filter')
</pre>

(Note the additional filter_name keyword passed to add_column().)

Now if you visit http://localhost:8000/toastergui/minibuilds, the outcome column in the table should show the filter icon; clicking on this opens the popup which allows a user to select the filter to apply. Because we have no actions on the filter, the only option is the "All" one:

[[File:MiniBuildsTable-outcomefilter.png|750px|Adding an outcome filter to the MiniBuildsTable]]

outcome_filter is the string used to refer to the filter in the querystring; see the section [How filters are applied] for more details about how filters are applied.

=== Add table filter actions ===

The next step is to add actions to the filter.

TableFilterAction is the base class for an action associated with a filter. The following types of filter action (subclasses of TableFilterAction) are already implemented:

* TableFilterActionToggle: filter the records shown in the table by some arbitrary criteria; the action is either on or off.
* TableFilterActionDay: filter the records shown by day (yesterday or today).
* TableFilterActionDateRange: filter the records shown by a from/to date range.

In the case of TableFilterActionDay and TableFilterActionDateRange, you specify the field which is used for the filter action. In the case of TableFilterActionToggle, you can use arbitrary criteria for the action.

To add an action to a filter, create an instance of the desired action class and use the TableFilter.add_action() method to associate it with a filter. For example, here's how to add two actions to the outcome_filter defined earlier; one to show successful builds, and the other to show failed builds:

<pre>
# already imported in tables.py, but shown here for completeness
from toastergui.tablefilter import TableFilter
from toastergui.tablefilter import TableFilterActionToggle

class MiniBuildsTable(ToasterTable):
# ... other methods ...

def setup_filters(self, *args, **kwargs):
# filter by outcome (succeeded or failed)
outcome_filter = TableFilter(
'outcome_filter',
'Filter builds by outcome'
)

successful_builds_action = TableFilterActionToggle(
'successful_builds',
'Successful builds',
Q(outcome=Build.SUCCEEDED)
)

failed_builds_action = TableFilterActionToggle(
'failed_builds',
'Failed builds',
Q(outcome=Build.FAILED)
)

outcome_filter.add_action(successful_builds_action)
outcome_filter.add_action(failed_builds_action)

# add the filter to the ToasterTable
self.add_filter(outcome_filter)
</pre>

I'll break this down further to give some detail about how the filter action is created. Here's the code which creates the filter action to show successful builds only:

<pre>
successful_builds_action = TableFilterActionToggle(
'successful_builds',
'Successful builds',
Q(outcome=Build.SUCCEEDED)
)
</pre>

The arguments to TableFilterActionToggle have the following meanings:
* 'successful_builds' is the name of the action. This is used to map from the filter parameter in the querystring.
* 'Successful builds' is the label shown next to the radio button which activates this action in the popup.
* Q(outcome=Build.SUCCEEDED) shows the criteria used to filter the records in the table's queryset when the filter action is applied.

The next section explains how we go from clicking on a radio button in the filter popup to filtering the records shown in the table.

=== How filters are applied ===

The short version:

When a ToasterTable is rendered, a filter icon is shown on any column which has a filter_name defined for it. Clicking on this icon populates the filter dialog, then opens it so the user can select a filter to apply. When the user clicks on the "Apply" button in the dialog, the page is requested with the filter name, filter action, and filter value in the querystring. These parameters are then used to filter the records shown in the table.

The long version:

* Each filter icon has a filter name associated with it.
* When a filter icon is clicked, the Toaster UI makes an Ajax request to the Toaster back-end for data about that filter name.
* When the filter data is received (in JSON format), the filter dialog is populated with radio buttons (one per action), labels (one per action) and any additional fields (e.g. date range fields for TableFilterActionDateRange actions). The count of records which will be returned by that action is part of the data returned by the server; if this is 0, the label and radio button are disabled. See static/js/table.js for the code which populates the filter dialog.
* The dialog is opened so the user can choose a filter to apply. The user clicks radio buttons, fills in fields etc.
* When the user clicks on the "Apply" button, the URL for the page is modified in place to reflect the filter criteria. The filter is represented in the URL by two querystring parameters:
** filter=<filter name>:<filter action> : the filter name maps to the name used when creating the TableFilter object; and the filter action corresponds to one of the names of a TableFilterAction object added to the TableFilter. For example, filter=outcome_filter:successful_builds will map to the outcome_filter TableFilter, and its successful_builds TableFilterAction.
** filter_value=<filter value string> : for a TableFilterActionToggle filter, this is always "on", to show that the filter is applied; for a TableFilterActionDay this is either "today" or "yesterday"; for a TableFilterActionDateRange, this is a "from,to" date range in the format "2015-12-09,2015-12-11".
* The table data is fetched via Ajax, using the filter and filter_value parameters as part of the URL. These set the filter name, action and value to use for filtering.
* The back-end applies the requested filter to the queryset (as well as any existing search string) and returns it as JSON.
* The Toaster UI code redraws the table with the filtered queryset.

File:MiniBuildsTable-outcomefilter.png

2016-04-05T13:06:24Z

Elliot Smith:

ToasterTable

2016-04-05T13:06:03Z

2016-04-04T15:07:57Z

Elliot Smith:

'''This is a WIP'''

== Introduction to ToasterTable ==

ToasterTable is a set of classes in the toastergui module of the Toaster Django application. It is used to present a set of database records in a style and with functionality consistent with the rest of Toaster.

The functionality provided by ToasterTable includes:
* Sorting
* Column hiding/showing
* Filtering
* Paging
* Search

When a ToasterTable is updated, a new set of records matching the filter criteria, search string, sort order and page number is fetched from the back-end using Ajax requests. The ToasterTable is then updated in place from the JSON in the response, using JavaScript to redraw the table rows.

As a developer, you don't need to worry (necessarily) about how this works: you just implement a subclass of ToasterTable, specify a template for rendering it, and supply a URL mapping.

The following sections explain how to use ToasterTable. The filenames in these sections refer to files in the toastergui/ directory.

Note that to be able to follow the example, you will need to set up Toaster for development work. This is described in [http://www.yoctoproject.org/docs/2.0.1/toaster-manual/toaster-manual.html the Toaster manual].

== How to create a ToasterTable ==

Most of the tables needed for Toaster are already implemented. Many of the key tables in Toaster already use ToasterTable, but some don't. A full list of unconverted tables is in https://bugzilla.yoctoproject.org/show_bug.cgi?id=8363.

This means that you will typically be porting an existing table to ToasterTable, rather than adding a new table from scratch. However, to keep things simple, this section explains how to create a ToasterTable from scratch. Knowing how to do this should make it easier to port an existing table to ToasterTable in future.

To provide a worked example, I'll create a new table which is a variant of the existing "all builds" table, using a reduced number of columns and filters.

=== Create the table class ===

A quick overview of where things are and where they go when creating a table:

* widgets.py contains the ToasterTable class. This is the base class for all ToasterTables.
* New tables are added to tables.py.
* A table in tables.py maps to a URL within the Toaster application; the mapping is defined in urls.py.
* Templates for the table are added to templates/.
* If you need [https://docs.djangoproject.com/en/1.9/howto/custom-template-tags/ custom template tags], these go in templatetags/.

To create a new table, first add the class definition to tables.py:

<nowiki>
# import any models you need for the table
from orm.models import Build

class MiniBuildsTable(ToasterTable):
def __init__(self, *args, **kwargs):
super(MiniBuildsTable, self).__init__(*args, **kwargs)
self.default_orderby = '-completed_on'
self.title = 'Mini Builds Table'

def setup_queryset(self, *args, **kwargs):
self.queryset = Build.objects.all()

def setup_columns(self, *args, **kwargs):
self.add_column(title='Completed on',
help_text='The date and time when the build finished',
hideable=False,
orderable=True,
static_data_name='completed_on',
static_data_template='{{data.completed_on | date:"d/m/y H:i"}}')
</nowiki>

The key to creating a ToasterTable is to decide which data is going to be shown in the table and define this in the ''setup_queryset()'' method. This will typically be a queryset derived from one model in the Toaster application. To see the available models, check the orm/models.py file.

Each row in the table corresponds to a record in the queryset. The ''setup_queryset()'' method specifies how to get this queryset for the table: it must set the self.queryset property to the Django queryset containing the data. In the MiniBuildsTable, we show all of the builds by default (using the standard Django model API).

Each column in the ToasterTable corresponds to a field in the queryset. Inside a table row, a cell corresponds to one or more fields from each record in the queryset. A cell can show various aspects of a record: a formatted version of a field's value, an amalgam of multiple fields, a computation based on a group of related records etc. For example, in a row of the MiniBuildsTable, a cell might contain the outcome of the build, the number of tasks which failed during the build, or the time since the build completed.

The ''setup_columns()'' method defines which columns should be shown in the table. In this initial version of MiniBuildsTable, only the "completed_on" column is shown. This column has the following properties:
* It cannot be hidden (hideable=False).
* It can be used to order the table (orderable=True).
* static_data_name sets the name of the column so that it can be married up with the sort order specified in the querystring, and with the default_orderby for the ToasterTable (see below).
* static_data_template specifies how to render the value from the record into the HTML for the page. The data.completed_on reference in the template demonstrates how to access

The self.default_orderby member set in __init__() specifies the default column to use for ordering the table. In this case, the completed_on field is used to sort the builds in the table; the "-" specifies reverse order, so the newest build appears at the top of the table. Note that the value for self.default_orderby should match the name of a field in the model, and the name of a column added to the table.

See the [adding more columns] section for full details of the column options available.

=== Adding the template ===

This should go into templates/, as this is where Django looks for view templates.

In all cases, you will need to include the toastertable.html template from your template. You will also need to add links for the jQuery UI CSS and JS files. (In an ideal world, this would be in toastertable.html, but for historical reasons, it isn't yet.)

You will also need to define an xhr_table_url variable in your template. This is used to request new data for the table when filters or search terms are applied, or if a new page is requested. ToasterTable will send a request for the table JSON to this URL, then automatically refresh the table display with the new data when it's received.

As an example, here's a minimal template for the Mini Builds page, which would be in templates/minibuilds.html.

<nowiki>
{% extends 'base.html' %}
{% load static %}

{% block extraheadcontent %}
<link rel="stylesheet" href="{% static 'css/jquery-ui.min.css' %}" type='text/css'>
<link rel="stylesheet" href="{% static 'css/jquery-ui.structure.min.css' %}" type='text/css'>
<link rel="stylesheet" href="{% static 'css/jquery-ui.theme.min.css' %}" type='text/css'>
<script src="{% static 'js/jquery-ui.min.js' %}"></script>
{% endblock %}

{% block title %}{{title}}{% endblock %}

{% block pagecontent %}
<h1 class="page-header top-air">{{title}}</h1>
{% url 'minibuilds' as xhr_table_url %}
{% include 'toastertable.html' %}
{% endblock %}
</nowiki>

Depending on which side menus or other wrapping you need around your table, you may need to modify an existing template (particularly if you are porting a table to ToasterTable). Note that this template extends the base.html template, which provides the header/footer for Toaster itself, but doesn't have a side menu. For an example of a more complex page, see builds-toastertable.html.

The important things to remember, though, are in the example template above; in particular, the need to import the jQuery UI assets, set xhr_table_url, and include the toastertable.html template.

=== Add a URL mapping for the table ===

To be able to view the Mini Builds page, it needs a mapping in the urls.py file:

<nowiki>
urlpatterns = patterns('toastergui.views',
// ... other mappings ...

url(r'^minibuilds/$',
tables.MiniBuildsTable.as_view(template_name='minibuilds.html'),
name='minibuilds')
)
</nowiki>

This is standard Django template mapping code. The only wrinkle is that because a ToasterTable is a Django TemplateView subclass, we call Django's as_view() method on our table to render it, passing the name of the template file.

Now, with Toaster running locally, you should be able to visit

http://localhost:8000/minibuilds/

in a browser and see your Mini Builds table (note that it will be empty unless you've run some builds).

== More ToasterTable ==

The following sections flesh out the options for adding columns to a ToasterTable, adding more data to the page, and using the table filter APIs.

=== Adding more columns ===

???

Column options

Adding a column to a table:
- Sorting by a column
- Hiding/showing a column
- Filtering: The Filter API is used to add filtering functionality to particular columns in the table.

=== Other context data ===

Data not in each record of the queryset, but which needs to be visible on the page. As an example, the project builds page shows all of the builds for a project; but the project name needs to be shown at the top of the page. In this case, the project is added to the extra context data for the ToasterTable and accessed in the template.

As an example, we could compute the total number of builds Toaster has recorded and show this in the title at the top of the Mini Builds page.

???

Note that if you want this value to update as the table is filtered, you'll need to do it in JavaScript. This is because the page for a ToasterTable isn't re-rendered when its data changes: only the HTML for the table is updated. Any additional updates required on the page have to be done similarly, via JavaScript.

For example, to update the title of the page as the table is filtered, we could add this code to the template:

???

(see example for all builds table)

=== Computed and derived column values ===

??? see ProjectsTable in tables.py

Prefer to add methods to models rather than compute values on the fly in the template. This is because these values will very often be useful in other contexts; if they are defined in the view in an ad hoc way, they can't easily be reused.

== Filter API ==

The classes for creating filters are defined in tablefilter.py.

A filter applies to a single column in the table. Any actions on a filter should filter records according to criteria which make sense with respect to that column. For example, you wouldn't add a filter to the "Completed on" column of the "All Builds" table which filters builds according to whether they failed or succeeded; but you could add a filter which filters the builds based on when they finished.

A filter adds an option to the column heading to filter in/out particular records or to remove filtering altogether.

Each filter has one or more actions associated with it. It also gets an action by default which turns off the filter and shows all the records again.

Each action changes the records shown in its associated ToasterTable, by applying a set of criteria to the query used to fetch records (e.g. show records for a single date, for a date range, or matching/not matching some other criteria related to the column).

How to add a filter to a field

=== Table filters ===

The TableFilter class acts as a container for a group of TableFilterAction objects.

TableFilterAction is the base class for an action associated with a filter. See the next section for the types of filter action already implemented.

=== Table filter actions ===

TableFilterActionToggle
TableFilterActionDay
TableFilterActionDateRange

ToasterTable

2016-04-04T15:07:33Z

Elliot Smith: Created page with "== Introduction to ToasterTable == '''This is a WIP''' ToasterTable is a set of classes in the toastergui module of the Toaster Django application. It is used to present a s..."

== Introduction to ToasterTable ==

'''This is a WIP'''

ToasterTable is a set of classes in the toastergui module of the Toaster Django application. It is used to present a set of database records in a style and with functionality consistent with the rest of Toaster.

The functionality provided by ToasterTable includes:
* Sorting
* Column hiding/showing
* Filtering
* Paging
* Search

When a ToasterTable is updated, a new set of records matching the filter criteria, search string, sort order and page number is fetched from the back-end using Ajax requests. The ToasterTable is then updated in place from the JSON in the response, using JavaScript to redraw the table rows.

As a developer, you don't need to worry (necessarily) about how this works: you just implement a subclass of ToasterTable, specify a template for rendering it, and supply a URL mapping.

The following sections explain how to use ToasterTable. The filenames in these sections refer to files in the toastergui/ directory.

Note that to be able to follow the example, you will need to set up Toaster for development work. This is described in [http://www.yoctoproject.org/docs/2.0.1/toaster-manual/toaster-manual.html the Toaster manual].

== How to create a ToasterTable ==

Most of the tables needed for Toaster are already implemented. Many of the key tables in Toaster already use ToasterTable, but some don't. A full list of unconverted tables is in https://bugzilla.yoctoproject.org/show_bug.cgi?id=8363.

This means that you will typically be porting an existing table to ToasterTable, rather than adding a new table from scratch. However, to keep things simple, this section explains how to create a ToasterTable from scratch. Knowing how to do this should make it easier to port an existing table to ToasterTable in future.

To provide a worked example, I'll create a new table which is a variant of the existing "all builds" table, using a reduced number of columns and filters.

=== Create the table class ===

A quick overview of where things are and where they go when creating a table:

* widgets.py contains the ToasterTable class. This is the base class for all ToasterTables.
* New tables are added to tables.py.
* A table in tables.py maps to a URL within the Toaster application; the mapping is defined in urls.py.
* Templates for the table are added to templates/.
* If you need [https://docs.djangoproject.com/en/1.9/howto/custom-template-tags/ custom template tags], these go in templatetags/.

To create a new table, first add the class definition to tables.py:

<nowiki>
# import any models you need for the table
from orm.models import Build

class MiniBuildsTable(ToasterTable):
def __init__(self, *args, **kwargs):
super(MiniBuildsTable, self).__init__(*args, **kwargs)
self.default_orderby = '-completed_on'
self.title = 'Mini Builds Table'

def setup_queryset(self, *args, **kwargs):
self.queryset = Build.objects.all()

def setup_columns(self, *args, **kwargs):
self.add_column(title='Completed on',
help_text='The date and time when the build finished',
hideable=False,
orderable=True,
static_data_name='completed_on',
static_data_template='{{data.completed_on | date:"d/m/y H:i"}}')
</nowiki>

The key to creating a ToasterTable is to decide which data is going to be shown in the table and define this in the ''setup_queryset()'' method. This will typically be a queryset derived from one model in the Toaster application. To see the available models, check the orm/models.py file.

Each row in the table corresponds to a record in the queryset. The ''setup_queryset()'' method specifies how to get this queryset for the table: it must set the self.queryset property to the Django queryset containing the data. In the MiniBuildsTable, we show all of the builds by default (using the standard Django model API).

Each column in the ToasterTable corresponds to a field in the queryset. Inside a table row, a cell corresponds to one or more fields from each record in the queryset. A cell can show various aspects of a record: a formatted version of a field's value, an amalgam of multiple fields, a computation based on a group of related records etc. For example, in a row of the MiniBuildsTable, a cell might contain the outcome of the build, the number of tasks which failed during the build, or the time since the build completed.

The ''setup_columns()'' method defines which columns should be shown in the table. In this initial version of MiniBuildsTable, only the "completed_on" column is shown. This column has the following properties:
* It cannot be hidden (hideable=False).
* It can be used to order the table (orderable=True).
* static_data_name sets the name of the column so that it can be married up with the sort order specified in the querystring, and with the default_orderby for the ToasterTable (see below).
* static_data_template specifies how to render the value from the record into the HTML for the page. The data.completed_on reference in the template demonstrates how to access

The self.default_orderby member set in __init__() specifies the default column to use for ordering the table. In this case, the completed_on field is used to sort the builds in the table; the "-" specifies reverse order, so the newest build appears at the top of the table. Note that the value for self.default_orderby should match the name of a field in the model, and the name of a column added to the table.

See the [adding more columns] section for full details of the column options available.

=== Adding the template ===

This should go into templates/, as this is where Django looks for view templates.

In all cases, you will need to include the toastertable.html template from your template. You will also need to add links for the jQuery UI CSS and JS files. (In an ideal world, this would be in toastertable.html, but for historical reasons, it isn't yet.)

You will also need to define an xhr_table_url variable in your template. This is used to request new data for the table when filters or search terms are applied, or if a new page is requested. ToasterTable will send a request for the table JSON to this URL, then automatically refresh the table display with the new data when it's received.

As an example, here's a minimal template for the Mini Builds page, which would be in templates/minibuilds.html.

<nowiki>
{% extends 'base.html' %}
{% load static %}

{% block extraheadcontent %}
<link rel="stylesheet" href="{% static 'css/jquery-ui.min.css' %}" type='text/css'>
<link rel="stylesheet" href="{% static 'css/jquery-ui.structure.min.css' %}" type='text/css'>
<link rel="stylesheet" href="{% static 'css/jquery-ui.theme.min.css' %}" type='text/css'>
<script src="{% static 'js/jquery-ui.min.js' %}"></script>
{% endblock %}

{% block title %}{{title}}{% endblock %}

{% block pagecontent %}
<h1 class="page-header top-air">{{title}}</h1>
{% url 'minibuilds' as xhr_table_url %}
{% include 'toastertable.html' %}
{% endblock %}
</nowiki>

Depending on which side menus or other wrapping you need around your table, you may need to modify an existing template (particularly if you are porting a table to ToasterTable). Note that this template extends the base.html template, which provides the header/footer for Toaster itself, but doesn't have a side menu. For an example of a more complex page, see builds-toastertable.html.

The important things to remember, though, are in the example template above; in particular, the need to import the jQuery UI assets, set xhr_table_url, and include the toastertable.html template.

=== Add a URL mapping for the table ===

To be able to view the Mini Builds page, it needs a mapping in the urls.py file:

<nowiki>
urlpatterns = patterns('toastergui.views',
// ... other mappings ...

url(r'^minibuilds/$',
tables.MiniBuildsTable.as_view(template_name='minibuilds.html'),
name='minibuilds')
)
</nowiki>

This is standard Django template mapping code. The only wrinkle is that because a ToasterTable is a Django TemplateView subclass, we call Django's as_view() method on our table to render it, passing the name of the template file.

Now, with Toaster running locally, you should be able to visit

http://localhost:8000/minibuilds/

in a browser and see your Mini Builds table (note that it will be empty unless you've run some builds).

== More ToasterTable ==

The following sections flesh out the options for adding columns to a ToasterTable, adding more data to the page, and using the table filter APIs.

=== Adding more columns ===

???

Column options

Adding a column to a table:
- Sorting by a column
- Hiding/showing a column
- Filtering: The Filter API is used to add filtering functionality to particular columns in the table.

=== Other context data ===

Data not in each record of the queryset, but which needs to be visible on the page. As an example, the project builds page shows all of the builds for a project; but the project name needs to be shown at the top of the page. In this case, the project is added to the extra context data for the ToasterTable and accessed in the template.

As an example, we could compute the total number of builds Toaster has recorded and show this in the title at the top of the Mini Builds page.

???

Note that if you want this value to update as the table is filtered, you'll need to do it in JavaScript. This is because the page for a ToasterTable isn't re-rendered when its data changes: only the HTML for the table is updated. Any additional updates required on the page have to be done similarly, via JavaScript.

For example, to update the title of the page as the table is filtered, we could add this code to the template:

???

(see example for all builds table)

=== Computed and derived column values ===

??? see ProjectsTable in tables.py

Prefer to add methods to models rather than compute values on the fly in the template. This is because these values will very often be useful in other contexts; if they are defined in the view in an ad hoc way, they can't easily be reused.

== Filter API ==

The classes for creating filters are defined in tablefilter.py.

A filter applies to a single column in the table. Any actions on a filter should filter records according to criteria which make sense with respect to that column. For example, you wouldn't add a filter to the "Completed on" column of the "All Builds" table which filters builds according to whether they failed or succeeded; but you could add a filter which filters the builds based on when they finished.

A filter adds an option to the column heading to filter in/out particular records or to remove filtering altogether.

Each filter has one or more actions associated with it. It also gets an action by default which turns off the filter and shows all the records again.

Each action changes the records shown in its associated ToasterTable, by applying a set of criteria to the query used to fetch records (e.g. show records for a single date, for a date range, or matching/not matching some other criteria related to the column).

How to add a filter to a field

=== Table filters ===

The TableFilter class acts as a container for a group of TableFilterAction objects.

TableFilterAction is the base class for an action associated with a filter. See the next section for the types of filter action already implemented.

=== Table filter actions ===

TableFilterActionToggle
TableFilterActionDay
TableFilterActionDateRange

Toaster

2016-04-04T15:07:15Z

Elliot Smith: /* Toaster How-to's */

[[Category:Toaster]]
[[Toaster]] is a web-based interface to OpenEmbedded and BitBake.
[[File:Screenshot toaster.png|thumb|Screenshot of Toaster 2.1]]
General discussion about '''Toaster''' happens on a dedicated mailing list: [https://lists.yoctoproject.org/listinfo/toaster https://lists.yoctoproject.org/listinfo/toaster]

=== Using Toaster ===

Toaster can run in various modes and setups.

* '''Local mode''' - in this mode Toaster is setup for use as a local development tool. It can be used to configure builds or just as a receiver for builds done on the command line with bitbake. You can launch it like this:

$ source oe-init-build-env
$ source toaster start

You then navigate to the link in your browser (e.g. http://localhost:8000) and configure a project. Or start building in the normal way with bitbake via the command line. Toaster will automatically pick up the builds and you will be able to see them on the build dashboard in your browser.

* '''Production mode''' - All the same functionality as the local mode but with the web server setup as a shared service for multiple developers to use, this sets up Toaster as a wsgi application and [[Setting up a production instance of Toaster|requires additional configuration]].

=== Toaster How-to's ===

Specific pages with Toaster how-tos are available below.

* [[Contribute to Toaster]]
* [[Testing Toaster]]
* [[Setting up a local instance of Toaster]]
* [[Setting up a production instance of Toaster]] - documentation for Interactive mode
* [https://www.yoctoproject.org/documentation/toaster-manual-18 How to use the Toaster web interface]
* [[How to delete information from the Toaster database]]
* [[How to support permission management in Build Mode for Toaster]]
* [[ToasterTable]]: Toaster's internal API for adding sortable, filterable, paged tables

=== About Toaster ===
[[File:Screenshot toaster analyis.png|thumb|Analysis of builds using Toaster]]

* [[File:Working_with_design.pdf]]
* [https://bugzilla.yoctoproject.org/buglist.cgi?list_id=213820&columnlist=status_whiteboard%2Cassigned_to%2Ctarget_milestone%2Cbug_status%2Cshort_desc%2Cbug_severity%2Cpriority&classification=Build%20System%20%26%20Metadata&query_based_on=Toaster-Opens&query_format=advanced&bug_status=NEW&bug_status=ACCEPTED&bug_status=IN%20PROGRESS%20DESIGN&bug_status=IN%20PROGRESS%20DESIGN%20COMPLETE&bug_status=IN%20PROGRESS%20IMPLEMENTATION&bug_status=REOPENED&bug_status=NEEDINFO&bug_status=WaitForUpstream&component=toaster&product=Toaster&known_name=Toaster-Opens Bug list]
* [[Toaster architecture design]]
* [[Toaster and bitbake communications]]
* [[Toaster testing plan]]

=== In progress documentation ===

We are currently preparing the documentation for the Toaster build functionality. The content here is just a brain dump of what we need to cover (in no particular order). Feel free to add and create content as you see fit:

*[[Using virtualenv]]
*[[Setting up a production instance of Toaster]]
*[[manage.py commands]] - this should include an explanation of lsupdates
*[[Start Toaster in managed mode]]

Toaster and bitbake communications

2015-11-19T14:55:48Z

Elliot Smith: /* How do toasterui + buildinfohelper manage events? */

This is a write up of some investigation I did into how toaster asks bitbake to do stuff, and how it hears about what bitbake is doing. It's not definitive and not guaranteed to be correct, but it might provide some pointers if you are working on toaster. If anyone knows better, please feel free to correct this.

Any paths given below are relative to the root of the poky/poky-contrib source tree.

== toaster asking bitbake to do stuff ==

First off I wanted to figure out how clicking on the "Build" button in the toaster interface triggers a bitbake build.

toaster only asks bitbake to perform builds when in managed mode. This is the point I started from when doing this investigation. I think analysis mode is very similar, except toaster doesn't ask bitbake to do anything, it just listens to stuff which is already happening.

=== runbuilds ===

The conversation with bitbake is handled via the runbuilds command (bitbake/lib/toaster/bldcontrol/management/commands/runbuilds.py). This command is run in a loop once every second from the toaster start script (bitbake/bin/toaster).

The runbuilds.schedule() function looks for BuildRequests in the database. A '''BuildRequest''' is created each time you press a "Build" button in the toaster web interface. Those build requests are created in orm/model.py schedule_build on the project object. When creating this BuildRequest also created are BRLayer for each of the layers in your project, BRTarget for the target to be built and BRBitbake for the version of bitbake to use, these are all defined in bldcontrol/models.py, and contain copies of the information from the project.

The localhostbecontroller takes this BuildRequest

* schedule() gets a localhostbecontroller (be = build environment) instance (there is a becontroller for remote bitbakes, but I don't think the implementation is complete) and assigns it to a variable bec.

* schedule() then calls bec.triggerBuild() for a build request which is in the BuildRequest.REQ_QUEUED state. Only one build request gets picked up each time the runbuilds script runs: the one with the largest ID.

* schedule() also creates a build identification variable for each build request, combining the primary key of the build request with the primary key of the build environment controller's build environment (!)).

=== localhostbecontroller ===

This is the build environment controller, which handles instances of build environments. It is also responsible for cloning any layers that are in the build request.

It is a subclass of BuildEnvironmentController.

* triggerBuild() calls getBBController(), which returns a BitBakeController instance (bbctrl). getBBController() actually instantiates the controller if it isn't already instantiated, passing it a "server" object. This server object is an instance of bb.server.xmlrpc.BitBakeXMLRPCClient().

* triggerBuild() then calls the BitBakeController build() method: bbctrl.build()

=== BitBakeController (bbctrl) ===

This is constructed with the server object, which is an XMLRPC connection to a BitBake server.

* build() invokes the "buildTargets" command on the connection using runCommand().

=== BitBakeXMLRPCClient (bitbake/lib/bb/server/xmlrpc.py) ===

This is a subclass of BitBakeServer.

It has a "connection" member, which has runCommand() invoked on it.

When constructed by getBBController(), initServer() is called as part of its construction. This is a method from BitBakeServer which sets the interface address (default: (localhost,0)) and creates an XMLRPCServer() using this interface address.

Once initServer() is done, establishConnection() is invoked, which creates the "real" socket.

The established connection (which runCommand() is invoked on) is a BitBakeXMLRPCServerConnection.

=== BitBakeServer ===

BitbakeServer is a subclass of BitBakeBaseServer (declared in bitbake/lib/bb/server/__init__.py); this actually has the addcooker() method which associates a bitbake cooker instance with the connection.

BitBakeBaseServer acts like a decorator around a server implementation; in our case, it's a BitBakeXMLRPCServerConnection.

Calling addcooker() on the BitBakeBaseServer also calls it on any server implementation (serverImpl) which is wrapped by BitBakeServer.

=== BitBakeXMLRPCServerConnection ===

This is a subclass of BitBakeBaseServerConnection.

When runCommand() is invoked on this, it's passed on to the cooker object associated with it, i.e. it actually invokes self.cooker.command.runCommand().

The cooker object is associated with the BitBakeXMLRPCServerConnection when the bitbake/lib/bb/main.py script runs.

=== main.py ===

The main script which starts a bitbake server and ui. This is what runs when you use "bitbake" from the command line.

toaster starts the bitbake server with the --server-only switch, which calls bitbake_main() in this file; this in turn calls the main() function.

This instantiates a bb.cooker.BBCooker and adds it to the server implementation via addcooker(). The cooker is what actually enables commands to be sent to the bitbake server.

=== BBCooker (bitbake/lib/bb/cooker.py) ===

This has a "command" property which is an instance of BBCommand; this is what runCommand() finally gets invoked on.

=== BBCommand ===

runCommand() calls a method from an instance of CommandSync (all the synchronous commands bitbake understands) or CommandAsync (the asynchronous ones), depending on the type of command passed to runCommand(). Both the CommandSync and CommandAsync instances are added to the BBCooker when it is created.

For example, runCommand("getVariable", ...) is invoked via CommandsSync.getVariable().

The commands which go through BBCommand.runCommand() now make their way to the bitbake server over XMLRPC.

== toaster listening to what bitbake is doing ==

At this point, I realised I probably understood enough to see how bitbake was being invoked from toaster: asking toaster to start a build sends a "buildTargets" command to the bitbake XMLRPC server, via a rather indirect series of objects and method calls.

What I wanted to know now was how toaster listens to the result of that command. At a high level, I understood that it gathered events from the XMLRPC connection and converted them into database objects. However, I didn't really get the code path.

I started from BuildInfoHelper, which is where bitbake events are converted into toaster db objects.

=== BuildInfoHelper ===

This is passed build events from the toasterui.py which routes them off to the buildinfohelper based on the event types.

BuildInfoHelper is responsible for constructing toaster ORM objects from events. The BuildInfoHelper is constructed with a server (instance of BitBakeXMLRPCServerConnection in the case of toaster), so it can also interrogate the bitbake server for extra environmental data via getVariable().

BuildInfoHelper also tries to match the data from the build recipe events from toasterui.py to data in toaster's database, so that toaster can "learn" from the build, this means that we get recipe and package information from the build which otherwise we are unable to determine.

The task of matching the recipes and associated data is done when we get the recipe information from the store task event, this contains a path of the recipe e.g. "/home/yocto/cloned_layers/_mylayer_master.toaster_cloned/mydir/example.bb... the buildinfohelper then tries to find a layer in toaster that has a checkout directory location that starts with the one provided by the event.

It does this by asking the "bc" BuildController (which can only be localhostbecontroller.py at the moment as this is the only controller which contains the implementation) to return the git checkout path via 'getGitCloneDirectory' this returns what it believes was the git checkout base location, in our example hopefully something like "/home/yocto/cloned_layers/_mylayer_master.toaster_cloned" then the buildinfohelper adds the directory name (brl.dirpath) from the information in the build request layer, e.g. "mydir" after all this reconstruction we hope that the recipe path we got from the event starts with the reconstructed path. e.g. Does "/home/yocto/cloned_layers/_mylayer_master.toaster_cloned/mydir/example.bb" start with "/home/yocto/cloned_layers/_mylayer_master.toaster_cloned/mydir/" if yes then we have a layer in toaster to which we can associate the information from the build to.

One problem here is that the getGitCloneDirectory function a) doesn't exist in the other hostcontrollers and b) doesn't always return the correct value due to some special case

=== toasterui.py ===

This has a main() function which sets up an event listener loop.

main() is passed a server, eventHandler and params. The eventHandler is the object which listens to bitbake events.

toasterui is called from main.py (see below).

=== main.py (bitbake/lib/bb/main.py) ===

The toasterui.py main() method is invoked from main.py.

It's passed two parameters:

# server_connection.connection
# server_connection.events

The second of these is the eventHandler which is set up in a loop in toasterui.py.

server_connection comes from a call to establishConnection() in main.py. The server is constructed via a servermodule, which is dynamically chosen in main.py according to the parameters used to invoke it (-t). It will either be "process" or "xmlrpc".

The UI type is set in the main.py script via the -u option.

toaster invokes bitbake with: -t xmlrpc -u toasterui

which means that we get the toasterui.main() called, and we get an xmlrpc bitbake server.

Back to server_connection.events...

This refers to an xmlrpc bitbake server connection's events object.

So toasterui.main() gets:

# server_connection.connection => BitBakeXMLRPCServerConnection.connection
# server_connection.events => BitBakeXMLRPCServerConnection.events

The events object is an instance of uievent.BBUIEventQueue in our case, as we're using toasterui.

=== uievent.BBUIEventQueue (bitbake/lib/bb/ui/uievent.py) ===

This is our event handler ("events") in toasterui.main(), which is receiving events on the XMLRPC connection (see later).

toasterui.main() calls events.waitEvent(0.25), which looks for events on the queue every 0.25s. If the queue has events, one is popped off.

Each event popped from the queue is passed to uihelper.BBUIHelper.eventHandler(), which adds build tracking information (how many packages built, tasks completed etc.)

The event then goes to the BuildInfoHelper, where it gets stored in toaster's database.

==== How do events get on BBUIEventQueue? ====

BBUIEventQueue is instantiated on the BitBakeXMLRPCServerConnection object.

It is passed a BBServer, which is an xmlrpclib.ServerProxy (from the Python standard library).

The ServerProxy has a method corresponding to each method on the server it is proxying for; in this case, the proxied server is a BBServer, so the proxy has the same methods as BBServer.

Event handling is set up by calling self.BBServer.registerEventHandler() from BBUIEventQueue.

registerEventHandler() is defined in bitbake/lib/bb/server/xmlrpc.py, BitBakeServerCommands. This in turn calls bb.event.register_UIHandler, defined in bitbake/lib/bb/event.py.

When an event is fired, each handler registered for it is invoked with that event (in event.py).

The main function for firing events in event.py is fire_from_worker(), which is called from bitbake/lib/bb/runqueue.py.

Events are constructed from xmlrpc messages coming from the bitbake server (see runqueue.py, runQueuePipe.read()).

==== How do toasterui + buildinfohelper manage events? ====

toasterui runs as part of the bitbake instance: bitbake is invoked with a -u toasterui option, which means that toasterui is instantiated as the event handler for the bitbake instance.

As events occur in bitbake, they are passed to toasterui; toasterui then hands those events off to buildinfohelper, which uses the event data to create records in the Toaster database. Note that buildinfohelper has to start up the Django database machinery manually for this to be possible, and that this is happening outside the main Django instance, in a separate process. This might be part of the reason why we get database locking issues with Django 1.8.

As buildinfohelper receives bitbake events, it sets variables in its internal_state dictionary. These variables are used to represent events which are "partial" from the perspective of Toaster: that is, events which can't create a complete, useful record in Toaster's database.

The best example of this comes from bitbake's Task* events. We receive two events for a task in buildinfohelper:

# The event notifying that the task started, e.g. TaskStarted, runQueueTaskStarted, sceneQueueTaskStarted
# The end state of the task, e.g. TaskFailedSilent, TaskCompleted, runQueueTaskFailed, sceneQueueTaskFailed, runQueueTaskCompleted, sceneQueueTaskCompleted, runQueueTaskSkipped

Because Toaster represents a Task as a single entity, with an outcome state, we can't add a complete Task record when a task starts: we add a partial Task record, then update that when the task end event is received.

To get this to work, buildinfohelper saves the partial Task to the database when the task start event is received; then retrieves and updates that record when the task end event arrives. In between these two points, buildinfohelper keeps some internal state about tasks which have started but which don't have a "done" outcome. However, the identifiers used in this internal state are composed of the task file + name, which is a fairly arbitrary algorithm (I quote: "we do a bit of guessing"). When the task end event is received, it is matched up to the internal state (where we have a list of tasks which haven't ended) using this fairly arbitrary identifier. This seems like it might be prone to error.

The reason this approach has been used, though, is because (as far as I can tell) bitbake doesn't provide any identifiers on its events which would enable them to be tied together. A TaskStarted event doesn't specify which task started (tasks don't have unique IDs), just the name of that task and its .bb file; there is also nothing to tie a task to the build it's associated with. This means that any event handler waiting for bitbake events has to manually tie together events and maintain local state to be able to do that.

The following internal state is maintained in buildinfohelper for this purpose:

* lvs (layer versions): layer versions known to Toaster
* recipes: recipes known to Toaster
* backlog: list of log events which haven't been saved for the current build yet
* brbe: the build request and the build environment primary keys, concatenated together around a colon (e.g. "1:2")
* build: the ongoing build (there's an implicit assumption that bitbake can only run one build at a time)
* taskdata: a list of ongoing tasks (i.e. task events for which we have received a task started event, but haven't yet received a task ended event); as task end events are received, the task goes into the database and is removed from taskdata
* targets: targets for the current build
* task_order: a counter which is added to each task as it is saved to the database, which marks the order in which the task events were received; it gets incremented each time a certain type of task event is received

Note that lvs and recipes are set before the build started event, using metadata events fired by bitbake.

While the build is ongoing, the internal state affects how data is added to the database: it is used to implicitly associate events with the build which is assumed to occur between a BuildStarted event and the next BuildCompleted/BuildFailed event.

Here's an example of the workflow to make it clearer:

# toasterui receives a BuildStarted event and passes it to buildinfohelper.
# buildinfohelper creates a database record B for the new build, and stores it as a property on itself.
# toasterui receives a TaskStarted event with name "foo" and file "/bar/bar/humbug"; it passes the event to buildinfohelper.
# buildinfohelper stores the partial task in the database as T, associating it with the build record B; it also stores T under the key "/bar/bar/humbug:foo" in taskdata.
# toasterui receives a TaskCompleted event with name "foo" and file "/bar/bar/humbug"; it passes the event to buildinfohelper.
# buildinfohelper marries up the new TaskCompleted event with the partial record T, by matching the new event's key "/bar/bar/humbug:foo" with the key for the TaskStarted event which is already in taskdata (see 4); T is updated in the database.
# toasterui receives a BuildCompleted event and passes it to buildinfohelper.
# buildinfohelper assumes that the BuildCompleted event applies to the build B already stored in its internal state. It updates the database record for B with data about when the build ended, build artifacts etc.
# toasterui creates a new buildinfohelper, ready to deal with the next build.

backlog is a list of log events which haven't yet been attached to a build; if a log event occurs before the build has been saved, it is added to the list; then, once a BuildStarted event has occurred, the list of events in the backlog is added to the database and associated with that build.

== Conclusion ==

At this point, I felt I understood enough about how events are processed for my purpose, so I didn't dig any further. I knew where I could amend an event to add/remove properties on it, which is what I was after.

Toaster and bitbake communications

2015-11-19T14:55:09Z

Elliot Smith: /* How do events get on BBUIEventQueue? */

This is a write up of some investigation I did into how toaster asks bitbake to do stuff, and how it hears about what bitbake is doing. It's not definitive and not guaranteed to be correct, but it might provide some pointers if you are working on toaster. If anyone knows better, please feel free to correct this.

Any paths given below are relative to the root of the poky/poky-contrib source tree.

== toaster asking bitbake to do stuff ==

First off I wanted to figure out how clicking on the "Build" button in the toaster interface triggers a bitbake build.

toaster only asks bitbake to perform builds when in managed mode. This is the point I started from when doing this investigation. I think analysis mode is very similar, except toaster doesn't ask bitbake to do anything, it just listens to stuff which is already happening.

=== runbuilds ===

The conversation with bitbake is handled via the runbuilds command (bitbake/lib/toaster/bldcontrol/management/commands/runbuilds.py). This command is run in a loop once every second from the toaster start script (bitbake/bin/toaster).

The runbuilds.schedule() function looks for BuildRequests in the database. A '''BuildRequest''' is created each time you press a "Build" button in the toaster web interface. Those build requests are created in orm/model.py schedule_build on the project object. When creating this BuildRequest also created are BRLayer for each of the layers in your project, BRTarget for the target to be built and BRBitbake for the version of bitbake to use, these are all defined in bldcontrol/models.py, and contain copies of the information from the project.

The localhostbecontroller takes this BuildRequest

* schedule() gets a localhostbecontroller (be = build environment) instance (there is a becontroller for remote bitbakes, but I don't think the implementation is complete) and assigns it to a variable bec.

* schedule() then calls bec.triggerBuild() for a build request which is in the BuildRequest.REQ_QUEUED state. Only one build request gets picked up each time the runbuilds script runs: the one with the largest ID.

* schedule() also creates a build identification variable for each build request, combining the primary key of the build request with the primary key of the build environment controller's build environment (!)).

=== localhostbecontroller ===

This is the build environment controller, which handles instances of build environments. It is also responsible for cloning any layers that are in the build request.

It is a subclass of BuildEnvironmentController.

* triggerBuild() calls getBBController(), which returns a BitBakeController instance (bbctrl). getBBController() actually instantiates the controller if it isn't already instantiated, passing it a "server" object. This server object is an instance of bb.server.xmlrpc.BitBakeXMLRPCClient().

* triggerBuild() then calls the BitBakeController build() method: bbctrl.build()

=== BitBakeController (bbctrl) ===

This is constructed with the server object, which is an XMLRPC connection to a BitBake server.

* build() invokes the "buildTargets" command on the connection using runCommand().

=== BitBakeXMLRPCClient (bitbake/lib/bb/server/xmlrpc.py) ===

This is a subclass of BitBakeServer.

It has a "connection" member, which has runCommand() invoked on it.

When constructed by getBBController(), initServer() is called as part of its construction. This is a method from BitBakeServer which sets the interface address (default: (localhost,0)) and creates an XMLRPCServer() using this interface address.

Once initServer() is done, establishConnection() is invoked, which creates the "real" socket.

The established connection (which runCommand() is invoked on) is a BitBakeXMLRPCServerConnection.

=== BitBakeServer ===

BitbakeServer is a subclass of BitBakeBaseServer (declared in bitbake/lib/bb/server/__init__.py); this actually has the addcooker() method which associates a bitbake cooker instance with the connection.

BitBakeBaseServer acts like a decorator around a server implementation; in our case, it's a BitBakeXMLRPCServerConnection.

Calling addcooker() on the BitBakeBaseServer also calls it on any server implementation (serverImpl) which is wrapped by BitBakeServer.

=== BitBakeXMLRPCServerConnection ===

This is a subclass of BitBakeBaseServerConnection.

When runCommand() is invoked on this, it's passed on to the cooker object associated with it, i.e. it actually invokes self.cooker.command.runCommand().

The cooker object is associated with the BitBakeXMLRPCServerConnection when the bitbake/lib/bb/main.py script runs.

=== main.py ===

The main script which starts a bitbake server and ui. This is what runs when you use "bitbake" from the command line.

toaster starts the bitbake server with the --server-only switch, which calls bitbake_main() in this file; this in turn calls the main() function.

This instantiates a bb.cooker.BBCooker and adds it to the server implementation via addcooker(). The cooker is what actually enables commands to be sent to the bitbake server.

=== BBCooker (bitbake/lib/bb/cooker.py) ===

This has a "command" property which is an instance of BBCommand; this is what runCommand() finally gets invoked on.

=== BBCommand ===

runCommand() calls a method from an instance of CommandSync (all the synchronous commands bitbake understands) or CommandAsync (the asynchronous ones), depending on the type of command passed to runCommand(). Both the CommandSync and CommandAsync instances are added to the BBCooker when it is created.

For example, runCommand("getVariable", ...) is invoked via CommandsSync.getVariable().

The commands which go through BBCommand.runCommand() now make their way to the bitbake server over XMLRPC.

== toaster listening to what bitbake is doing ==

At this point, I realised I probably understood enough to see how bitbake was being invoked from toaster: asking toaster to start a build sends a "buildTargets" command to the bitbake XMLRPC server, via a rather indirect series of objects and method calls.

What I wanted to know now was how toaster listens to the result of that command. At a high level, I understood that it gathered events from the XMLRPC connection and converted them into database objects. However, I didn't really get the code path.

I started from BuildInfoHelper, which is where bitbake events are converted into toaster db objects.

=== BuildInfoHelper ===

This is passed build events from the toasterui.py which routes them off to the buildinfohelper based on the event types.

BuildInfoHelper is responsible for constructing toaster ORM objects from events. The BuildInfoHelper is constructed with a server (instance of BitBakeXMLRPCServerConnection in the case of toaster), so it can also interrogate the bitbake server for extra environmental data via getVariable().

BuildInfoHelper also tries to match the data from the build recipe events from toasterui.py to data in toaster's database, so that toaster can "learn" from the build, this means that we get recipe and package information from the build which otherwise we are unable to determine.

The task of matching the recipes and associated data is done when we get the recipe information from the store task event, this contains a path of the recipe e.g. "/home/yocto/cloned_layers/_mylayer_master.toaster_cloned/mydir/example.bb... the buildinfohelper then tries to find a layer in toaster that has a checkout directory location that starts with the one provided by the event.

It does this by asking the "bc" BuildController (which can only be localhostbecontroller.py at the moment as this is the only controller which contains the implementation) to return the git checkout path via 'getGitCloneDirectory' this returns what it believes was the git checkout base location, in our example hopefully something like "/home/yocto/cloned_layers/_mylayer_master.toaster_cloned" then the buildinfohelper adds the directory name (brl.dirpath) from the information in the build request layer, e.g. "mydir" after all this reconstruction we hope that the recipe path we got from the event starts with the reconstructed path. e.g. Does "/home/yocto/cloned_layers/_mylayer_master.toaster_cloned/mydir/example.bb" start with "/home/yocto/cloned_layers/_mylayer_master.toaster_cloned/mydir/" if yes then we have a layer in toaster to which we can associate the information from the build to.

One problem here is that the getGitCloneDirectory function a) doesn't exist in the other hostcontrollers and b) doesn't always return the correct value due to some special case

=== toasterui.py ===

This has a main() function which sets up an event listener loop.

main() is passed a server, eventHandler and params. The eventHandler is the object which listens to bitbake events.

toasterui is called from main.py (see below).

=== main.py (bitbake/lib/bb/main.py) ===

The toasterui.py main() method is invoked from main.py.

It's passed two parameters:

# server_connection.connection
# server_connection.events

The second of these is the eventHandler which is set up in a loop in toasterui.py.

server_connection comes from a call to establishConnection() in main.py. The server is constructed via a servermodule, which is dynamically chosen in main.py according to the parameters used to invoke it (-t). It will either be "process" or "xmlrpc".

The UI type is set in the main.py script via the -u option.

toaster invokes bitbake with: -t xmlrpc -u toasterui

which means that we get the toasterui.main() called, and we get an xmlrpc bitbake server.

Back to server_connection.events...

This refers to an xmlrpc bitbake server connection's events object.

So toasterui.main() gets:

# server_connection.connection => BitBakeXMLRPCServerConnection.connection
# server_connection.events => BitBakeXMLRPCServerConnection.events

The events object is an instance of uievent.BBUIEventQueue in our case, as we're using toasterui.

=== uievent.BBUIEventQueue (bitbake/lib/bb/ui/uievent.py) ===

This is our event handler ("events") in toasterui.main(), which is receiving events on the XMLRPC connection (see later).

toasterui.main() calls events.waitEvent(0.25), which looks for events on the queue every 0.25s. If the queue has events, one is popped off.

Each event popped from the queue is passed to uihelper.BBUIHelper.eventHandler(), which adds build tracking information (how many packages built, tasks completed etc.)

The event then goes to the BuildInfoHelper, where it gets stored in toaster's database.

==== How do events get on BBUIEventQueue? ====

BBUIEventQueue is instantiated on the BitBakeXMLRPCServerConnection object.

It is passed a BBServer, which is an xmlrpclib.ServerProxy (from the Python standard library).

The ServerProxy has a method corresponding to each method on the server it is proxying for; in this case, the proxied server is a BBServer, so the proxy has the same methods as BBServer.

Event handling is set up by calling self.BBServer.registerEventHandler() from BBUIEventQueue.

registerEventHandler() is defined in bitbake/lib/bb/server/xmlrpc.py, BitBakeServerCommands. This in turn calls bb.event.register_UIHandler, defined in bitbake/lib/bb/event.py.

When an event is fired, each handler registered for it is invoked with that event (in event.py).

The main function for firing events in event.py is fire_from_worker(), which is called from bitbake/lib/bb/runqueue.py.

Events are constructed from xmlrpc messages coming from the bitbake server (see runqueue.py, runQueuePipe.read()).

==== How do toasterui + buildinfohelper manage events? ====

toasterui runs as part of the bitbake instance: bitbake is invoked with a -u toasterui option, which means that toasterui is instantiated as the event handler for the bitbake instance.

As events occur in bitbake, they are passed to toasterui; toasterui then hands those events off to buildinfohelper, which uses the event data to create records in the Toaster database. Note that buildinfohelper has to start up the Django database machinery manually for this to be possible, and that this is happening outside the main Django instance, in a separate process. This might be part of the reason why we get database locking issues with Django 1.8.

As buildinfohelper receives bitbake events, it sets variables in its internal_state dictionary. These variables are used to represent events which are "partial" from the perspective of Toaster: that is, events which can't create a complete, useful record in Toaster's database.

The best example of this comes from bitbake's Task* events. We receive two events for a task in buildinfohelper:

1. The event notifying that the task started, e.g. TaskStarted, runQueueTaskStarted, sceneQueueTaskStarted
2. The end state of the task, e.g. TaskFailedSilent, TaskCompleted, runQueueTaskFailed, sceneQueueTaskFailed, runQueueTaskCompleted, sceneQueueTaskCompleted, runQueueTaskSkipped

Because Toaster represents a Task as a single entity, with an outcome state, we can't add a complete Task record when a task starts: we add a partial Task record, then update that when the task end event is received.

To get this to work, buildinfohelper saves the partial Task to the database when the task start event is received; then retrieves and updates that record when the task end event arrives. In between these two points, buildinfohelper keeps some internal state about tasks which have started but which don't have a "done" outcome. However, the identifiers used in this internal state are composed of the task file + name, which is a fairly arbitrary algorithm (I quote: "we do a bit of guessing"). When the task end event is received, it is matched up to the internal state (where we have a list of tasks which haven't ended) using this fairly arbitrary identifier. This seems like it might be prone to error.

The reason this approach has been used, though, is because (as far as I can tell) bitbake doesn't provide any identifiers on its events which would enable them to be tied together. A TaskStarted event doesn't specify which task started (tasks don't have unique IDs), just the name of that task and its .bb file; there is also nothing to tie a task to the build it's associated with. This means that any event handler waiting for bitbake events has to manually tie together events and maintain local state to be able to do that.

The following internal state is maintained in buildinfohelper for this purpose:

* lvs (layer versions): layer versions known to Toaster
* recipes: recipes known to Toaster
* backlog: list of log events which haven't been saved for the current build yet
* brbe: the build request and the build environment primary keys, concatenated together around a colon (e.g. "1:2")
* build: the ongoing build (there's an implicit assumption that bitbake can only run one build at a time)
* taskdata: a list of ongoing tasks (i.e. task events for which we have received a task started event, but haven't yet received a task ended event); as task end events are received, the task goes into the database and is removed from taskdata
* targets: targets for the current build
* task_order: a counter which is added to each task as it is saved to the database, which marks the order in which the task events were received; it gets incremented each time a certain type of task event is received

Note that lvs and recipes are set before the build started event, using metadata events fired by bitbake.

While the build is ongoing, the internal state affects how data is added to the database: it is used to implicitly associate events with the build which is assumed to occur between a BuildStarted event and the next BuildCompleted/BuildFailed event.

Here's an example of the workflow to make it clearer:

1. toasterui receives a BuildStarted event and passes it to buildinfohelper.
2. buildinfohelper creates a database record B for the new build, and stores it as a property on itself.
3. toasterui receives a TaskStarted event with name "foo" and file "/bar/bar/humbug"; it passes the event to buildinfohelper.
4. buildinfohelper stores the partial task in the database as T, associating it with the build record B; it also stores T under the key "/bar/bar/humbug:foo" in taskdata.
5. toasterui receives a TaskCompleted event with name "foo" and file "/bar/bar/humbug"; it passes the event to buildinfohelper.
6. buildinfohelper marries up the new TaskCompleted event with the partial record T, by matching the new event's key "/bar/bar/humbug:foo" with the key for the TaskStarted event which is already in taskdata (see 4); T is updated in the database.
7. toasterui receives a BuildCompleted event and passes it to buildinfohelper.
8. buildinfohelper assumes that the BuildCompleted event applies to the build B already stored in its internal state. It updates the database record for B with data about when the build ended, build artifacts etc.
9. toasterui creates a new buildinfohelper, ready to deal with the next build.

backlog is a list of log events which haven't yet been attached to a build; if a log event occurs before the build has been saved, it is added to the list; then, once a BuildStarted event has occurred, the list of events in the backlog is added to the database and associated with that build.

== Conclusion ==

At this point, I felt I understood enough about how events are processed for my purpose, so I didn't dig any further. I knew where I could amend an event to add/remove properties on it, which is what I was after.

Setting up a production instance of Toaster

2015-11-09T11:46:24Z

Elliot Smith: /* Installation */

[[Category:Toaster]]

A production instance of Toaster is one in which you wish to share the Toaster instance with remote and multiple users. It is also the setup which can cope with heavier loads on the web service. These instructions setup toaster in Build mode where builds and projects are run, viewed and defined by the Toaster web interface.

== Requirements ==
* [http://www.yoctoproject.org/docs/2.0/yocto-project-qs/yocto-project-qs.html#packages Build requirements]
* Apache webserver
* mod-wsgi for Apache webserver
* Mysql database server

Ubuntu 14.04.3:
<code>
$ sudo apt-get install apache2 libapache2-mod-wsgi mysql-server virtualenv libmysqlclient-dev
</code>

Fedora 22/RH:
<code>
$ sudo dnf install httpd mod_wsgi python-virtualenv gcc mysql-devel
</code>

== Installation ==

'''1.''' Checkout a copy of Poky into the web server directory. We're going to be using /var/www/toaster.
<code>
$ mkdir -p /var/www/toaster
$ cd /var/www/toaster/
$ git clone git://git.yoctoproject.org/poky
$ git checkout jethro # change for any release name required
</code>

'''2.''' Initialise a virtualenv and install Toaster dependencies. (Use virtualenv to keep the python packages isolated from your system provided packages - not required but recommended, alternative use your OS's package manager to install the packages)

<code>
$ cd /var/www/toaster/
$ virtualenv venv
$ source ./venv/bin/activate
$ pip install -r ./poky/bitbake/toaster-requirements.txt
$ pip install mysql
$ pip install MySQL-python
</code>

'''3.''' Configure toaster edit /var/www/toaster/poky/bitbake/lib/toaster/toastermain/settings.py

Edit the DATABASE settings:
<code>
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.mysql',
'NAME': 'toaster_data',
'USER': 'toaster',
'PASSWORD': 'yourpasswordhere',
'HOST': 'localhost',
'PORT': '3306',
}
</code>

Edit the SECRET_KEY:
<code>
SECRET_KEY = 'YOUR SECRET RANDOM KEY HERE'
</code>

Edit the STATIC_ROOT:
<code>
STATIC_ROOT = '/var/www/toaster/static_files/'
</code>

'''4.'''' Now add the database and user to your mysql server that we just defined
<code>
$ mysql -u root -p
mysql> CREATE DATABASE toaster_data;
mysql> CREATE USER 'toaster'@'localhost' identified by 'yourpasswordhere';
mysql> GRANT all on toaster_data.* to 'toaster'@'localhost';
mysql> quit
</code>
n.b. You may want to decide on fewer [https://dev.mysql.com/doc/refman/5.1/en/grant.html privileges] to the toaster user.

'''5.''' Get toaster to create the database schema, default data and collect up the statically service files

<code>
$ cd /var/www/toaster/poky/
$ ./bitbake/lib/toaster/manage.py syncdb --migrate
$ ./bitbake/lib/toaster/manage.py loadconf ./meta-yocto/conf/toasterconf.json
$ ./bitbake/lib/toaster/manage.py lsupdates
$ ./bitbake/lib/toaster/manage.py collectstatic
</code>

'''6.''' Add a config file for Toaster to your Apache web server's configurations available directory.

Ubuntu/Debian put it here: /etc/apache2/conf-available/toaster.conf
Fedora/RH usually here: /etc/httpd/conf.d/toaster.conf
<code>

Alias /static /var/www/toaster/static_files
<Directory /var/www/toaster/static_files>
Order allow,deny
Allow from all
Require all granted
</Directory>
WSGIDaemonProcess toaster_wsgi python-path=/var/www/toaster/poky/bitbake/lib/toaster:/var/www/toaster/venv/lib/python2.7/site-packages
WSGIScriptAlias / "/var/www/toaster/poky/bitbake/lib/toaster/toastermain/wsgi.py"
<Location />
WSGIProcessGroup toaster_wsgi
</Location>

</code>

In Ubuntu/Debain you will need to enable the config and module in Apache webserver
<code>
$ sudo a2enmod wsgi
$ sudo a2enconf toaster
</code>

Restart Apache web server to make sure all new configuration is loaded

Ubuntu/Debian:
<code>
$ sudo service apache2 restart
</code>

Fedora/RH:
<code>
$ sudo service httpd restart
</code>

'''7.''' Install the build runner service

This service needs to be running in order to dispatch builds the command that needs to be run is:

<code>
/var/www/toaster/poky/bitbake/lib/toaster/manage.py runbuilds
</code>

Sample script:
<code>
#!/bin/sh
# toaster run builds dispatcher
cd /var/www/toaster/
source ./venv/bin/activate
./bitbake/lib/toaster/manage.py runbuilds
</code>

N.b. You may wish to add a service entry to your OS's init system so that it starts up on start up as well as adding a dedicated user.

Now open up a browser and you can start using Toaster!

Toaster future release planning

2015-10-21T14:55:22Z

Elliot Smith: /* Elliot's */

[[Category:Toaster]]

== 2.1 Planning ==

=== Wish lists ===

==== Michael's ====
* Finish image-customisation 8070, 8081, 8082, 8103, 8104, 8117, 8128, 8132, 8091
* Remove pseudo API cruft / template context to json response stuff
* Consolidate used API into it's own view
* Finish toaster tables porting
* Get selenium test wrapper
* Create a toasterclient.py to interact with toaster to avoid need for 'command line builds'
** This would use toaster's API as the entry point to using bitbake
** It could be called by CI systems
** It could be used by auto builders etc
** Means that there are no two modes anywhere in Toaster
** Allows project configuration to be changed in toaster from the command line

==== Brian's ====
This is *not* ordered! This is more of a brain dump :)
* Collapse analysis and managed mode
* Working CI for each commit into toaster-next as well as for project peoples testME branches on poky-contrib:
** run django tests
** run selenium tests
** leverage sstate to make runs generally shorter
* remove git assumptions from code
* rationalize/simplify the configuration scripts
* if bitbake-memres comes out, interact with it gracefully (this is my preferred method of collapsing analysis and managed)
* fix the tmpdir/release build issue &/or discuss the need to support building old releases with a UI rather than limiting toaster to build the release it is part of. This is inherently fragile.
* ask bb to parse any layer added to a project and such as imported layers, and layers specified in the configuration file to give us recipe information.
* improve our build event following:
** note parse events
** update/retrieve ui so that we can tell something is happening before the first build event comes
** eliminate the remaining random tracebacks from try/except failures.
* allow configuration in addition to the addition/removal of layers/packages
** kernel config
* add user control/authentication
* update to Django 1.8. Always stay on Django LTS versions
* add layer update from layers.openembedded.org to the UI
* add Django tests for building e.g. we should be able to build core-image-sato from a django test and validate the db
* add mysql script to simplify setup with mysql/apache
* pull in new look and feel
* add django tests to validate all artifacts - if we say they can download it and it appears on a page anywhere, it should have a test
* improve docs of code on wiki. We have a nice start (ty E & M); I'd like to improve it.
** events - more info and list types supported
** what logic is in jscript , what is in python
** database schema definition and description esp interrelationships.
** list django tests, mostly so we can see what we are missing.
* build mode triggerred off of database entry, not autonomous process.
* add asynchronicity. I have noted a number of places where commands result in a sluggish ui. I'm nervous about this one.
** for example, starting a build that requires us to clone layers has a noticeable delay as compared to starting a local only project build.
* verify we can stop builds cleanly.
* get the rest of our logs into our local toaster dir
* allow us to run and stop 2 toasters on the same machine without them interfering with each other
* (2.2 or later for sure) discuss how to support multiple back end bb servers.
** probably will need to be able to add servers/remove servers/see server status/build state
* be able to see more details about build. For example, see the "jobs" being processed like you can in knotty
* revisit what variables we let people change. sstate_dir and dl_dir should certainly be allowed to change
* discuss way we do migrations. possibly change. thoughts:
** only support migrations from last release to current. (e.g. if you were running toaster jethro you could update to jethro++, but could not from fido to jethro)
** take migration from the startup script. it's only for releasetorelease.
*** allow it to be in for master and ease of development w/in a release and pull at end as we do with moving releases.
** whatever we decide, add a test set for it so we know it works
* Be able to delete things from the ui
** old builds
** old projects
** imported layers we no longer want
* Be able to set from the ui a policy to delete all builds/recipes/packages that are older than XXX &/or be able to set maximum size on db and delete everything older than XXX when we near that size
* update the builds on the web page so we dont have to hit refresh to see the build progress.

==== Elliot's ====

* Make sure all files lint properly
* Include lint in CI builds
* Improve test coverage of all areas of the UI and back-end
* Clean up toasterui.py so it's easier to see what's going on in there
* Fix localhostbecontroller so it doesn't rely on what's in the log file when figuring out whether the bitbake server has started
* Provide a way to modify settings in settings.py without editing that file (e.g. with a local overlay configuration file)
* Move the logic, environment variables, directory setup and script calls from bin/toaster into Django commands: this would allow you to run toaster correctly, with all the required setup, without having to use bin/toaster; it would also mean that we could provide cleaner production setup instructions, as a user could run a series of Django commands (with their own parameters) to setup their environment; bin/toaster would then just execute a series of Django commands

== All bugs for 2.1+ ==

{{#bugzilla:
|columns=id,milestone,summary,status,priority
|milestone=future,2.1,2.1 M1,2.1 M2,2.1 M3,2.1 M4
|sort=priority
|product=Toaster
|status=!resolved
}}

[https://bugzilla.yoctoproject.org/buglist.cgi?bug_status=UNCONFIRMED&bug_status=NEW&bug_status=ACCEPTED&bug_status=IN%20PROGRESS%20DESIGN&bug_status=IN%20PROGRESS%20DESIGN%20COMPLETE&bug_status=IN%20PROGRESS%20IMPLEMENTATION&bug_status=IN%20PROGRESS%20REVIEW&bug_status=REOPENED&bug_status=NEEDINFO&bug_status=WaitForUpstream&columnlist=product%2Cbug_status%2Cshort_desc%2Cchangeddate%2Ctarget_milestone&component=toaster&f1=target_milestone&list_id=476868&o1=greaterthan&product=Toaster&query_format=advanced&resolution=---&v1=2.0&query_based_on= Bugzilla query]

Toaster future release planning

2015-10-21T11:35:59Z

Elliot Smith: /* Wish lists */

[[Category:Toaster]]

== 2.1 Planning ==

=== Wish lists ===

==== Michael's ====
* Finish image-customisation 8070, 8081, 8082, 8103, 8104, 8117, 8128, 8132, 8091
* Remove pseudo API cruft / template context to json response stuff
* Consolidate used API into it's own view
* Finish toaster tables porting
* Get selenium test wrapper
* Create a toasterclient.py to interact with toaster to avoid need for 'command line builds'
** This would use toaster's API as the entry point to using bitbake
** It could be called by CI systems
** It could be used by auto builders etc
** Means that there are no two modes anywhere in Toaster
** Allows project configuration to be changed in toaster from the command line

==== Brian's ====
This is *not* ordered! This is more of a brain dump :)
* Collapse analysis and managed mode
* Working CI for each commit into toaster-next as well as for project peoples testME branches on poky-contrib:
** run django tests
** run selenium tests
** leverage sstate to make runs generally shorter
* remove git assumptions from code
* rationalize/simplify the configuration scripts
* if bitbake-memres comes out, interact with it gracefully (this is my preferred method of collapsing analysis and managed)
* fix the tmpdir/release build issue &/or discuss the need to support building old releases with a UI rather than limiting toaster to build the release it is part of. This is inherently fragile.
* ask bb to parse any layer added to a project and such as imported layers, and layers specified in the configuration file to give us recipe information.
* improve our build event following:
** note parse events
** update/retrieve ui so that we can tell something is happening before the first build event comes
** eliminate the remaining random tracebacks from try/except failures.
* allow configuration in addition to the addition/removal of layers/packages
** kernel config
* add user control/authentication
* update to Django 1.8. Always stay on Django LTS versions
* add layer update from layers.openembedded.org to the UI
* add Django tests for building e.g. we should be able to build core-image-sato from a django test and validate the db
* add mysql script to simplify setup with mysql/apache
* pull in new look and feel
* add django tests to validate all artifacts - if we say they can download it and it appears on a page anywhere, it should have a test
* improve docs of code on wiki. We have a nice start (ty E & M); I'd like to improve it.
** events - more info and list types supported
** what logic is in jscript , what is in python
** database schema definition and description esp interrelationships.
** list django tests, mostly so we can see what we are missing.
* build mode triggerred off of database entry, not autonomous process.
* add asynchronicity. I have noted a number of places where commands result in a sluggish ui. I'm nervous about this one.
** for example, starting a build that requires us to clone layers has a noticeable delay as compared to starting a local only project build.
* verify we can stop builds cleanly.
* get the rest of our logs into our local toaster dir
* allow us to run and stop 2 toasters on the same machine without them interfering with each other
* (2.2 or later for sure) discuss how to support multiple back end bb servers.
** probably will need to be able to add servers/remove servers/see server status/build state
* be able to see more details about build. For example, see the "jobs" being processed like you can in knotty
* revisit what variables we let people change. sstate_dir and dl_dir should certainly be allowed to change
* discuss way we do migrations. possibly change. thoughts:
** only support migrations from last release to current. (e.g. if you were running toaster jethro you could update to jethro++, but could not from fido to jethro)
** take migration from the startup script. it's only for releasetorelease.
*** allow it to be in for master and ease of development w/in a release and pull at end as we do with moving releases.
** whatever we decide, add a test set for it so we know it works
* Be able to delete things from the ui
** old builds
** old projects
** imported layers we no longer want
* Be able to set from the ui a policy to delete all builds/recipes/packages that are older than XXX &/or be able to set maximum size on db and delete everything older than XXX when we near that size
* update the builds on the web page so we dont have to hit refresh to see the build progress.

==== Elliot's ====

* Make sure all files lint properly
* Include lint in CI builds
* Improve test coverage of all areas of the UI and back-end
* Clean up toasterui.py so it's easier to see what's going on in there
* Fix localhostbecontroller so it doesn't rely on what's in the log file when figuring out whether the bitbake server has started

== All bugs for 2.1+ ==

{{#bugzilla:
|columns=id,milestone,summary,status,priority
|milestone=future,2.1,2.1 M1,2.1 M2,2.1 M3,2.1 M4
|sort=priority
|product=Toaster
|status=!resolved
}}

[https://bugzilla.yoctoproject.org/buglist.cgi?bug_status=UNCONFIRMED&bug_status=NEW&bug_status=ACCEPTED&bug_status=IN%20PROGRESS%20DESIGN&bug_status=IN%20PROGRESS%20DESIGN%20COMPLETE&bug_status=IN%20PROGRESS%20IMPLEMENTATION&bug_status=IN%20PROGRESS%20REVIEW&bug_status=REOPENED&bug_status=NEEDINFO&bug_status=WaitForUpstream&columnlist=product%2Cbug_status%2Cshort_desc%2Cchangeddate%2Ctarget_milestone&component=toaster&f1=target_milestone&list_id=476868&o1=greaterthan&product=Toaster&query_format=advanced&resolution=---&v1=2.0&query_based_on= Bugzilla query]

Contribute to Toaster

2015-09-24T13:28:55Z

Elliot Smith: /* Submitting patch sets for integration into Bitbake */

[[Category:Toaster]]
This page summarises the Toaster development process. We hope this will help you start contributing to the project.

== What can I do? ==

The [https://bugzilla.yoctoproject.org/buglist.cgi?product=Toaster Yocto Project Bugzilla instance] lists all the things that need to be done:

* If the issue says <strong>GUI design available</strong> in the Whiteboard field, there is a design specification document attached to the issue that you should follow. Send questions / comments about it to the [https://lists.yoctoproject.org/listinfo/toaster Toaster mailing list]
* If the issue says <strong>GUI design pending</strong> in the Whiteboard field, there is some design work still to be done. Feel free to take the issue and send an email to the [https://lists.yoctoproject.org/listinfo/toaster Toaster mailing list] to find out why the design work is not done yet

== Set up the local repository ==

For development of Toaster we recommend setting up a local install of Toaster. General install instructions are available in the main [https://www.yoctoproject.org/documentation/toaster-manual Toaster documentation]

Clone the poky repository
<code>
$ git clone git://git.yoctoproject.org/poky
$ cd poky
</code>

Install a python virtual environment to sandbox the python modules from your OS.
Enter Activate the python virtual environment in your current shell.
<code>
$ virtualenv venv
$ source ./venv/bin/activate
</code>

Install the python module dependencies for Toaster
<code>
$ pip install -r ./bitbake/toaster-requirements.txt
</code>

Run the setup and start script, follow instructions displayed
<code>
$ TOASTER_MANAGED=1 TOASTER_DEVEL=1 ./bitbake/bin/toaster
</code>

== Submitting patches ==

Publishing your patches to Toaster is a two step process.
# Sending patches to Toaster Project for review
# Submitting the patches that you reviewed to the upstream repository

Toaster code lives in Bitbake repository at [http://git.openembedded.org/bitbake/|http://git.openembedded.org/bitbake/].
All contributions must be upstreamed to the Bitbake repository in order to make it to the "master" branch of the poky/ repository.

=== Sending patches to Toaster Project ===

<strong>NOTE:</strong> The format of the commit message should be like this

<pre>
toaster: <short one line summary>

long(er) description

[YOCTO #0000]

Signed-off-by: First Last <name@domain.com>
</pre>

Where YOCTO #0000 is the related bug number if there is one. Signed off by with your git commit -s credentials.

We accept patches on the [https://www.yoctoproject.org/tools-resources/community/mailing-lists toaster mailing list] by "git send-email" please include in your subject line "[review-request][PATCH]"

e.g.
<code>
$ git send-email HEAD^ --subject-prefix="review-request][PATCH"
</code>

A comprehensive document about commit messages is available on the [http://www.openembedded.org/wiki/Commit_Patch_Message_Guidelines openembedded wiki]

More help learning git is available on [https://try.github.io github] and [http://git-scm.com/documentation/ the official documentation]

=== Sending branches to Toaster Project ===

If you wish to submit whole branches please use the poky-contrib repository see [[Poky Contributions#Poky_Contrib_Branch]] for setup guide.

Once you have pushed a branch please then send an email to the [https://www.yoctoproject.org/tools-resources/community/mailing-lists toaster mailing list] with the subject in the following format:

<code>
[review-request] my_branch_name
</code>

In the body of the email it's useful to describe your branch's functionality, which commits and a link to the git web.

If you need any assistance please post on the mailing list.

=== Submitting patch sets for integration into Bitbake ===

All Toaster patches need to be submitted upstream to the Bitbake repository after they have been reviewed on the Toaster mailing list. Since development happens on the poky-contrib repository, but the patches need to be merged to the Bitbake repository, the following process should be executed.

1) Fetch the target branch and checkout the target branch
<code>git fetch contrib the/target/branch && git checkout the/target/branch</code>

2) Create a new branch for submission
<code> git checkout -b yourname/submit/the/target/branch </code>

3) Make sure the branch is rebased on current poky master.
<code>git pull --rebase origin master</code>

4) Add signed off by to the commit messages
<code>git filter-branch -f --msg-filter 'cat && echo "Signed-off-by: $(git config --get user.name) <$(git config --get user.email)>"' master..HEAD</code>

5) Push the modified commit messages and rebased version to poky-contrib
<code>git push -u contrib yourname/submit/the/target/branch </code>

6) Use the create-pull-request script (from poky) to create a pull request
<code>./scripts/create-pull-request -d bitbake -s "Fixes and clean ups" -u contrib</code>

7) Review their content, especially the summary mail:
<code>edit ./pull-<pid>/0000-cover-letter.patch</code>

When you are satisfied, you can send them with:
<code>./scripts/send-pull-request -a -p ./pull-<pid></code>

to bitbake-devel@lists.openembedded.org

==== Submitting patches for prior releases ====

The procedure is the same, but using the prior release as the base branch instead of the "master" branch in bitbake.

Also, make sure that you add the name of the prior release for which the patchset is intended in the prefix of the patchset, as parameter to the "create-pull-request" command, e.g. '''-p 1.26''' for the 1.26 branch.

==== Gotchas ====

Sometimes the mailer will refuse to send patches, especially on binary or long-line files. The proper way to go around that is to reply to the patchset you've submitted to the mailing list, asking for a git pull directly from the poky-contrib branch.

'

== Code syle guide ==

=== Templates ===

Django has a template language which allows us to render pages based on the data (context). We use the template language to setup the initial state of the page and to create re-usable components that can be included in other pages.

The recommend template code style is as follows

'''Yes please:'''

<pre>
{{var}}

<div>
{# Maintaining indentation #}
{% if %}
<p>this</p>
{% else %}
<p>that</p>
{% endif %}
</div>

{% comment %}
This is a longer comment that describes all the things
that are below in quite a bit of detail because they're
a little more difficult to understand.
{% endcomment %}

{% for layer in layers_list %}
{{layer}}
{% endfor %}

</pre>

'''No thank you:'''
<pre>
{{var}}
<div>
{# Maintaining indentation #}
{%if%}<p>this</p>{%else%}<p>that</p>{%endif%}
</div>

{#This is a longer comment that describes all the things that are below in quite a bit of detail because they're a little more difficult to understand. #}
{%for o in layers_list%}{{o}}{%endfor%}
</pre>

Note:
* Maintain indentation as you would with other languages
* White space after '%'
* Comment blocks for longer comments

=== Javascript ===

'''Yes please:'''
<pre>

"use strict";

/* These hold some numbers */
var oneVar = 1;
var twoVar = 2;

var cheesesTypes = {
cheddar : 1,
stilton : 2,
emmental : 3,
};

function doThingsHere(){
return 1;
}

/* If one equals two do some other things and make sure that
* if the the click handler is setup correctly.
*/
if (one === two) {
var cheese = "cheddar";
oneVar = doThingsHere();

$(this).click(function (event){
alert("Hello");
});
}

$("#little-mouse").focusout(function(){
alert("bye")
});

if (oneVar)
noThingHere();
else
doThingHere();
</pre>

'''No thank you:'''
<pre>
// These hold some numbers
oneVar = 1
twoVar = 2

var cheesesTypes = { cheddar : 1, stilton : 2, emmental : 3, }

function doThingsHere ()
{
return 1;
}

//If one equals two do some other things and make sure that if the the click handler is setup correctly.
if( one === two ) {
var cheese = "cheddar";
oneVar = doThingsHere();

$(this).click(function(event){ alert("Hello"); });
}

document.getElementById("little-mouse").addEventListener("focusout", function(){
alert("bye")
});

if (oneVar)
{
noThingHere();
} else { doThingHere(); }
</pre>

Note:
* Variables should be marked with "var"
* Semicolons should be used
* Keep as close to 80 cols as possible
* Use 2 space per indentation
* Open curly braces after parenthesis for functions and close on a new line
* Use camelCase for function names and variable names

Make use of running your Javascript through jshint we have a .jshint configuration file in that js directory (toastergui/static/js)

e.g. install jshint and add to your current PATH, then run:
<code>
$ npm install jshint; export PATH=$PATH:$PWD/node_modules/.bin/
$ jshint ./toastergui/static/js/base.js
</code>

=== HTML ===

'''Yes please:'''
<pre>
<div id="something-area">
<p class="important">This is some text</p>
</div>

<div>
<p id="important-text>This is some text</p>
</div>

</pre>

'''No thank you:'''
<pre>
<div id="somethingarea">
<p class="Important">This is some text</p></div>
<div id="somethingarea">
<p id="ImportantText">This is
some text
</p>

</div>
</pre>

Note:
* 2 space indentation
* Lower case, ids hyphenated when multiple words
* No duplicate ids

* Run your HTML through a [http://validator.w3.org/#validate_by_input HTML validator] available for [http://validator.w3.org/source/ local install]. The w3c validator it's self doesn't currently validate html5, it uses as a back end [https://validator.github.io/validator/ Nu Html Checker] which can be installed as a standalone service, full instructions in the readme.

Quick install instructions:
<pre>
$ mkdir html5-validator && cd html5-validator
$ export JAVA_HOME=/usr/lib/jvm/java-6-openjdk
$ git clone https://github.com/validator/validator.git
$ python build/build.py all
$ python build/build.py all
</pre>

HTML can be indented quickly using tidy, for example:
<pre>
tidy -xml --indent auto --indent-spaces 2 --quiet yes -w -1 --show-body-only yes ./index.html
</pre>

=== Python ===

Lenient [https://www.python.org/dev/peps/pep-0008 pep8]
Ignoring most of the whitespace around character issues (E124,E203,E201,E265,E303,E302,E231) see toaster/.pep8 and [http://pep8.readthedocs.org/en/latest/intro.html#error-codes error code list]

Fix all issues identified by running code through pep8. We have a fairly lenient config file (toaster/.pep8).

e.g.
<code>
$ pep8 ./toastergui/urls.py
</code>

Run code through pylint and fix identified issues - Some can be reasonably ignored such as doc strings for every function or star-args. No pylintrc config provided here as most issues identified are highly contextual and should be ignored on a case by case basis.

<code>
$ pylint --load-plugins pylint_django toastergui/tests.py
</code>

== Working with design ==

Yes, the Yocto Project is one of those lucky projects with designers around to help in UI matters. We have a document explaining how to work with the design contributors: [[File:Working_with_design.pdf]]

Contribute to Toaster

2015-09-24T13:27:35Z

Elliot Smith: /* Sending patches to Toaster Project */

[[Category:Toaster]]
This page summarises the Toaster development process. We hope this will help you start contributing to the project.

== What can I do? ==

The [https://bugzilla.yoctoproject.org/buglist.cgi?product=Toaster Yocto Project Bugzilla instance] lists all the things that need to be done:

* If the issue says <strong>GUI design available</strong> in the Whiteboard field, there is a design specification document attached to the issue that you should follow. Send questions / comments about it to the [https://lists.yoctoproject.org/listinfo/toaster Toaster mailing list]
* If the issue says <strong>GUI design pending</strong> in the Whiteboard field, there is some design work still to be done. Feel free to take the issue and send an email to the [https://lists.yoctoproject.org/listinfo/toaster Toaster mailing list] to find out why the design work is not done yet

== Set up the local repository ==

For development of Toaster we recommend setting up a local install of Toaster. General install instructions are available in the main [https://www.yoctoproject.org/documentation/toaster-manual Toaster documentation]

Clone the poky repository
<code>
$ git clone git://git.yoctoproject.org/poky
$ cd poky
</code>

Install a python virtual environment to sandbox the python modules from your OS.
Enter Activate the python virtual environment in your current shell.
<code>
$ virtualenv venv
$ source ./venv/bin/activate
</code>

Install the python module dependencies for Toaster
<code>
$ pip install -r ./bitbake/toaster-requirements.txt
</code>

Run the setup and start script, follow instructions displayed
<code>
$ TOASTER_MANAGED=1 TOASTER_DEVEL=1 ./bitbake/bin/toaster
</code>

== Submitting patches ==

Publishing your patches to Toaster is a two step process.
# Sending patches to Toaster Project for review
# Submitting the patches that you reviewed to the upstream repository

Toaster code lives in Bitbake repository at [http://git.openembedded.org/bitbake/|http://git.openembedded.org/bitbake/].
All contributions must be upstreamed to the Bitbake repository in order to make it to the "master" branch of the poky/ repository.

=== Sending patches to Toaster Project ===

<strong>NOTE:</strong> The format of the commit message should be like this

<pre>
toaster: <short one line summary>

long(er) description

[YOCTO #0000]

Signed-off-by: First Last <name@domain.com>
</pre>

Where YOCTO #0000 is the related bug number if there is one. Signed off by with your git commit -s credentials.

We accept patches on the [https://www.yoctoproject.org/tools-resources/community/mailing-lists toaster mailing list] by "git send-email" please include in your subject line "[review-request][PATCH]"

e.g.
<code>
$ git send-email HEAD^ --subject-prefix="review-request][PATCH"
</code>

A comprehensive document about commit messages is available on the [http://www.openembedded.org/wiki/Commit_Patch_Message_Guidelines openembedded wiki]

More help learning git is available on [https://try.github.io github] and [http://git-scm.com/documentation/ the official documentation]

=== Sending branches to Toaster Project ===

If you wish to submit whole branches please use the poky-contrib repository see [[Poky Contributions#Poky_Contrib_Branch]] for setup guide.

Once you have pushed a branch please then send an email to the [https://www.yoctoproject.org/tools-resources/community/mailing-lists toaster mailing list] with the subject in the following format:

<code>
[review-request] my_branch_name
</code>

In the body of the email it's useful to describe your branch's functionality, which commits and a link to the git web.

If you need any assistance please post on the mailing list.

=== Submitting patch sets for integration into Bitbake ===

All Toaster patches need to be submitted upstream to the Bitbake repository after they have been reviewed on the Toaster mailing list. Since development happens on the poky-contrib repository, but the patches need to be merged to the Bitbake repository, the following process should be executed.

1) Fetch the target branch and checkout the target branch
<code>git fetch contrib the/target/branch && git checkout the/target/branch</code>

2) Create a new branch for submission
<code> git checkout -b yourname/submit/the/target/branch </code>

3) Make sure the branch is rebased on current master.
<code>git pull --rebase origin master</code>

4) Add signed off by to the commit messages
<code>git filter-branch -f --msg-filter 'cat && echo "Signed-off-by: $(git config --get user.name) <$(git config --get user.email)>"' master..HEAD</code>

5) Push the modified commit messages and rebased version to poky-contrib
<code>git push -u contrib yourname/submit/the/target/branch </code>

6) Use the create-pull-request script (from poky) to create a pull request
<code>./scripts/create-pull-request -d bitbake -s "Fixes and clean ups" -u contrib</code>

7) Review their content, especially the summary mail:
<code>edit ./pull-<pid>/0000-cover-letter.patch</code>

When you are satisfied, you can send them with:
<code>./scripts/send-pull-request -a -p ./pull-<pid></code>

to bitbake-devel@lists.openembedded.org

==== Submitting patches for prior releases ====

The procedure is the same, but using the prior release as the base branch instead of the "master" branch in bitbake.

Also, make sure that you add the name of the prior release for which the patchset is intended in the prefix of the patchset, as parameter to the "create-pull-request" command, e.g. '''-p 1.26''' for the 1.26 branch.

==== Gotchas ====

Sometimes the mailer will refuse to send patches, especially on binary or long-line files. The proper way to go around that is to reply to the patchset you've submitted to the mailing list, asking for a git pull directly from the poky-contrib branch.

'

== Code syle guide ==

=== Templates ===

Django has a template language which allows us to render pages based on the data (context). We use the template language to setup the initial state of the page and to create re-usable components that can be included in other pages.

The recommend template code style is as follows

'''Yes please:'''

<pre>
{{var}}

<div>
{# Maintaining indentation #}
{% if %}
<p>this</p>
{% else %}
<p>that</p>
{% endif %}
</div>

{% comment %}
This is a longer comment that describes all the things
that are below in quite a bit of detail because they're
a little more difficult to understand.
{% endcomment %}

{% for layer in layers_list %}
{{layer}}
{% endfor %}

</pre>

'''No thank you:'''
<pre>
{{var}}
<div>
{# Maintaining indentation #}
{%if%}<p>this</p>{%else%}<p>that</p>{%endif%}
</div>

{#This is a longer comment that describes all the things that are below in quite a bit of detail because they're a little more difficult to understand. #}
{%for o in layers_list%}{{o}}{%endfor%}
</pre>

Note:
* Maintain indentation as you would with other languages
* White space after '%'
* Comment blocks for longer comments

=== Javascript ===

'''Yes please:'''
<pre>

"use strict";

/* These hold some numbers */
var oneVar = 1;
var twoVar = 2;

var cheesesTypes = {
cheddar : 1,
stilton : 2,
emmental : 3,
};

function doThingsHere(){
return 1;
}

/* If one equals two do some other things and make sure that
* if the the click handler is setup correctly.
*/
if (one === two) {
var cheese = "cheddar";
oneVar = doThingsHere();

$(this).click(function (event){
alert("Hello");
});
}

$("#little-mouse").focusout(function(){
alert("bye")
});

if (oneVar)
noThingHere();
else
doThingHere();
</pre>

'''No thank you:'''
<pre>
// These hold some numbers
oneVar = 1
twoVar = 2

var cheesesTypes = { cheddar : 1, stilton : 2, emmental : 3, }

function doThingsHere ()
{
return 1;
}

//If one equals two do some other things and make sure that if the the click handler is setup correctly.
if( one === two ) {
var cheese = "cheddar";
oneVar = doThingsHere();

$(this).click(function(event){ alert("Hello"); });
}

document.getElementById("little-mouse").addEventListener("focusout", function(){
alert("bye")
});

if (oneVar)
{
noThingHere();
} else { doThingHere(); }
</pre>

Note:
* Variables should be marked with "var"
* Semicolons should be used
* Keep as close to 80 cols as possible
* Use 2 space per indentation
* Open curly braces after parenthesis for functions and close on a new line
* Use camelCase for function names and variable names

Make use of running your Javascript through jshint we have a .jshint configuration file in that js directory (toastergui/static/js)

e.g. install jshint and add to your current PATH, then run:
<code>
$ npm install jshint; export PATH=$PATH:$PWD/node_modules/.bin/
$ jshint ./toastergui/static/js/base.js
</code>

=== HTML ===

'''Yes please:'''
<pre>
<div id="something-area">
<p class="important">This is some text</p>
</div>

<div>
<p id="important-text>This is some text</p>
</div>

</pre>

'''No thank you:'''
<pre>
<div id="somethingarea">
<p class="Important">This is some text</p></div>
<div id="somethingarea">
<p id="ImportantText">This is
some text
</p>

</div>
</pre>

Note:
* 2 space indentation
* Lower case, ids hyphenated when multiple words
* No duplicate ids

* Run your HTML through a [http://validator.w3.org/#validate_by_input HTML validator] available for [http://validator.w3.org/source/ local install]. The w3c validator it's self doesn't currently validate html5, it uses as a back end [https://validator.github.io/validator/ Nu Html Checker] which can be installed as a standalone service, full instructions in the readme.

Quick install instructions:
<pre>
$ mkdir html5-validator && cd html5-validator
$ export JAVA_HOME=/usr/lib/jvm/java-6-openjdk
$ git clone https://github.com/validator/validator.git
$ python build/build.py all
$ python build/build.py all
</pre>

HTML can be indented quickly using tidy, for example:
<pre>
tidy -xml --indent auto --indent-spaces 2 --quiet yes -w -1 --show-body-only yes ./index.html
</pre>

=== Python ===

Lenient [https://www.python.org/dev/peps/pep-0008 pep8]
Ignoring most of the whitespace around character issues (E124,E203,E201,E265,E303,E302,E231) see toaster/.pep8 and [http://pep8.readthedocs.org/en/latest/intro.html#error-codes error code list]

Fix all issues identified by running code through pep8. We have a fairly lenient config file (toaster/.pep8).

e.g.
<code>
$ pep8 ./toastergui/urls.py
</code>

Run code through pylint and fix identified issues - Some can be reasonably ignored such as doc strings for every function or star-args. No pylintrc config provided here as most issues identified are highly contextual and should be ignored on a case by case basis.

<code>
$ pylint --load-plugins pylint_django toastergui/tests.py
</code>

== Working with design ==

Yes, the Yocto Project is one of those lucky projects with designers around to help in UI matters. We have a document explaining how to work with the design contributors: [[File:Working_with_design.pdf]]

Toaster and bitbake communications

2015-09-11T14:53:16Z

Elliot Smith: /* main.py (bitbake/lib/bb/main.py) */

Toaster and bitbake communications

2015-09-11T14:51:14Z

Elliot Smith: Created page with "This is a write up of some investigation I did into how toaster asks bitbake to do stuff, and how it hears about what bitbake is doing. It's not definitive and not guaranteed ..."

Toaster

2015-09-11T14:47:55Z

Elliot Smith: /* About Toaster */

[[Category:Toaster]]
[[Toaster]] is a web-based interface to OpenEmbedded and BitBake. You might have heard it is a replacement for [https://www.yoctoproject.org/documentation/hob-manual Hob]. The answer is "it will be at some point, but not yet".

General discussion about '''Toaster''' happens on a dedicated mailing list: [https://lists.yoctoproject.org/listinfo/toaster https://lists.yoctoproject.org/listinfo/toaster]

=== Using Toaster ===

Toaster can run in various modes and setups. To ease out the understanding of documentation, we review here the terminology used throughout the documentation.

* Interactive mode - this is the mode released with Yocto Project Release 1.6. Functional components consist of a build recording component, 'toasterui', and a build stats and inspection user interface, the 'toastergui'. It is started with the command sequence listed below, and the builds are started using normal bitbake command line

$ source oe-init-build-env
$ source toaster start

* Managed mode - in this mode Toaster handles the build configuration GUI (through Project pages), and build scheduling and execution, in addition to the features launched with Yocto Project Release 1.6. The builds are triggered through the web interface,
the user as no direct access to the bitbake command line.

** Local managed mode, in short _local_, is the out-of-box mode available after a poky/ checkout. It allows the user to perform build on his local machine source code, with a local build directory, and is intended to help the user discover the interface, and configure and run local builds. You can launch it with

$ ./bitbake/bin/toaster

** Remote managed mode, or [[hosted Toaster]], is intended to be the production setup for running Toaster in organizations supporting multiple users and using customized Toaster installations.

=== Toaster How-to's ===

Specific pages with Toaster how-tos are available below.

* [[Contribute to Toaster]]
* [[Testing Toaster]]
* [[Setting up a local instance of Toaster]]
* [[Setting up a production instance of Toaster]] - documentation for Interactive mode
* [[Setting up a hosted managed mode for Toaster]] - configure and run build through Toaster itself, providing a service for your users
* [https://www.yoctoproject.org/documentation/toaster-manual-17 How to use the Toaster web interface]
* [[How to delete information from the Toaster database]]

=== About Toaster ===

* [[File:Working_with_design.pdf]]
* [https://bugzilla.yoctoproject.org/buglist.cgi?list_id=213820&columnlist=status_whiteboard%2Cassigned_to%2Ctarget_milestone%2Cbug_status%2Cshort_desc%2Cbug_severity%2Cpriority&classification=Build%20System%20%26%20Metadata&query_based_on=Toaster-Opens&query_format=advanced&bug_status=NEW&bug_status=ACCEPTED&bug_status=IN%20PROGRESS%20DESIGN&bug_status=IN%20PROGRESS%20DESIGN%20COMPLETE&bug_status=IN%20PROGRESS%20IMPLEMENTATION&bug_status=REOPENED&bug_status=NEEDINFO&bug_status=WaitForUpstream&component=toaster&product=Toaster&known_name=Toaster-Opens Bug list]
* [[Toaster architecture design]]
* [[Toaster and bitbake communications]]
* [[Toaster testing plan]]
* [[Toaster 1.9 design | Scope and design (1.9)]]
* [[Toaster 1.7 design | Scope and design (1.7 - 1.8)]]
* [[Toaster archive | Archive]]

=== In progress documentation ===

We are currently preparing the documentation for the Toaster build functionality. The content here is just a brain dump of what we need to cover (in no particular order). Feel free to add and create content as you see fit:

*[[Using virtualenv]]
*[[Toaster configuration]] - explain releases, layer sources, BitBake versions and default project configurations
*[[Set up Toaster locally]] - explain the set up process and the local version of the localconf.json file
*[[Set up production instance]] - explain the Django admin interface and the remove version of the localconf.json file
*[[manage.py commands]] - this should include an explanation of lsupdates
*[[Custom layer index]] - explain how to create your own layer index
*[[Start Toaster in managed mode]]