Yocto Project - User contributions [en]

Toaster custom data

2016-11-23T13:46:28Z

Michael Wood: /* Add a default layer from a git source */

When Toaster is first started up it uses Django fixtures to populate some initial data into it's database. Once this is done we then connect to the [http://layers.openembedded.org openembedded layer index] and download information about various metadata that is available for the configured releases.

== Fixtures directory lib/toaster/orm/fixtures ==

Fixtures are data dumps that can be loaded into Toaster's database to provide
configuration and data.

In this directory we have the fixtures which are loaded the first time you start Toaster.
This is to provide useful default values and metadata to Toaster.

* settings.xml This Contains Toaster wide settings, such as the default values for certain bitbake variables.
* poky.xml This is the default release data for supported poky based setup
* oe-core.xml This is the default release data for supported oe-core based setups

=== Custom data/configuration ===

* custom.xml

To add custom initial data/configuration to Toaster place a file called
"custom.xml" in bitbake/lib/toaster/orm/fixtures/ directory. If present it will be loaded into the database on start up, last.
We suggest that this is used to overlay any configuration already done.
All objects loaded with the same primary keys overwrite the existing data.
Data can be provided in XML, JSON and if installed YAML formats.

To understand the requirements of the schema and the field names and default values see their definitions in:
* bitbake/lib/toaster/orm/models.py
* bitbake/lib/toaster/bldcontrol/models.py
* [[Toaster database]]

Note that specifying primary keys are optional. The default behaviour is to replace any data which has the same primary key, this is useful if you wish to override data that is already being set use the same primary key. When there are relationships between different objects such as Many-to-One it is important to have the primary key set on the objects in the relationship so that this can be defined. E.g. A Layer object has many Layer Versions objects so to create this relationship a Layer needs a primary key specified and a Layer Version needs to specify that it 'belongs' to that Layer via the layer field .e.g
<field rel="ManyToOneRel" to="orm.layer" name="layer">10</field>

More information about django fixtures can be found at https://docs.djangoproject.com/en/1.8/howto/initial-data/

== To load data at any point in time ==

Use the django management command
manage.py loaddata <your fixture file>
For further information see the Django command documentation at:
https://docs.djangoproject.com/en/1.8/ref/django-admin/#django-admin-loaddata

== Examples ==
Examples given in xml format for increased verbosity.
* custom.xml

=== Override default package for projects ===

<pre>
<?xml version="1.0" encoding="utf-8"?>
<django-objects version="1.0">

<object model="orm.toastersetting" pk="3">
<field type="CharField" name="name">DEFCONF_PACKAGE_CLASSES</field>
<field type="CharField" name="value">package_ipk</field>
</object>

</django-objects>
</pre>

=== Add a default layer from a git source ===
<pre>
<?xml version="1.0" encoding="utf-8"?>
<django-objects version="1.0">

<object model="orm.layer" pk="10">
<field type="CharField" name="name">meta-my-layer</field>
<field type="CharField" name="layer_index_url"></field>
<field type="CharField" name="vcs_url">git://git.example.com/my_repo</field>
</object>
<object model="orm.layer_version" pk="13">
<field rel="ManyToOneRel" to="orm.layer" name="layer">10</field>
<field type="IntegerField" name="layer_source">0</field>
<field rel="ManyToOneRel" to="orm.release" name="release">1</field>
<field type="CharField" name="branch">v1.2.3</field>
<field type="CharField" name="dirpath">layers/meta-my-layer</field>
</object>

<object model="orm.releasedefaultlayer" pk="15">
<field rel="ManyToOneRel" to="orm.release" name="release">3</field>
<field type="CharField" name="layer_name">meta-my-layer</field>
</object>

</django-objects>
</pre>

=== Add an available layer from a local file system source ===
<pre>
<?xml version="1.0" encoding="utf-8"?>
<django-objects version="1.0">

<object model="orm.layer" pk="20">
<field type="CharField" name="name">meta-my-layer</field>
<field type="CharField" name="layer_index_url"></field>
<field type="CharField" name="local_source_dir">/home/developer/work/meta-my-layer</field>
</object>
<object model="orm.layer_version" pk="13">
<field rel="ManyToOneRel" to="orm.layer" name="layer">20</field>
<field type="IntegerField" name="layer_source">0</field>
<field rel="ManyToOneRel" to="orm.release" name="release">1</field>
</object>

</django-objects>
</pre>

Toaster custom data

2016-11-23T13:46:16Z

Michael Wood: /* Custom data/configuration */

When Toaster is first started up it uses Django fixtures to populate some initial data into it's database. Once this is done we then connect to the [http://layers.openembedded.org openembedded layer index] and download information about various metadata that is available for the configured releases.

== Fixtures directory lib/toaster/orm/fixtures ==

Fixtures are data dumps that can be loaded into Toaster's database to provide
configuration and data.

In this directory we have the fixtures which are loaded the first time you start Toaster.
This is to provide useful default values and metadata to Toaster.

* settings.xml This Contains Toaster wide settings, such as the default values for certain bitbake variables.
* poky.xml This is the default release data for supported poky based setup
* oe-core.xml This is the default release data for supported oe-core based setups

=== Custom data/configuration ===

* custom.xml

To add custom initial data/configuration to Toaster place a file called
"custom.xml" in bitbake/lib/toaster/orm/fixtures/ directory. If present it will be loaded into the database on start up, last.
We suggest that this is used to overlay any configuration already done.
All objects loaded with the same primary keys overwrite the existing data.
Data can be provided in XML, JSON and if installed YAML formats.

To understand the requirements of the schema and the field names and default values see their definitions in:
* bitbake/lib/toaster/orm/models.py
* bitbake/lib/toaster/bldcontrol/models.py
* [[Toaster database]]

Note that specifying primary keys are optional. The default behaviour is to replace any data which has the same primary key, this is useful if you wish to override data that is already being set use the same primary key. When there are relationships between different objects such as Many-to-One it is important to have the primary key set on the objects in the relationship so that this can be defined. E.g. A Layer object has many Layer Versions objects so to create this relationship a Layer needs a primary key specified and a Layer Version needs to specify that it 'belongs' to that Layer via the layer field .e.g
<field rel="ManyToOneRel" to="orm.layer" name="layer">10</field>

More information about django fixtures can be found at https://docs.djangoproject.com/en/1.8/howto/initial-data/

== To load data at any point in time ==

Use the django management command
manage.py loaddata <your fixture file>
For further information see the Django command documentation at:
https://docs.djangoproject.com/en/1.8/ref/django-admin/#django-admin-loaddata

== Examples ==
Examples given in xml format for increased verbosity.
* custom.xml

=== Override default package for projects ===

<pre>
<?xml version="1.0" encoding="utf-8"?>
<django-objects version="1.0">

<object model="orm.toastersetting" pk="3">
<field type="CharField" name="name">DEFCONF_PACKAGE_CLASSES</field>
<field type="CharField" name="value">package_ipk</field>
</object>

</django-objects>
</pre>

=== Add a default layer from a git source ===
<pre>
<?xml version="1.0" encoding="utf-8"?>
<django-objects version="1.0">

<object model="orm.layer" pk="10">
<field type="CharField" name="name">meta-my-layer</field>
<field type="CharField" name="layer_index_url"></field>
<field type="CharField" name="vcs_url">git://git.example.com/my_repo</field>
</object>
<object model="orm.layer_version" pk="13">
<field rel="ManyToOneRel" to="orm.layer" name="layer">10</field>
<field type="IntegerField" name="layer_source">0</field>
<field rel="ManyToOneRel" to="orm.release" name="release">1</field>
<field type="CharField" name="branch">v1.2.3</field>
<field type="CharField" name="dirpath">layers/meta-my-layer</field>
</object>

<object model="orm.releasedefaultlayer" pk="15">
<field rel="ManyToOneRel" to="orm.release" name="release">3</field>
<field type="CharField" name="layer_name">meta-my-layer</field>
</object>
</django-objects>
</pre>

=== Add an available layer from a local file system source ===
<pre>
<?xml version="1.0" encoding="utf-8"?>
<django-objects version="1.0">

<object model="orm.layer" pk="20">
<field type="CharField" name="name">meta-my-layer</field>
<field type="CharField" name="layer_index_url"></field>
<field type="CharField" name="local_source_dir">/home/developer/work/meta-my-layer</field>
</object>
<object model="orm.layer_version" pk="13">
<field rel="ManyToOneRel" to="orm.layer" name="layer">20</field>
<field type="IntegerField" name="layer_source">0</field>
<field rel="ManyToOneRel" to="orm.release" name="release">1</field>
</object>

</django-objects>
</pre>

Toaster custom data

2016-11-23T13:41:58Z

Michael Wood: /* Custom data/configuration */

When Toaster is first started up it uses Django fixtures to populate some initial data into it's database. Once this is done we then connect to the [http://layers.openembedded.org openembedded layer index] and download information about various metadata that is available for the configured releases.

== Fixtures directory lib/toaster/orm/fixtures ==

Fixtures are data dumps that can be loaded into Toaster's database to provide
configuration and data.

In this directory we have the fixtures which are loaded the first time you start Toaster.
This is to provide useful default values and metadata to Toaster.

* settings.xml This Contains Toaster wide settings, such as the default values for certain bitbake variables.
* poky.xml This is the default release data for supported poky based setup
* oe-core.xml This is the default release data for supported oe-core based setups

=== Custom data/configuration ===

* custom.xml

To add custom initial data/configuration to Toaster place a file called
"custom.xml" in this directory. If present it will be loaded into the database.
We suggest that this is used to overlay any configuration already done.
All objects loaded with the same primary keys overwrite the existing data.
Data can be provided in XML, JSON and if installed YAML formats.

To understand the requirements of the schema and the field names and default values see their definitions in:
* bitbake/lib/toaster/orm/models.py
* bitbake/lib/toaster/bldcontrol/models.py
* [[Toaster database]]

Note that specifying primary keys are optional. The default behaviour is to replace any data which has the same primary key, this is useful if you wish to override data that is already being set use the same primary key. When there are relationships between different objects such as Many-to-One it is important to have the primary key set on the objects in the relationship so that this can be defined. E.g. A Layer object has many Layer Versions objects so to create this relationship a Layer needs a primary key specified and a Layer Version needs to specify that it 'belongs' to that Layer via the layer field .e.g
<field rel="ManyToOneRel" to="orm.layer" name="layer">10</field>

More information about django fixtures can be found at https://docs.djangoproject.com/en/1.8/howto/initial-data/

== To load data at any point in time ==

Use the django management command
manage.py loaddata <your fixture file>
For further information see the Django command documentation at:
https://docs.djangoproject.com/en/1.8/ref/django-admin/#django-admin-loaddata

== Examples ==
Examples given in xml format for increased verbosity.
* custom.xml

=== Override default package for projects ===

<pre>
<?xml version="1.0" encoding="utf-8"?>
<django-objects version="1.0">

<object model="orm.toastersetting" pk="3">
<field type="CharField" name="name">DEFCONF_PACKAGE_CLASSES</field>
<field type="CharField" name="value">package_ipk</field>
</object>

</django-objects>
</pre>

=== Add a default layer from a git source ===
<pre>
<?xml version="1.0" encoding="utf-8"?>
<django-objects version="1.0">

<object model="orm.layer" pk="10">
<field type="CharField" name="name">meta-my-layer</field>
<field type="CharField" name="layer_index_url"></field>
<field type="CharField" name="vcs_url">git://git.example.com/my_repo</field>
</object>
<object model="orm.layer_version" pk="13">
<field rel="ManyToOneRel" to="orm.layer" name="layer">10</field>
<field type="IntegerField" name="layer_source">0</field>
<field rel="ManyToOneRel" to="orm.release" name="release">1</field>
<field type="CharField" name="branch">v1.2.3</field>
<field type="CharField" name="dirpath">layers/meta-my-layer</field>
</object>

<object model="orm.releasedefaultlayer" pk="15">
<field rel="ManyToOneRel" to="orm.release" name="release">3</field>
<field type="CharField" name="layer_name">meta-my-layer</field>
</object>
</django-objects>
</pre>

=== Add an available layer from a local file system source ===
<pre>
<?xml version="1.0" encoding="utf-8"?>
<django-objects version="1.0">

<object model="orm.layer" pk="20">
<field type="CharField" name="name">meta-my-layer</field>
<field type="CharField" name="layer_index_url"></field>
<field type="CharField" name="local_source_dir">/home/developer/work/meta-my-layer</field>
</object>
<object model="orm.layer_version" pk="13">
<field rel="ManyToOneRel" to="orm.layer" name="layer">20</field>
<field type="IntegerField" name="layer_source">0</field>
<field rel="ManyToOneRel" to="orm.release" name="release">1</field>
</object>

</django-objects>
</pre>

Toaster custom data

2016-11-23T13:41:10Z

Michael Wood: /* To load data at any point in time */

When Toaster is first started up it uses Django fixtures to populate some initial data into it's database. Once this is done we then connect to the [http://layers.openembedded.org openembedded layer index] and download information about various metadata that is available for the configured releases.

== Fixtures directory lib/toaster/orm/fixtures ==

Fixtures are data dumps that can be loaded into Toaster's database to provide
configuration and data.

In this directory we have the fixtures which are loaded the first time you start Toaster.
This is to provide useful default values and metadata to Toaster.

* settings.xml This Contains Toaster wide settings, such as the default values for certain bitbake variables.
* poky.xml This is the default release data for supported poky based setup
* oe-core.xml This is the default release data for supported oe-core based setups

=== Custom data/configuration ===

* custom.xml

To add custom initial data/configuration to Toaster place a file called
"custom.xml" in this directory. If present it will be loaded into the database.
We suggest that this is used to overlay any configuration already done.
All objects loaded with the same primary keys overwrite the existing data.
Data can be provided in XML, JSON and if installed YAML formats.

To understand the requirements of the schema and the field names and default values see their definitions in:
* orm/models.py
* bldcontrol/models.py
* [[Toaster database]]

Note that specifying primary keys are optional. The default behaviour is to replace any data which has the same primary key, this is useful if you wish to override data that is already being set use the same primary key. When there are relationships between different objects such as Many-to-One it is important to have the primary key set on the objects in the relationship so that this can be defined. E.g. A Layer object has many Layer Versions objects so to create this relationship a Layer needs a primary key specified and a Layer Version needs to specify that it 'belongs' to that Layer via the layer field .e.g
<field rel="ManyToOneRel" to="orm.layer" name="layer">10</field>

More information about django fixtures can be found at https://docs.djangoproject.com/en/1.8/howto/initial-data/

== To load data at any point in time ==

Use the django management command
manage.py loaddata <your fixture file>
For further information see the Django command documentation at:
https://docs.djangoproject.com/en/1.8/ref/django-admin/#django-admin-loaddata

== Examples ==
Examples given in xml format for increased verbosity.
* custom.xml

=== Override default package for projects ===

<pre>
<?xml version="1.0" encoding="utf-8"?>
<django-objects version="1.0">

<object model="orm.toastersetting" pk="3">
<field type="CharField" name="name">DEFCONF_PACKAGE_CLASSES</field>
<field type="CharField" name="value">package_ipk</field>
</object>

</django-objects>
</pre>

=== Add a default layer from a git source ===
<pre>
<?xml version="1.0" encoding="utf-8"?>
<django-objects version="1.0">

<object model="orm.layer" pk="10">
<field type="CharField" name="name">meta-my-layer</field>
<field type="CharField" name="layer_index_url"></field>
<field type="CharField" name="vcs_url">git://git.example.com/my_repo</field>
</object>
<object model="orm.layer_version" pk="13">
<field rel="ManyToOneRel" to="orm.layer" name="layer">10</field>
<field type="IntegerField" name="layer_source">0</field>
<field rel="ManyToOneRel" to="orm.release" name="release">1</field>
<field type="CharField" name="branch">v1.2.3</field>
<field type="CharField" name="dirpath">layers/meta-my-layer</field>
</object>

<object model="orm.releasedefaultlayer" pk="15">
<field rel="ManyToOneRel" to="orm.release" name="release">3</field>
<field type="CharField" name="layer_name">meta-my-layer</field>
</object>
</django-objects>
</pre>

=== Add an available layer from a local file system source ===
<pre>
<?xml version="1.0" encoding="utf-8"?>
<django-objects version="1.0">

<object model="orm.layer" pk="20">
<field type="CharField" name="name">meta-my-layer</field>
<field type="CharField" name="layer_index_url"></field>
<field type="CharField" name="local_source_dir">/home/developer/work/meta-my-layer</field>
</object>
<object model="orm.layer_version" pk="13">
<field rel="ManyToOneRel" to="orm.layer" name="layer">20</field>
<field type="IntegerField" name="layer_source">0</field>
<field rel="ManyToOneRel" to="orm.release" name="release">1</field>
</object>

</django-objects>
</pre>

Toaster custom data

2016-11-23T13:40:39Z

Michael Wood: Created page with "When Toaster is first started up it uses Django fixtures to populate some initial data into it's database. Once this is done we then connect to the [http://layers.openembedded..."

When Toaster is first started up it uses Django fixtures to populate some initial data into it's database. Once this is done we then connect to the [http://layers.openembedded.org openembedded layer index] and download information about various metadata that is available for the configured releases.

== Fixtures directory lib/toaster/orm/fixtures ==

Fixtures are data dumps that can be loaded into Toaster's database to provide
configuration and data.

In this directory we have the fixtures which are loaded the first time you start Toaster.
This is to provide useful default values and metadata to Toaster.

* settings.xml This Contains Toaster wide settings, such as the default values for certain bitbake variables.
* poky.xml This is the default release data for supported poky based setup
* oe-core.xml This is the default release data for supported oe-core based setups

=== Custom data/configuration ===

* custom.xml

To add custom initial data/configuration to Toaster place a file called
"custom.xml" in this directory. If present it will be loaded into the database.
We suggest that this is used to overlay any configuration already done.
All objects loaded with the same primary keys overwrite the existing data.
Data can be provided in XML, JSON and if installed YAML formats.

To understand the requirements of the schema and the field names and default values see their definitions in:
* orm/models.py
* bldcontrol/models.py
* [[Toaster database]]

Note that specifying primary keys are optional. The default behaviour is to replace any data which has the same primary key, this is useful if you wish to override data that is already being set use the same primary key. When there are relationships between different objects such as Many-to-One it is important to have the primary key set on the objects in the relationship so that this can be defined. E.g. A Layer object has many Layer Versions objects so to create this relationship a Layer needs a primary key specified and a Layer Version needs to specify that it 'belongs' to that Layer via the layer field .e.g
<field rel="ManyToOneRel" to="orm.layer" name="layer">10</field>

More information about django fixtures can be found at https://docs.djangoproject.com/en/1.8/howto/initial-data/

== To load data at any point in time ==

Use the django management command manage.py loaddata <your fixture file>
For further information see the Django command documentation at:
https://docs.djangoproject.com/en/1.8/ref/django-admin/#django-admin-loaddata

== Examples ==
Examples given in xml format for increased verbosity.
* custom.xml

=== Override default package for projects ===

<pre>
<?xml version="1.0" encoding="utf-8"?>
<django-objects version="1.0">

<object model="orm.toastersetting" pk="3">
<field type="CharField" name="name">DEFCONF_PACKAGE_CLASSES</field>
<field type="CharField" name="value">package_ipk</field>
</object>

</django-objects>
</pre>

=== Add a default layer from a git source ===
<pre>
<?xml version="1.0" encoding="utf-8"?>
<django-objects version="1.0">

<object model="orm.layer" pk="10">
<field type="CharField" name="name">meta-my-layer</field>
<field type="CharField" name="layer_index_url"></field>
<field type="CharField" name="vcs_url">git://git.example.com/my_repo</field>
</object>
<object model="orm.layer_version" pk="13">
<field rel="ManyToOneRel" to="orm.layer" name="layer">10</field>
<field type="IntegerField" name="layer_source">0</field>
<field rel="ManyToOneRel" to="orm.release" name="release">1</field>
<field type="CharField" name="branch">v1.2.3</field>
<field type="CharField" name="dirpath">layers/meta-my-layer</field>
</object>

<object model="orm.releasedefaultlayer" pk="15">
<field rel="ManyToOneRel" to="orm.release" name="release">3</field>
<field type="CharField" name="layer_name">meta-my-layer</field>
</object>
</django-objects>
</pre>

=== Add an available layer from a local file system source ===
<pre>
<?xml version="1.0" encoding="utf-8"?>
<django-objects version="1.0">

<object model="orm.layer" pk="20">
<field type="CharField" name="name">meta-my-layer</field>
<field type="CharField" name="layer_index_url"></field>
<field type="CharField" name="local_source_dir">/home/developer/work/meta-my-layer</field>
</object>
<object model="orm.layer_version" pk="13">
<field rel="ManyToOneRel" to="orm.layer" name="layer">20</field>
<field type="IntegerField" name="layer_source">0</field>
<field rel="ManyToOneRel" to="orm.release" name="release">1</field>
</object>

</django-objects>
</pre>

Contribute to Toaster

2016-11-22T16:50:54Z

Michael Wood:

[[Category:Toaster]]
This page summarises the Toaster development process. We hope this will help you start contributing to the project.

== What can I do? ==

The [https://bugzilla.yoctoproject.org/buglist.cgi?product=Toaster Yocto Project Bugzilla instance] lists all the things that need to be done:

* If the issue says <strong>GUI design available</strong> in the Whiteboard field, there is a design specification document attached to the issue that you should follow. Send questions / comments about it to the [https://lists.yoctoproject.org/listinfo/toaster Toaster mailing list]
* If the issue says <strong>GUI design pending</strong> in the Whiteboard field, there is some design work still to be done. Feel free to take the issue and send an email to the [https://lists.yoctoproject.org/listinfo/toaster Toaster mailing list] to find out why the design work is not done yet

== Set up the local repository ==

For development of Toaster we recommend setting up a local install of Toaster. Installation instructions are available in the main [http://www.yoctoproject.org/docs/latest/toaster-manual/toaster-manual.html#toaster-manual-start Toaster documentation]

== Submitting patches ==

Publishing your patches to Toaster is a two step process.
# Sending patches to the [https://lists.yoctoproject.org/listinfo/toaster/| Toaster mailing list] for review
# Submitting the patches that you reviewed to the upstream repository

By submitting your patches first to the Toaster mailing list, you can be sure the patches are reviewed by the people in the community who are familiar with the Toaster source code, and who have experience developing web applications.

That also means that, by the time the patches are submitted to the upstream mailing lists, they are in pretty good shape. That helps the project maintainers, and hopefully also helps you.

Toaster code lives in Bitbake repository at [http://git.openembedded.org/bitbake/|http://git.openembedded.org/bitbake/].
All contributions must be upstreamed to the Bitbake repository in order to make it to the "master" branch of the poky/ repository.

=== Workflow ===
We are now supporting a [http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/ poky-contrib] toaster-next branch. The purpose of this branch is to speed up our work so that we can base patches on top of patches that are waiting for upstream inclusion but have not yet made it into master. To facilitate this we have some extra rebasing actions needed.

To contribute to toaster you will also need authorization to write to the upstream yocto project repository. Contact a member of the toaster team for details.

1) Download master branch of the yocto project
<code> git clone git://git.yoctoproject.org/poky && cd poky </code>

2) Add poky-contrib to the local repository you set up above
<code> git remote add poky-contrib ssh://git@git.yoctoproject.org/poky-contrib </code>

3) Fetch the poky-contrib branches
<code> git fetch --all </code>

4) Start your feature branch off of toaster-next
<code> git checkout -b the/target/branch poky-contrib/toaster-next </code>

5) Do Work

6) Test the changes. Run the Django unit tests. People put effort into these so we should make sure we use them. This assumes you have phantomjs installed. This can usually be done from the distribution <code> apt-get install phantomjs</code>, for example. If you want to test against Chrome or Firefox, see the README in bitbake/lib/toaster/tests/browser.
<code> pip3 install selenium </code>
<code>TOASTER_TESTS_BROWSER=phantomjs bitbake/lib/toaster/manage.py test orm toastergui tests.browser </code>
<br>
Note: If you would like to run the tests in a container so they are repeatable and do not continually break due to host upgrades see [[Running Toaster Tests with Containers]]

7) Rebase on toaster-next. It has probably changed while you were working (unless you are really really fast!)
<code> git rebase poky-contrib/toaster-next </code>

8) Send to the toaster-mailing list using one of the methods outlined below.

=== Sending patches to Toaster Project ===

<strong>NOTE:</strong> The format of the commit message should be like this

<pre>
toaster: <module> <short one line summary>

long(er) description

[YOCTO #0000]

Signed-off-by: First Last <name@domain.com>
</pre>

Where YOCTO #0000 is the related bug number if there is one. Signed off by with your git commit -s credentials.

We accept patches on the [https://www.yoctoproject.org/tools-resources/community/mailing-lists toaster mailing list] ( toaster@yoctoproject.org ) by "git send-email".

e.g.
<code>
$ git send-email HEAD^
</code>

Alternatively, you can use the utilities in the script directory to prepare your patch

1) Use the create-pull-request script (from poky) to create a pull request while on your feature branch
<code>./scripts/create-pull-request -s "toaster: Fixes and clean ups" -u poky-contrib -r poky-contrib/toaster-next</code>

2) Review their content, especially the summary mail:
<code>edit ./pull-<pid>/0000-cover-letter.patch</code>

3)When you are satisfied, you can send them with:
<code>./scripts/send-pull-request -a -p ./pull-<pid></code> -t toaster@yoctoproject.org

A comprehensive document about commit messages is available on the [http://www.openembedded.org/wiki/Commit_Patch_Message_Guidelines openembedded wiki]

More help learning git is available on [https://try.github.io github] and [http://git-scm.com/documentation/ the official documentation]

=== Sending branches to Toaster Project ===

If you wish to submit whole branches please use the poky-contrib repository see [[Poky Contributions#Poky_Contrib_Branch]] for setup guide.

Once you have pushed a branch please then send an email to the [https://www.yoctoproject.org/tools-resources/community/mailing-lists toaster mailing list] with the subject in the following format:

<code>
[review-request] my_branch_name
</code>

In the body of the email it's useful to describe your branch's functionality, which commits and a link to the git web.

If you need any assistance please post on the mailing list.

=== Submitting patch sets for integration into Bitbake ===

Toaster patches are normally submitted upstream to the BitBake repository by the reviewer (not the author). This tells the upstream maintainers that the patches have been reviewed by the people who are familiar with the Toaster source code, and makes their busy lives a bit easier.

Since development happens on the poky-contrib repository, but the patches need to be merged to the Bitbake repository, the following process should be executed.

1) Bring toaster-next up to date with master
<code> git fetch poky-contrib toaster-next&& git fetch origin master</code>
<code> git checkout -b toaster-next poky-contrib/toaster-next </code>
<code> git rebase origin/master </code>

2) Checkout the target branch
<code>git checkout the/target/branch</code>

3) Create a new branch for submission
<code> git checkout -b yourname/submit/the/target/branch </code>

4) Make sure the branch is rebased on current poky-contrib toaster-next.
<code>git rebase poky-contrib/toaster-next</code>

5) Test the changes. Run the Django unit tests. People put effort into these so we should make sure we use them. his assumes you have phantomjs installed. This can usually be done from the distribution <code> apt-get install phantomjs</code>, for example. If you want to test against Chrome or Firefox, see the README in bitbake/lib/toaster/tests/browser.
<code> pip3 install selenium </code>
<code>TOASTER_TESTS_BROWSER=phantomjs bitbake/lib/toaster/manage.py test orm toastergui tests.browser </code>

6) Add signed off by to the commit messages
<code>git filter-branch -f --msg-filter 'cat && echo "Signed-off-by: $(git config --get user.name) <$(git config --get user.email)>"' toaster-next..HEAD</code>

7) Push the modified commit messages and rebased version to poky-contrib
<code>git push -u poky-contrib yourname/submit/the/target/branch </code>

8) Use the create-pull-request script (from poky) to create a pull request
<code>./scripts/create-pull-request -d bitbake -s "toaster: Fixes and clean ups" -u poky-contrib -r poky-contrib/toaster-next</code>
<br>
<strong>Note: If the patch creates any NEW files, the integration scripts that pull it into bitbake will fail. So, if new files are created as part of this patch set, you need to explicitly point that out in the body of the email for the patch set or do it as a PR rather than as a patch set.
</strong>

9) Review their content, especially the summary mail:
<code>edit ./pull-<pid>/0000-cover-letter.patch</code>

10) Push the branch you just signed off on and sent upstream to toaster-next
<code> git push -f -u poky-contrib yourname/submit/the/target/branch:toaster-next </code>

When you are satisfied, you can send them with:
<code>./scripts/send-pull-request -a -p ./pull-<pid></code> -t bitbake-devel@lists.openembedded.org

==== Submitting patches for prior releases ====

The procedure is the same, but using the prior release as the base branch instead of the "master" branch in bitbake.

Also, make sure that you add the name of the prior release for which the patchset is intended in the prefix of the patchset, as parameter to the "create-pull-request" command, e.g. '''-p 1.26''' for the 1.26 branch.

==== Gotchas ====
===== Too Big =====
Sometimes the mailer will refuse to send patches, especially on binary or long-line files. The proper way to go around that is to reply to the patchset you've submitted to the mailing list, asking for a git pull directly from the poky-contrib branch.
===== Different Lists =====

The toasterconf.json files live in the meta and meta-yocto layer. Patches to the meta-yocto/conf/toasterconf.json go to poky@yoctoproject.org. Patches to meta/conf/toasterjson.conf go to openembedded-core@lists.openembedded.org. Patches for toaster.bbclass also go to the openembedded--core list. If you have a branch that spans both (let's say you made coupled changes to meta/classes/toaster.bbclass and bitbake/lib/bb/ui/buildinfohelper.py. then your branch needs to be split into 2 different patch set submissions. Let's assume there are 5 commits, 2 to meta and 3 to bitbake. Currently, you can do this suboptimal workaround:
<code>
./scripts/create-pull-request -d bitbake -s "toaster: Fixes and clean ups" -u poky-contrib -r poky-contrib/toaster-next
./scripts/create-pull-request -d meta -s "toaster: Fixes and clean ups" -u poky-contrib -r poky-contrib/toaster-next
</code>
This will create 2 pull-XXXX directories, one for openembedded and one for bitbake-devel. Unfortunately, the meta pull-XXXX dir will have patches numbered 1-5 with 3 0 byte files and the bitbake pull-XXXX dir will have patches numbered 1-5 with 2 0 byte files. You delete the 0 byte files and use a text editor to do the renumbering. At some point it would be nice to have the create-pull-request do this for us but this level of file modification would probably mean moving it out of a shell script and into python. Once the files have been renumbered (in the subject line and the cover letter) the send-pull-request can be used as referenced above.
<br><p>
A different workaround that may work better for you is to create two branches which lets you use the create-pull-request and send-pull-request scripts normally on each branch. Just make sure to target them correctly
</p>
<code>
git checkout -b yourname/submit/the/target/branch-ForBitbake yourname/submit/the/target/branch
git rebase -i
# delete all the commits that are OE specific in the set you wish to submit
git push -u poky-contrib yourname/submit/the/target/branchForBitbake:yourname/submit/the/target/branchForBitbake
./scripts/create-pull-request -d bitbake -s "toaster: Fixes and clean ups" -u poky-contrib -r poky-contrib/toaster-next
git checkout -b yourname/submit/the/target/branch-ForOE yourname/submit/the/target/branch
git rebase -i
# delete all the commits that are bitbake specific in the set you wish to submit
git push -u poky-contrib yourname/submit/the/target/branchForOE:yourname/submit/the/target/branchForOE
./scripts/create-pull-request -d meta -s "toaster: Fixes and clean ups" -u poky-contrib -r poky-contrib/toaster-next
</code>

===== One Patch of a Long Patch Set Needs Resubmission =====
Suppose you upstream a 10 commit patch set to the bitbake-devel list and someone finds an issue with patch #3. Regenerating the whole series is silly so how do you address this? First, follow the bitbake submission steps until you end up on the yourname/submit/the/target/branch branch. Then you can (note <strong>the reset --hard will wipe any local changes in your working dir so commit or stash first</strong>):<br>
<code>
git checkout -b yourname/submit/the/target/branch-newHEAD
git reset --hard <commit of resubmission issue>
git commit --amend --signoff
git send-email --in-reply-to="longNumber.git.me@mycomp.com" --subject-prefix="bitbake-devel] [PATCHVX 03/10" --to=bitbake-devel@lists.openembedded.org --no-chain-reply-to --suppress-cc=all -M -1 --relative=bitbake
</code><br>
The longNumber.git.me@mycomp.com comes from the email message id for the particular patch #3 so that the email threading works. In gmail, you can click on the dropdown button on the right side of the screen and choose "Show Original". This will have a field in the header like Message-Id: <4551b56f132497c055f39567946a5d3be347d770.1468363530.git.myemailusername@mycompany.com> The entire string except the '<>' are used. for example:<br>
<code>
--in-reply-to="4551b56f132497c055f39567946a5d3be347d770.1468363530.git.myemailusername@mycompany.com"
</code><br>

Don't forget to switch back to the yourname/submit/the/target/branch and push that to poky-contrib/toaster-next! This is a little harder since you will have previously pushed a different version of the patch to poky-contrib/toaster-next. You can do a rebase/merge by hand but that can be prone to error. Here's the working sequence assuming it's been a few days between the first poky-contrib/toaster-next push and the one you are working on now (meaning both poky-contrib/toaster-next and origin/master have changes).<br>
The first issue you are likely to face is that the git filter-branch command in the standard instructions may sign off on too many commits. If you know you just want to sign off on the last 7 commits on the yourname/submit/the/target/branch you can:
<code>
git filter-branch -f --msg-filter 'cat && echo "Signed-off-by: $(git config --get user.name) <$(git config --get user.email)>"' HEAD~7..HEAD
</code>
Once your signoff is copacetic, updating poky-contrib/toaster-next can be done as follows:
<code>
git fetch poky-contrib toaster-next&& git fetch origin master
git checkout -b toaster-next poky-contrib/toaster-next
git rebase origin/master
git checkout yourname/submit/the/target/branch
git rebase --strategy recursive -X theirs -X patience toaster-next
git push -fu poky-contrib poky-contrib yourname/submit/the/target/branch:toaster-next
</code>

=== Submitting patches for documentation ===

Documentation patches should be sent to [https://lists.yoctoproject.org/listinfo/yocto Yocto mailing list] with [yocto-docs] in the subject, CC Scott Rifenbark (and make sure you send it to his gmail, not his defunct Intel address). For his email address, look at [http://lists.openembedded.org/pipermail/bitbake-devel/2015-October/006632.html this post].

== Code syle guide ==

=== Templates ===

Django has a template language which allows us to render pages based on the data (context). We use the template language to setup the initial state of the page and to create re-usable components that can be included in other pages.

The recommend template code style is as follows

'''Yes please:'''

<pre>
{{var}}

<div>
{# Maintaining indentation #}
{% if %}
<p>this</p>
{% else %}
<p>that</p>
{% endif %}
</div>

{% comment %}
This is a longer comment that describes all the things
that are below in quite a bit of detail because they're
a little more difficult to understand.
{% endcomment %}

{% for layer in layers_list %}
{{layer}}
{% endfor %}

</pre>

'''No thank you:'''
<pre>
{{var}}
<div>
{# Maintaining indentation #}
{%if%}<p>this</p>{%else%}<p>that</p>{%endif%}
</div>

{#This is a longer comment that describes all the things that are below in quite a bit of detail because they're a little more difficult to understand. #}
{%for o in layers_list%}{{o}}{%endfor%}
</pre>

Note:
* Maintain indentation as you would with other languages
* White space after '%'
* Comment blocks for longer comments

=== Javascript ===

'''Yes please:'''
<pre>

"use strict";

/* These hold some numbers */
var oneVar = 1;
var twoVar = 2;

var cheesesTypes = {
cheddar : 1,
stilton : 2,
emmental : 3,
};

function doThingsHere(){
return 1;
}

/* If one equals two do some other things and make sure that
* if the the click handler is setup correctly.
*/
if (one === two) {
var cheese = "cheddar";
oneVar = doThingsHere();

$(this).click(function (event){
alert("Hello");
});
}

$("#little-mouse").focusout(function(){
alert("bye")
});

if (oneVar)
noThingHere();
else
doThingHere();
</pre>

'''No thank you:'''
<pre>
// These hold some numbers
oneVar = 1
twoVar = 2

var cheesesTypes = { cheddar : 1, stilton : 2, emmental : 3, }

function doThingsHere ()
{
return 1;
}

//If one equals two do some other things and make sure that if the the click handler is setup correctly.
if( one === two ) {
var cheese = "cheddar";
oneVar = doThingsHere();

$(this).click(function(event){ alert("Hello"); });
}

document.getElementById("little-mouse").addEventListener("focusout", function(){
alert("bye")
});

if (oneVar)
{
noThingHere();
} else { doThingHere(); }
</pre>

Note:
* Variables should be marked with "var"
* Semicolons should be used
* Keep as close to 80 cols as possible
* Use 2 space per indentation
* Open curly braces after parenthesis for functions and close on a new line
* Use camelCase for function names and variable names

Make use of running your Javascript through jshint we have a .jshint configuration file in that js directory (toastergui/static/js)

e.g. install jshint and add to your current PATH, then run:
<code>
$ npm install jshint; export PATH=$PATH:$PWD/node_modules/.bin/
$ jshint ./toastergui/static/js/base.js
</code>

=== HTML ===

'''Yes please:'''
<pre>
<div id="something-area">
<p class="important">This is some text</p>
</div>

<div>
<p id="important-text>This is some text</p>
</div>

</pre>

'''No thank you:'''
<pre>
<div id="somethingarea">
<p class="Important">This is some text</p></div>
<div id="somethingarea">
<p id="ImportantText">This is
some text
</p>

</div>
</pre>

Note:
* 2 space indentation
* Lower case, ids hyphenated when multiple words
* No duplicate ids

* Run your HTML through a [http://validator.w3.org/#validate_by_input HTML validator] available for [http://validator.w3.org/source/ local install]. The w3c validator it's self doesn't currently validate html5, it uses as a back end [https://validator.github.io/validator/ Nu Html Checker] which can be installed as a standalone service, full instructions in the readme.

Quick install instructions:
<pre>
$ mkdir html5-validator && cd html5-validator
$ export JAVA_HOME=/usr/lib/jvm/java-6-openjdk
$ git clone https://github.com/validator/validator.git
$ python build/build.py all
$ python build/build.py all
</pre>

HTML can be indented quickly using tidy, for example:
<pre>
tidy -xml --indent auto --indent-spaces 2 --quiet yes -w -1 --show-body-only yes ./index.html
</pre>

=== Python ===

Lenient [https://www.python.org/dev/peps/pep-0008 pep8]
Ignoring most of the whitespace around character issues (E124,E203,E201,E265,E303,E302,E231) see toaster/.pep8 and [http://pep8.readthedocs.org/en/latest/intro.html#error-codes error code list]

Fix all issues identified by running code through pep8. We have a fairly lenient config file (toaster/.pep8).

e.g.
<code>
$ pep8 ./toastergui/urls.py
</code>

Run code through pylint and fix identified issues - Some can be reasonably ignored such as doc strings for every function or star-args. No pylintrc config provided here as most issues identified are highly contextual and should be ignored on a case by case basis.

<code>
$ pylint --load-plugins pylint_django toastergui/tests.py
</code>

== Working with design ==

Yes, the Yocto Project is one of those lucky projects with designers around to help in UI matters. We have a document explaining how to work with the design contributors: [[File:Working_with_design.pdf]]

Setting up a production instance of Toaster

2016-08-04T17:55:27Z

Michael Wood:

[[Category:Toaster]]
----
{| style="color:black; background-color:#ffffcc" width="100%" cellpadding="10" class="wikitable"
|This page is the development version of the documentation to provide the latest information, if you're using a release please refer to [https://www.yoctoproject.org/documentation/archived the published manual]'''
|}
----
A production instance of Toaster is one in which you wish to share the Toaster instance with remote and multiple users. It is also the setup which can cope with heavier loads on the web service. These instructions setup toaster in Build mode where builds and projects are run, viewed and defined by the Toaster web interface.

== Requirements ==
* [http://www.yoctoproject.org/docs/2.0/yocto-project-qs/yocto-project-qs.html#packages Build requirements]
* Apache webserver
* mod-wsgi for Apache webserver
* Mysql database server

Ubuntu 14.04.3:
<code>
$ sudo apt-get install apache2 libapache2-mod-wsgi mysql-server python-virtualenv libmysqlclient-dev python-dev python-mysqldb
</code>

Fedora 22/RH:
<code>
$ sudo dnf install httpd mod_wsgi python-virtualenv gcc mysql-devel
</code>

== Installation ==

'''1.''' Checkout a copy of Poky into the web server directory. We're going to be using /var/www/toaster.
<code>
$ mkdir -p /var/www/toaster
$ cd /var/www/toaster/
$ git clone git://git.yoctoproject.org/poky
$ cd poky
$ git checkout jethro # change for any release name required
</code>

'''2.''' Initialise a virtualenv and install Toaster dependencies. (Use virtualenv to keep the python packages isolated from your system provided packages - not required but recommended, alternative use your OS's package manager to install the packages)

<code>
$ cd /var/www/toaster/
$ virtualenv venv
$ source ./venv/bin/activate
$ pip install -r ./poky/bitbake/toaster-requirements.txt
$ pip install mysqlclient
</code>

'''3.''' Configure toaster edit /var/www/toaster/poky/bitbake/lib/toaster/toastermain/settings.py

Edit the DATABASE settings:
<code>
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.mysql',
'NAME': 'toaster_data',
'USER': 'toaster',
'PASSWORD': 'yourpasswordhere',
'HOST': 'localhost',
'PORT': '3306',
}
</code>

Edit the [https://docs.djangoproject.com/en/1.8/howto/deployment/checklist/#secret-key SECRET_KEY]:
<code>
SECRET_KEY = 'YOUR SECRET RANDOM KEY HERE'
</code>

Edit the STATIC_ROOT:
<code>
STATIC_ROOT = '/var/www/toaster/static_files/'
</code>

'''4.'''' Now add the database and user to your mysql server that we just defined
<code>
$ mysql -u root -p
mysql> CREATE DATABASE toaster_data;
mysql> CREATE USER 'toaster'@'localhost' identified by 'yourpasswordhere';
mysql> GRANT all on toaster_data.* to 'toaster'@'localhost';
mysql> quit
</code>
n.b. You may want to decide on fewer [https://dev.mysql.com/doc/refman/5.1/en/grant.html privileges] to the toaster user.

'''5.''' Get toaster to create the database schema, default data, update the TOASTER_DIR which is the build work dir and collect up the statically served files

<code>
$ cd /var/www/toaster/poky/
$ ./bitbake/lib/toaster/manage.py migrate
$ ./bitbake/lib/toaster/manage.py loadconf ./meta-yocto/conf/toasterconf.json
$ TOASTER_DIR=/var/www/toaster/poky/ ./bitbake/lib/toaster/manage.py checksettings
$ ./bitbake/lib/toaster/manage.py lsupdates
$ ./bitbake/lib/toaster/manage.py collectstatic
</code>

'''6.''' Add a config file for Toaster to your Apache web server's configurations available directory.

Ubuntu/Debian put it here: /etc/apache2/conf-available/toaster.conf
Fedora/RH usually here: /etc/httpd/conf.d/toaster.conf
<code>

Alias /static /var/www/toaster/static_files
<Directory /var/www/toaster/static_files>
Order allow,deny
Allow from all
Require all granted
</Directory>
WSGIDaemonProcess toaster_wsgi python-path=/var/www/toaster/poky/bitbake/lib/toaster:/var/www/toaster/venv/lib/python2.7/site-packages
WSGIScriptAlias / "/var/www/toaster/poky/bitbake/lib/toaster/toastermain/wsgi.py"
<Location />
WSGIProcessGroup toaster_wsgi
</Location>

</code>

In Ubuntu/Debain you will need to enable the config and module in Apache webserver
<code>
$ sudo a2enmod wsgi
$ sudo a2enconf toaster
</code>

Restart Apache web server to make sure all new configuration is loaded

Ubuntu/Debian:
<code>
$ sudo service apache2 restart
</code>

Fedora/RH:
<code>
$ sudo service httpd restart
</code>

'''7.''' Install the build runner service

This service needs to be running in order to dispatch builds the command that needs to be run is:

<code>
$ source oe-init-build-env
$ source toaster start noweb
</code>

Example upstart service /etc/init/toaster-buildservice.conf :
<pre>
author "Michael W"
description "start and stop toaster build service"
version "1.0"

start on started networking
stop on runlevel [!2345]

respawn

script
exec su toasterbuilder -c "cd /var/www/toaster/poky && source oe-init-build-env && source toaster start noweb"
end script
</pre>

Example systemd service usually in /lib/systemd/system/toaster.service:
<pre>
[Unit]
Description=Toaster runbuilds

[Service]
Type=forking
User=toaster
ExecStart=/var/www/toaster/toaster-service.sh start
ExecStop=/var/www/toaster/toaster-service.sh stop

[Install]
WantedBy=multi-user.target
</pre>
script /var/www/toaster/toaster-service.sh:
<pre>

#!/bin/bash

cd /var/www/toaster/poky/

source oe-init-build-env
if [ "$1" == 'start' ]; then
source ../bitbake/bin/toaster start noweb
fi

if [ "$1" == 'stop' ]; then
source ../bitbake/bin/toaster stop
fi
</pre>

N.b. You may wish to add a service entry to your OS's init system so that it starts up on start up as well as adding a dedicated user.

Now open up a browser and you can start using Toaster!

Setting up a production instance of Toaster

2016-08-04T15:14:06Z

Michael Wood: Add systemd example

[[Category:Toaster]]
----
{| style="color:black; background-color:#ffffcc" width="100%" cellpadding="10" class="wikitable"
|This page is the development version of the documentation to provide the latest information, if you're using a release please refer to [https://www.yoctoproject.org/documentation/archived the published manual]'''
|}
----
A production instance of Toaster is one in which you wish to share the Toaster instance with remote and multiple users. It is also the setup which can cope with heavier loads on the web service. These instructions setup toaster in Build mode where builds and projects are run, viewed and defined by the Toaster web interface.

== Requirements ==
* [http://www.yoctoproject.org/docs/2.0/yocto-project-qs/yocto-project-qs.html#packages Build requirements]
* Apache webserver
* mod-wsgi for Apache webserver
* Mysql database server

Ubuntu 14.04.3:
<code>
$ sudo apt-get install apache2 libapache2-mod-wsgi mysql-server python-virtualenv libmysqlclient-dev python-dev python-mysqldb
</code>

Fedora 22/RH:
<code>
$ sudo dnf install httpd mod_wsgi python-virtualenv gcc mysql-devel
</code>

== Installation ==

'''1.''' Checkout a copy of Poky into the web server directory. We're going to be using /var/www/toaster.
<code>
$ mkdir -p /var/www/toaster
$ cd /var/www/toaster/
$ git clone git://git.yoctoproject.org/poky
$ cd poky
$ git checkout jethro # change for any release name required
</code>

'''2.''' Initialise a virtualenv and install Toaster dependencies. (Use virtualenv to keep the python packages isolated from your system provided packages - not required but recommended, alternative use your OS's package manager to install the packages)

<code>
$ cd /var/www/toaster/
$ virtualenv venv
$ source ./venv/bin/activate
$ pip install -r ./poky/bitbake/toaster-requirements.txt
$ pip install mysqlclient
</code>

'''3.''' Configure toaster edit /var/www/toaster/poky/bitbake/lib/toaster/toastermain/settings.py

Edit the DATABASE settings:
<code>
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.mysql',
'NAME': 'toaster_data',
'USER': 'toaster',
'PASSWORD': 'yourpasswordhere',
'HOST': 'localhost',
'PORT': '3306',
}
</code>

Edit the [https://docs.djangoproject.com/en/1.8/howto/deployment/checklist/#secret-key SECRET_KEY]:
<code>
SECRET_KEY = 'YOUR SECRET RANDOM KEY HERE'
</code>

Edit the STATIC_ROOT:
<code>
STATIC_ROOT = '/var/www/toaster/static_files/'
</code>

'''4.'''' Now add the database and user to your mysql server that we just defined
<code>
$ mysql -u root -p
mysql> CREATE DATABASE toaster_data;
mysql> CREATE USER 'toaster'@'localhost' identified by 'yourpasswordhere';
mysql> GRANT all on toaster_data.* to 'toaster'@'localhost';
mysql> quit
</code>
n.b. You may want to decide on fewer [https://dev.mysql.com/doc/refman/5.1/en/grant.html privileges] to the toaster user.

'''5.''' Get toaster to create the database schema, default data, update the TOASTER_DIR which is the build work dir and collect up the statically served files

<code>
$ cd /var/www/toaster/poky/
$ ./bitbake/lib/toaster/manage.py migrate
$ ./bitbake/lib/toaster/manage.py loadconf ./meta-yocto/conf/toasterconf.json
$ TOASTER_DIR=/var/www/toaster/poky/ ./bitbake/lib/toaster/manage.py checksettings
$ ./bitbake/lib/toaster/manage.py lsupdates
$ ./bitbake/lib/toaster/manage.py collectstatic
</code>

'''6.''' Add a config file for Toaster to your Apache web server's configurations available directory.

Ubuntu/Debian put it here: /etc/apache2/conf-available/toaster.conf
Fedora/RH usually here: /etc/httpd/conf.d/toaster.conf
<code>

Alias /static /var/www/toaster/static_files
<Directory /var/www/toaster/static_files>
Order allow,deny
Allow from all
Require all granted
</Directory>
WSGIDaemonProcess toaster_wsgi python-path=/var/www/toaster/poky/bitbake/lib/toaster:/var/www/toaster/venv/lib/python2.7/site-packages
WSGIScriptAlias / "/var/www/toaster/poky/bitbake/lib/toaster/toastermain/wsgi.py"
<Location />
WSGIProcessGroup toaster_wsgi
</Location>

</code>

In Ubuntu/Debain you will need to enable the config and module in Apache webserver
<code>
$ sudo a2enmod wsgi
$ sudo a2enconf toaster
</code>

Restart Apache web server to make sure all new configuration is loaded

Ubuntu/Debian:
<code>
$ sudo service apache2 restart
</code>

Fedora/RH:
<code>
$ sudo service httpd restart
</code>

'''7.''' Install the build runner service

This service needs to be running in order to dispatch builds the command that needs to be run is:

<code>
$ source oe-init-build-env
$ source toaster start noweb
</code>

Example upstart service /etc/init/toaster-buildservice.conf :
<pre>
author "Michael W"
description "start and stop toaster build service"
version "1.0"

start on started networking
stop on runlevel [!2345]

respawn

script
exec su toasterbuilder -c "cd /var/www/toaster/poky && source oe-init-build-env && source toaster start noweb"
end script
</pre>

Example systemd service usually in /lib/systemd/system/toaster.service:
<pre>
[Unit]
Description=Toaster runbuilds

[Service]
Type=forking
User=toaster
ExecStart=cd /var/www/toaster/poky/ && source oe-init-build-env && source toaster noweb start
ExecStop=cd /var/www/toaster/poky/ && source oe-init-build-env && source toaster noweb stop
WorkingDirectory=/var/www/toaster/poky

[Install]
WantedBy=multi-user.target
</pre>

N.b. You may wish to add a service entry to your OS's init system so that it starts up on start up as well as adding a dedicated user.

Now open up a browser and you can start using Toaster!

Toaster travis testing

2016-07-06T15:08:56Z

Michael Wood:

When a patch comes in on the mailing list a bot monitoring patchwork using [http://patchwork-freedesktop.readthedocs.io/en/latest/testing.html#git-pw-helper-commands git-pw] checks out poky-contrib toaster-next and applies the patches it also applies a patch to add a .travis.yml file to the root of the repository. These then get pushed to a [https://github.com/toastertester/toaster-next github] repository and the travis CI system then triggers these tests.

.travis.yml
<pre>

language: python
python: "3.5"

sudo: required # We don't actually need sudo but this forces the selection of the trusty beta image
dist: trusty # We have to use trusty for the newer glibc that is needed by geckodriver

addons:
firefox: "47.0"

cache: pip

git:
depth: 3

install:
- virtualenv --python=python3 $HOME/virtualenv/
- virtualenv --python=python2 $HOME/virtualenv/
- source $HOME/virtualenv/bin/activate
- pip3 install -r ./bitbake/toaster-requirements.txt
- pip3 install -r ./bitbake/lib/toaster/tests/toaster-tests-requirements.txt
- pip3 install flake8
- export DISPLAY=:99.0
- sh -e /etc/init.d/xvfb start
- sleep 3 # give xvfb some time to start
- mkdir $HOME/bin/
- curl -L https://github.com/mozilla/geckodriver/releases/download/v0.8.0/geckodriver-0.8.0-linux64.gz | zcat > $HOME/bin/wires
- chmod +x $HOME/bin/wires

before_script:
- export PATH=$PATH:$HOME/bin/
- export TOASTER_TESTS_BROWSER=marionette
- firefox --version
- env
- wires --help # verify binary runs OK

script:
- ./bitbake/lib/toaster/manage.py test tests.browser
- ./bitbake/lib/toaster/manage.py test toastergui
- flake8 ./bitbake/lib/toaster/ --select F --exit-zero # We have too many errors at the moment
</pre>

Toaster travis testing

2016-07-06T14:59:58Z

Michael Wood: Created page with "When a patch comes in on the mailing list a bot monitoring patchwork using git-pw checks out poky-contrib toaster-next and applies the patches it also applies a patch to add a..."

When a patch comes in on the mailing list a bot monitoring patchwork using git-pw checks out poky-contrib toaster-next and applies the patches it also applies a patch to add a .travis.yml file to the root of the repository. These then get pushed to a github https://github.com/toastertester/toaster-next repository and the travis CI system then triggers these tests.

.travis.yml
<pre>

language: python
python: "3.5"

sudo: required # We don't actually need sudo but this forces the selection of the trusty beta image
dist: trusty # We have to use trusty for the newer glibc that is needed by geckodriver

addons:
firefox: "47.0"

cache: pip

git:
depth: 3

install:
- virtualenv --python=python3 $HOME/virtualenv/
- virtualenv --python=python2 $HOME/virtualenv/
- source $HOME/virtualenv/bin/activate
- pip3 install -r ./bitbake/toaster-requirements.txt
- pip3 install -r ./bitbake/lib/toaster/tests/toaster-tests-requirements.txt
- pip3 install flake8
- export DISPLAY=:99.0
- sh -e /etc/init.d/xvfb start
- sleep 3 # give xvfb some time to start
- mkdir $HOME/bin/
- curl -L https://github.com/mozilla/geckodriver/releases/download/v0.8.0/geckodriver-0.8.0-linux64.gz | zcat > $HOME/bin/wires
- chmod +x $HOME/bin/wires

before_script:
- export PATH=$PATH:$HOME/bin/
- export TOASTER_TESTS_BROWSER=marionette
- firefox --version
- env
- wires --help # verify binary runs OK

script:
- ./bitbake/lib/toaster/manage.py test tests.browser
- ./bitbake/lib/toaster/manage.py test toastergui
- flake8 ./bitbake/lib/toaster/ --select F --exit-zero # We have too many errors at the moment
</pre>

Setting up a production instance of Toaster

2016-06-28T16:04:08Z

Michael Wood:

[[Category:Toaster]]
----
{| style="color:black; background-color:#ffffcc" width="100%" cellpadding="10" class="wikitable"
|This page is the development version of the documentation to provide the latest information, if you're using a release please refer to [https://www.yoctoproject.org/documentation/archived the published manual]'''
|}
----
A production instance of Toaster is one in which you wish to share the Toaster instance with remote and multiple users. It is also the setup which can cope with heavier loads on the web service. These instructions setup toaster in Build mode where builds and projects are run, viewed and defined by the Toaster web interface.

== Requirements ==
* [http://www.yoctoproject.org/docs/2.0/yocto-project-qs/yocto-project-qs.html#packages Build requirements]
* Apache webserver
* mod-wsgi for Apache webserver
* Mysql database server

Ubuntu 14.04.3:
<code>
$ sudo apt-get install apache2 libapache2-mod-wsgi mysql-server python-virtualenv libmysqlclient-dev python-dev python-mysqldb
</code>

Fedora 22/RH:
<code>
$ sudo dnf install httpd mod_wsgi python-virtualenv gcc mysql-devel
</code>

== Installation ==

'''1.''' Checkout a copy of Poky into the web server directory. We're going to be using /var/www/toaster.
<code>
$ mkdir -p /var/www/toaster
$ cd /var/www/toaster/
$ git clone git://git.yoctoproject.org/poky
$ cd poky
$ git checkout jethro # change for any release name required
</code>

'''2.''' Initialise a virtualenv and install Toaster dependencies. (Use virtualenv to keep the python packages isolated from your system provided packages - not required but recommended, alternative use your OS's package manager to install the packages)

<code>
$ cd /var/www/toaster/
$ virtualenv venv
$ source ./venv/bin/activate
$ pip install -r ./poky/bitbake/toaster-requirements.txt
$ pip install mysqlclient
</code>

'''3.''' Configure toaster edit /var/www/toaster/poky/bitbake/lib/toaster/toastermain/settings.py

Edit the DATABASE settings:
<code>
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.mysql',
'NAME': 'toaster_data',
'USER': 'toaster',
'PASSWORD': 'yourpasswordhere',
'HOST': 'localhost',
'PORT': '3306',
}
</code>

Edit the [https://docs.djangoproject.com/en/1.8/howto/deployment/checklist/#secret-key SECRET_KEY]:
<code>
SECRET_KEY = 'YOUR SECRET RANDOM KEY HERE'
</code>

Edit the STATIC_ROOT:
<code>
STATIC_ROOT = '/var/www/toaster/static_files/'
</code>

'''4.'''' Now add the database and user to your mysql server that we just defined
<code>
$ mysql -u root -p
mysql> CREATE DATABASE toaster_data;
mysql> CREATE USER 'toaster'@'localhost' identified by 'yourpasswordhere';
mysql> GRANT all on toaster_data.* to 'toaster'@'localhost';
mysql> quit
</code>
n.b. You may want to decide on fewer [https://dev.mysql.com/doc/refman/5.1/en/grant.html privileges] to the toaster user.

'''5.''' Get toaster to create the database schema, default data, update the TOASTER_DIR which is the build work dir and collect up the statically served files

<code>
$ cd /var/www/toaster/poky/
$ ./bitbake/lib/toaster/manage.py migrate
$ ./bitbake/lib/toaster/manage.py loadconf ./meta-yocto/conf/toasterconf.json
$ TOASTER_DIR=/var/www/toaster/poky/ ./bitbake/lib/toaster/manage.py checksettings
$ ./bitbake/lib/toaster/manage.py lsupdates
$ ./bitbake/lib/toaster/manage.py collectstatic
</code>

'''6.''' Add a config file for Toaster to your Apache web server's configurations available directory.

Ubuntu/Debian put it here: /etc/apache2/conf-available/toaster.conf
Fedora/RH usually here: /etc/httpd/conf.d/toaster.conf
<code>

Alias /static /var/www/toaster/static_files
<Directory /var/www/toaster/static_files>
Order allow,deny
Allow from all
Require all granted
</Directory>
WSGIDaemonProcess toaster_wsgi python-path=/var/www/toaster/poky/bitbake/lib/toaster:/var/www/toaster/venv/lib/python2.7/site-packages
WSGIScriptAlias / "/var/www/toaster/poky/bitbake/lib/toaster/toastermain/wsgi.py"
<Location />
WSGIProcessGroup toaster_wsgi
</Location>

</code>

In Ubuntu/Debain you will need to enable the config and module in Apache webserver
<code>
$ sudo a2enmod wsgi
$ sudo a2enconf toaster
</code>

Restart Apache web server to make sure all new configuration is loaded

Ubuntu/Debian:
<code>
$ sudo service apache2 restart
</code>

Fedora/RH:
<code>
$ sudo service httpd restart
</code>

'''7.''' Install the build runner service

This service needs to be running in order to dispatch builds the command that needs to be run is:

<code>
$ source oe-init-build-env
$ source toaster noweb start
</code>

Example upstart service /etc/init/toaster-buildservice.conf :
<pre>
author "Michael W"
description "start and stop toaster build service"
version "1.0"

start on started networking
stop on runlevel [!2345]

respawn

script
exec su toasterbuilder -c "cd /var/www/toaster/poky && source oe-init-build-env && source toaster noweb start"
end script
</pre>

N.b. You may wish to add a service entry to your OS's init system so that it starts up on start up as well as adding a dedicated user.

Now open up a browser and you can start using Toaster!

Setting up a production instance of Toaster

2016-06-28T16:02:15Z

Michael Wood:

[[Category:Toaster]]
----
{| style="color:black; background-color:#ffffcc" width="100%" cellpadding="10" class="wikitable"
|This page is the development version of the documentation to provide the latest information, if you're using a release please refer to [https://www.yoctoproject.org/documentation/archived the published manual]'''
|}
----
A production instance of Toaster is one in which you wish to share the Toaster instance with remote and multiple users. It is also the setup which can cope with heavier loads on the web service. These instructions setup toaster in Build mode where builds and projects are run, viewed and defined by the Toaster web interface.

== Requirements ==
* [http://www.yoctoproject.org/docs/2.0/yocto-project-qs/yocto-project-qs.html#packages Build requirements]
* Apache webserver
* mod-wsgi for Apache webserver
* Mysql database server

Ubuntu 14.04.3:
<code>
$ sudo apt-get install apache2 libapache2-mod-wsgi mysql-server python-virtualenv libmysqlclient-dev python-dev python-mysqldb
</code>

Fedora 22/RH:
<code>
$ sudo dnf install httpd mod_wsgi python-virtualenv gcc mysql-devel
</code>

== Installation ==

'''1.''' Checkout a copy of Poky into the web server directory. We're going to be using /var/www/toaster.
<code>
$ mkdir -p /var/www/toaster
$ cd /var/www/toaster/
$ git clone git://git.yoctoproject.org/poky
$ cd poky
$ git checkout jethro # change for any release name required
</code>

'''2.''' Initialise a virtualenv and install Toaster dependencies. (Use virtualenv to keep the python packages isolated from your system provided packages - not required but recommended, alternative use your OS's package manager to install the packages)

<code>
$ cd /var/www/toaster/
$ virtualenv venv
$ source ./venv/bin/activate
$ pip install -r ./poky/bitbake/toaster-requirements.txt
$ pip install mysqlclient
</code>

'''3.''' Configure toaster edit /var/www/toaster/poky/bitbake/lib/toaster/toastermain/settings.py

Edit the DATABASE settings:
<code>
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.mysql',
'NAME': 'toaster_data',
'USER': 'toaster',
'PASSWORD': 'yourpasswordhere',
'HOST': 'localhost',
'PORT': '3306',
}
</code>

Edit the [https://docs.djangoproject.com/en/1.8/howto/deployment/checklist/#secret-key SECRET_KEY]:
<code>
SECRET_KEY = 'YOUR SECRET RANDOM KEY HERE'
</code>

Edit the STATIC_ROOT:
<code>
STATIC_ROOT = '/var/www/toaster/static_files/'
</code>

'''4.'''' Now add the database and user to your mysql server that we just defined
<code>
$ mysql -u root -p
mysql> CREATE DATABASE toaster_data;
mysql> CREATE USER 'toaster'@'localhost' identified by 'yourpasswordhere';
mysql> GRANT all on toaster_data.* to 'toaster'@'localhost';
mysql> quit
</code>
n.b. You may want to decide on fewer [https://dev.mysql.com/doc/refman/5.1/en/grant.html privileges] to the toaster user.

'''5.''' Get toaster to create the database schema, default data, update the TOASTER_DIR which is the build work dir and collect up the statically served files

<code>
$ cd /var/www/toaster/poky/
$ ./bitbake/lib/toaster/manage.py migrate
$ ./bitbake/lib/toaster/manage.py loadconf ./meta-yocto/conf/toasterconf.json
$ TOASTER_DIR=/var/www/toaster/poky/ ./bitbake/lib/toaster/manage.py checksettings
$ ./bitbake/lib/toaster/manage.py lsupdates
$ ./bitbake/lib/toaster/manage.py collectstatic
</code>

'''6.''' Add a config file for Toaster to your Apache web server's configurations available directory.

Ubuntu/Debian put it here: /etc/apache2/conf-available/toaster.conf
Fedora/RH usually here: /etc/httpd/conf.d/toaster.conf
<code>

Alias /static /var/www/toaster/static_files
<Directory /var/www/toaster/static_files>
Order allow,deny
Allow from all
Require all granted
</Directory>
WSGIDaemonProcess toaster_wsgi python-path=/var/www/toaster/poky/bitbake/lib/toaster:/var/www/toaster/venv/lib/python2.7/site-packages
WSGIScriptAlias / "/var/www/toaster/poky/bitbake/lib/toaster/toastermain/wsgi.py"
<Location />
WSGIProcessGroup toaster_wsgi
</Location>

</code>

In Ubuntu/Debain you will need to enable the config and module in Apache webserver
<code>
$ sudo a2enmod wsgi
$ sudo a2enconf toaster
</code>

Restart Apache web server to make sure all new configuration is loaded

Ubuntu/Debian:
<code>
$ sudo service apache2 restart
</code>

Fedora/RH:
<code>
$ sudo service httpd restart
</code>

'''7.''' Install the build runner service

This service needs to be running in order to dispatch builds the command that needs to be run is:

<code>
$ source oe-init-build-env
$ source toaster noweb start
</code>

Example upstart service /etc/init/toaster-buildservice.conf :
<code>

author "Michael W"
description "start and stop toaster build service"
version "1.0"

start on started networking
stop on runlevel [!2345]

respawn

script
exec su toasterbuilder -c "cd /var/www/toaster/poky && source oe-init-build-env && source toaster noweb start"
end script

</code>

N.b. You may wish to add a service entry to your OS's init system so that it starts up on start up as well as adding a dedicated user.

Now open up a browser and you can start using Toaster!

ToasterTable

2016-05-17T17:15:19Z

Michael Wood:

[[Category:Toaster]]
== Introduction to ToasterTable ==

ToasterTable is a set of classes in the toastergui module of the Toaster Django application. It is used to present a set of database records in a style and with functionality consistent with the rest of Toaster.

The functionality provided by ToasterTable includes:
* Sorting
* Column hiding/showing
* Filtering
* Paging
* Search

When a ToasterTable is updated, a new set of records matching the filter criteria, search string, sort order and page number is fetched from the back-end using Ajax requests. The ToasterTable is then updated in place from the JSON in the response, using JavaScript to redraw the table rows.

As a developer, you don't need to worry (necessarily) about how this works: you just implement a subclass of ToasterTable, specify a template for rendering it, and supply a URL mapping.

The following sections explain how to use ToasterTable. The filenames in these sections refer to files in the toastergui/ directory.

Note that to be able to follow the example, you will need to set up Toaster for development work. This is described in [http://www.yoctoproject.org/docs/2.0.1/toaster-manual/toaster-manual.html the Toaster manual].

== How to create a ToasterTable ==

Most of the tables needed for Toaster are already implemented. Many of the key tables in Toaster already use ToasterTable, but some don't. A full list of unconverted tables is in https://bugzilla.yoctoproject.org/show_bug.cgi?id=8363.

This means that you will typically be porting an existing table to ToasterTable, rather than adding a new table from scratch. However, to keep things simple, this section explains how to create a ToasterTable from scratch. Knowing how to do this should make it easier to port an existing table to ToasterTable in future.

To provide a worked example, I'll create a new table which is a variant of the existing "all builds" table, using a reduced number of columns and filters.

=== Create the table class ===

A quick overview of where things are and where they go when creating a table:

* widgets.py contains the ToasterTable class. This is the base class for all ToasterTables.
* New tables are added to tables.py.
* A table in tables.py maps to a URL within the Toaster application; the mapping is defined in urls.py.
* Templates for the table are added to templates/.
* If you need [https://docs.djangoproject.com/en/1.9/howto/custom-template-tags/ custom template tags], these go in templatetags/.

To create a new table, first add the class definition to tables.py:

<pre>
# import any models you need for the table
from orm.models import Build

class MiniBuildsTable(ToasterTable):
def __init__(self, *args, **kwargs):
super(MiniBuildsTable, self).__init__(*args, **kwargs)
self.default_orderby = '-completed_on'
self.title = 'Mini Builds Table'

def setup_queryset(self, *args, **kwargs):
self.queryset = Build.objects.all()

def setup_columns(self, *args, **kwargs):
self.add_column(title='Completed on',
help_text='The date and time when the build finished',
hideable=False,
orderable=True,
static_data_name='completed_on',
static_data_template='{{data.completed_on | date:"d/m/y H:i"}}')
</pre>

The key to creating a ToasterTable is to decide which data is going to be shown in the table and define this in the ''setup_queryset()'' method. This will typically be a queryset derived from one model in the Toaster application. To see the available models, check the orm/models.py file.

Each row in the table corresponds to a record in the queryset. The ''setup_queryset()'' method specifies how to get this queryset for the table: it must set the self.queryset property to the Django queryset containing the data. In the MiniBuildsTable, we show all of the builds by default (using the standard Django model API).

Each column in the ToasterTable corresponds to a field in the queryset. Inside a table row, a cell corresponds to one or more fields from each record in the queryset. A cell can show various aspects of a record: a formatted version of a field's value, an amalgam of multiple fields, a computation based on a group of related records etc. For example, in a row of the MiniBuildsTable, a cell might contain the outcome of the build, the number of tasks which failed during the build, or the time since the build completed.

The ''setup_columns()'' method defines which columns should be shown in the table. In this initial version of MiniBuildsTable, only the "completed_on" column is shown. This column has the following properties:
* It cannot be hidden (hideable=False).
* It can be used to order the table (orderable=True).
* static_data_name sets the name of the column so that it can be married up with the sort order specified in the querystring, and with the default_orderby for the ToasterTable (see below).
* static_data_template specifies how to render a value from a row into the HTML for the page. The data.completed_on reference in the template demonstrates how to access field values from the record for the row: the object representing the current row can be reference via the 'data' object in the static_data_template string. The whole of Django's template machinery is available when specifying how to render properties of the record for a row: you can include other templates and use filters and template tags (your own or Django's). In this case, the date filter is used to format the date into a human-readable form.

The self.default_orderby member set in __init__() specifies the default column to use for ordering the table. In this case, the completed_on field is used to sort the builds in the table; the "-" specifies reverse order, so the newest build appears at the top of the table. Note that the value for self.default_orderby should match the name of a field in the model, and the name of a column added to the table.

See the [[#Column options|Column options]] section for full details of the column options available.

=== Add the template ===

This should go into templates/, as this is where Django looks for view templates.

In all cases, you will need to include the toastertable.html template from your template. You will also need to add links for the jQuery UI CSS and JS files. (In an ideal world, this would be in toastertable.html, but for historical reasons, it isn't yet.)

Finally, you should define an xhr_table_url variable in your template. This is used to request new data for the table when filters or search terms are applied, or if a new page is requested. ToasterTable will send a request for the table JSON to this URL, then automatically refresh the table display with the new data when it's received.

As an example, here's a minimal template for the Mini Builds page, which would go in templates/minibuilds.html:

<pre>
{% extends 'base.html' %}
{% load static %}

{% block extraheadcontent %}
<link rel="stylesheet" href="{% static 'css/jquery-ui.min.css' %}" type='text/css'>
<link rel="stylesheet" href="{% static 'css/jquery-ui.structure.min.css' %}" type='text/css'>
<link rel="stylesheet" href="{% static 'css/jquery-ui.theme.min.css' %}" type='text/css'>
<script src="{% static 'js/jquery-ui.min.js' %}"></script>
{% endblock %}

{% block title %}{{title}}{% endblock %}

{% block pagecontent %}
<h1 class="page-header top-air">{{title}}</h1>
{% url 'minibuilds' as xhr_table_url %}
{% include 'toastertable.html' %}
{% endblock %}
</pre>

Depending on which side menus or other wrapping you need around your table, you may need to create a template by adapting an existing one (particularly if you are porting a table to ToasterTable).

Note that the template above extends base.html, which provides the header/footer for Toaster itself but doesn't have a side menu. For an example of a more complex page which does have a side menu, see generic-toastertable-page.html and its parent template, baseprojectpage.html.

The important points to remember, though, are shown in the example template above; in particular, the need to import the jQuery UI assets, set xhr_table_url, and include the toastertable.html template.

=== Add a URL mapping for the table ===

To be able to view the Mini Builds page, it needs a mapping in the urls.py file:

<pre>
urlpatterns = patterns('toastergui.views',
// ... other mappings ...

url(r'^minibuilds/$',
tables.MiniBuildsTable.as_view(template_name='minibuilds.html'),
name='minibuilds')
)
</pre>

This is standard Django template mapping code. The only wrinkle is that because a ToasterTable is a Django TemplateView subclass, we call Django's as_view() method on our table class to render it, passing the name of the template file. (Usually, a view mapping would specify a function in the views.py file.)

Now, with Toaster running locally, you should be able to visit

http://localhost:8000/toastergui/minibuilds/

in a browser and see your Mini Builds table (note that it will be empty unless you've run some builds).

[[File:MiniBuildsTable.png|750px|Adding a project name column to MiniBuildsTable]]

=== Aside: turning off ToasterTable caching ===

By default, ToasterTable caches data to prevent the same data being fetched multiple times. However, during development, this may mean that you see stale data in the table (for example, I changed the setup_columns() method in MiniBuildsTable to remove a column, but when I refreshed the page, the column was still present).

To get around this, pass the nocache option in the querystring in your browser, e.g.

http://localhost:8000/toastergui/minibuilds?nocache=true

This ensures that ToasterTable doesn't cache data between page refreshes.

== Further ToasterTable configuration ==

The following sections flesh out the options for adding columns to a ToasterTable, adding more non-table data to the page, and using the table filter APIs.

=== Column options ===

As stated previously, columns are added to the table inside the setup_columns() method of your ToasterTable subclass. The following options can be set when adding a column to a table:

* '''title''' (string): The heading for the column.
* '''help_text''' (string; optional): Text which describes the column content of the column; this is shown in a popover when the question mark next to the column heading is hovered over. If not set, the column doesn't have a help icon.
* '''hideable''' (boolean; default=True): True if the user can hide the column (using the "Edit columns" drop-down).
* '''hidden''' (boolean; default=False): True if the column is hidden by default; the user can show the column using the "Edit columns" drop-down.
* '''field_name''' (string; optional): Name of the property to render into the field. Note that this can be a property or method on the model being rendered (e.g. for a Build object, completed_on) ; it could also be a property of a related object (e.g. for a Build, project__name).
* '''static_data_name''' (string; optional): This should ''not'' be set if field_name is set, but ''must'' be set if static_data_template is set.
* '''static_data_template''' (string; optional): The template to render for each row; the data for the row is interpolated into the template. This is more flexible than field_name, as you can add links, conditional statements and additional formatting each time the cell for this column is rendered for a record.
* '''orderable''' (boolean; default=False): True if the table can be ordered by this column. Note that if this is True, either field_name or static_data_name should match a name of one of the fields in the queryset for the ToasterTable. If not, Toaster will not be able to sort the table by that column.
* '''filter_name''' (string; optional): Name of the TableFilter associated with this column, which adds filtering behaviour for this column; see the [[#Filter API|Filter API]] section for full details.

As an example, I'll add two columns to the Mini Builds table.

First, a column to show the project name for the build:

<pre>
class MiniBuildsTable(ToasterTable):
# ... other code ...

def setup_columns(self, *args, **kwargs):
self.add_column(title='Completed on',
help_text='The date and time when the build finished',
hideable=False,
orderable=True,
static_data_name='completed_on',
static_data_template='{{data.completed_on | date:"d/m/y H:i"}}')

self.add_column(title='Project',
help_text='The project associated with this build',
hideable=True,
orderable=True,
field_name='project__name')
</pre>

The rest of the MiniBuildsTable remains the same: only setup_columns() changes. In this case, I'm showing the project name for the build using the field_name property for the column. This is simple, but doesn't allow me to modify how the value is formatted or add any HTML elements around it.

Try http://localhost:8000/toastergui/minibuilds/ again and you should see the new column. Note that this column can be hidden in the "Edit columns" drop-down, and can also be used to order the table.

[[File:MiniBuildsTable-projectname.png|750px|Adding a project name column to MiniBuildsTable]]

Next, a column to show the outcome for the build, with a link to that build's dashboard:

<pre>
class MiniBuildsTable(ToasterTable):
# ... other code ...

def setup_columns(self, *args, **kwargs):
outcome_template = '''
<a href="{% url 'builddashboard' data.id %}">
{% if data.outcome == 0 %}
succeeded
{% elif data.outcome == 1 %}
failed
{% endif %}
</a>
'''

self.add_column(title='Completed on',
help_text='The date and time when the build finished',
hideable=False,
orderable=True,
static_data_name='completed_on',
static_data_template='{{data.completed_on | date:"d/m/y H:i"}}')

self.add_column(title='Project',
help_text='The project associated with this build',
hideable=True,
orderable=True,
field_name='project__name')

self.add_column(title='Outcome',
help_text='The outcome of the build',
hideable=True,
orderable=True,
static_data_name='outcome',
static_data_template=outcome_template)
</pre>

In this case, static_data_template is used as we don't just want the value of the outcome field for the row: we also want to create a link. The template is slightly more complex than previous ones, so it's in a variable to make the code cleaner. Note the use of the built-in Django ''url'' template tag to get the URL for the build dashboard.

[[File:MiniBuildsTable-outcome.png|750px|Adding an outcome column to MiniBuildsTable]]

There is one issue with outcome_template: it compares the outcome field's value with integer values (0 for succeeded, 1 for failed) to determine what to show in the cell. This is a bit opaque for any developer coming to the project later, and also fragile if the integer values representing build outcomes change at a later date. It would be better to use the constants for Build outcomes (Build.SUCCEEDED, Build.FAILED) instead. However, the Build object is not accessible to the template, as it isn't passed in with the template context. The next section explains how to deal with this and similar missing context.

=== Additional context data ===

Sometimes a page will require data which is not in the queryset. This data may be shown on the page, or may influence how other data is rendered. For example, in the previous section, the outcome was displayed according to whether the build was a success or had some other outcome; but, rather than using hard-coded integer values, it would be better to use the Build.SUCCEEDED and Build.FAILED constants for this.

As another example, the Mini Builds page could show the time of the most recent build. While this could be done with some work in the template, it would be much cleaner to add a variable for "most_recent_build" to the template context instead.

In both cases, the required data must be added to the extra context data for the page. This can be done in one of two ways, depending on how you want to use it:

# If the data needs to be available to column templates, it should be added to the ''static_context_extra'' dictionary for the ToasterTable. For example, because we need to access constants on the Build class in the column templates, we should add Build to static_context_extra.
# If the data needs to be available to other templates (e.g. minibuilds.html in the case of Mini Builds), it should be set in the ''get_context_data()'' method.

==== Data for column templates ====

Data can be made available to column templates for a ToasterTable by adding it to the static_context_extra dictionary inside the __init__() method. This makes the data available for use in column templates (set using static_data_template). However, this data is ''only'' accessible from column templates, and is ''not'' available to the other templates used to render the page.

For example, to add the Build class to static_context_extra so that it can be used in a column template:

<pre>
class MiniBuildsTable(ToasterTable):
def __init__(self, *args, **kwargs):
super(MiniBuildsTable, self).__init__(*args, **kwargs)
self.default_orderby = '-completed_on'
self.title = 'Mini Builds Table'

# add the Build class to the static context
self.static_context_extra['Build'] = Build

def setup_queryset(self, *args, **kwargs):
self.queryset = Build.objects.all()

def setup_columns(self, *args, **kwargs):
# reference constants on Build from a column template
outcome_template = '''
<a href="{% url 'builddashboard' data.id %}">
{% if data.outcome == extra.Build.SUCCEEDED %}
succeeded
{% elif data.outcome == extra.Build.FAILED %}
failed
{% endif %}
</a>
'''

# ... the rest of setup_columns is the same ...
</pre>

To refer to data from static_context_extra in a column template, use the ''extra'' keyword (similar to how the ''data'' keyword works). In this case, I referenced the constants on the Build class with ''extra.Build.SUCCEEDED'' and ''extra.Build.FAILED''.

The page renders the same as before, but we no longer have hard-coded integer values in the column templates.

==== Data for other templates ====

Adding other data to the context for use in the page template is the same as for other Django TemplateView objects. See [https://docs.djangoproject.com/en/1.9/topics/class-based-views/generic-display/#adding-extra-context the Django documentation] for full details.

As a simple worked example, here's how we could show the most recent Toaster build in the Mini Builds page. First, we need to implement get_context_data() and add our own data to it; in this case, a "most_recent_build" property containing a Build object:

<pre>
class MiniBuildsTable(ToasterTable):
# ... other methods remain as they were ...

def get_context_data(self, **kwargs):
# invoke the super class' method, to include data like the page
# title in the context
context = super(MiniBuildsTable, self).get_context_data(**kwargs)

# add the most recent build to the context
all_builds = Build.objects.all().order_by('-completed_on')
context['most_recent_build'] = all_builds.first()

return context
</pre>

All of the properties on the context object returned by get_context_data() are available in the page template, as per the context for a standard Django template. To show the most recent build in the minibuilds.html template, we now use the most_recent_build property we added in get_context_data(), we modify the pagecontent block:

<pre>
{% block pagecontent %}
<h1 class="page-header top-air">{{title}}</h1>


{% if most_recent_build %}
<p>Most recent build completed: {{most_recent_build.completed_on}}</p>
{% endif %}

{% url 'minibuilds' as xhr_table_url %}
{% include 'toastertable.html' %}
{% endblock %}
</pre>

The result looks like this:

[[File:MiniBuildsTable-mostrecentbuild.png|750px|Showing the most recent build in the MiniBuildsTable]]

==== Dynamic data for other templates ====

If you need different data in the context depending on the URL (for example, you want an object retrieved using the ID in the URL), the pattern is similar to the above. The main difference is that you make use of the kwargs parameter passed to get_context_data(), which contains parameters derived from the page's URL.

As an example, the project builds table shows all of the builds for a project in a ToasterTable. It makes sense to add the project to the context so that its name can be shown at the top of the page. For a URL like http://localhost:8000/toastergui/project/X/builds, Toaster assigns the "X" in the URL to a parameter called ''pid''; this can be retrieved in get_context_data() and used to fetch the project for which we are showing builds:

<pre>
class ProjectBuildsTable(ToasterTable):
# ... other methods ...

def get_context_data(self, **kwargs):
context = super(ProjectBuildsTable, self).get_context_data(**kwargs)

# use the pid parameter extracted from the URL
context['project'] = Project.objects.get(pk=kwargs['pid'])

return context
</pre>

This can then be referenced the usual way in the template (e.g. <code>{{project}}</code>).

=== Search ===

Search is automatically enabled on a ToasterTable. However, you may find that you are unable to search on the fields you would like to. This section explains how to fix that.

By default, ToasterTable searches against the queryset's model, using an OR query with icontains (case insensitive, contains string) matching. The fields used for the search are defined by the search_allowed_fields property of the model.

For MiniBuildsTable, the queryset's model is orm.models.Build; search_allowed_fields (at the time of writing) for Build is set to:

<pre>
['machine', 'cooker_log_path', 'target__target', 'target__target_image_file__file_name']
</pre>

Doing a search for a string like "test" would therefore construct a query OR clause like the following (pseudo-SQL):

<pre>
... WHERE machine LIKE '%test%' OR cooker_log_path LIKE '%test%' OR target.target LIKE '%test%'
OR target.target_image_file.file_name LIKE '%test%'
</pre>

(The actual SQL clause is far more complicated, as Django would have to use various JOIN statements to combine rows from multiple database tables.)

Note that the search_allowed_fields don't have to be fields of the model: they can be fields in related models (here, the Target and TargetImageFile models related to a Build).

This means that it's not possible to search by project name in the MiniBuildsTable. To make this possible, we could modify search_allowed_fields for the Build model to:

<pre>
['machine', 'cooker_log_path', 'target__target', 'target__target_image_file__file_name', 'project__name']
</pre>

This would now allow the Mini Builds table to be searched by project name as well.

== Filter API ==

The classes for creating filters are defined in tablefilter.py.

A filter applies to a single column in the table. Any actions on a filter should filter records according to criteria which make sense with respect to that column. For example, you wouldn't add a filter to the "Completed on" column of the "All Builds" table which filters builds according to whether they failed or succeeded; but you would add a filter which filters the builds based on the time when the build was completed.

A filter adds an option to the column heading to filter in/out particular records or to remove filtering altogether.

Each filter has one or more actions associated with it. A filter also automatically gets a default action which turns off the filter and shows all the records again.

Each action changes the records shown in its associated ToasterTable, by applying a set of criteria to the query used to fetch records (e.g. show records for a single date, for a date range, or matching/not matching some other criteria related to the column).

To add a filter to a column, do the following:

# Create a TableFilter object for the column.
# Assign the filter to the column in the ToasterTable.
# Attach TableFilterAction objects to the TableFilter.

These steps are explained in detail below.

=== Add a table filter ===

The TableFilter class acts as a container for a group of TableFilterAction objects.

To create a TableFilter, instantiate it with a unique identifier (unique to the ToasterTable) and a text string which is displayed in the popup for the filter. The filters should be defined in the setup_filters() method for the ToasterTable.

Once the filter is defined, add it to the ToasterTable using the add_filter() method.

Here's an example of a TableFilter for the outcome column of MiniBuildsTable:

<pre>
# already imported in tables.py, but shown here for completeness
from toastergui.tablefilter import TableFilter

class MiniBuildsTable(ToasterTable):
# ... other methods ...

def setup_filters(self, *args, **kwargs):
# filter by outcome (succeeded or failed)
outcome_filter = TableFilter(
'outcome_filter',
'Filter builds by outcome'
)

# add the filter to the ToasterTable
self.add_filter(outcome_filter)
</pre>

Internally, add_filter() uses a TableFilterMap in the ToasterTable, which maps from column names to TableFilters.

Note that the filter has no actions, so won't actually filter the table yet.

=== Associate a filter with a column ===

To associate the filter with a column, pass the name of the filter in the filter_name argument for the column. For example, to associate the outcome_filter defined above with the outcome column, change setup_columns() like this:

<pre>
class MiniBuildsTable(ToasterTable):
# ... other methods ...

def setup_columns(self, *args, **kwargs):
# ... other column definitions ...

self.add_column(title='Outcome',
help_text='The outcome of the build',
hideable=True,
orderable=True,
static_data_name='outcome',
static_data_template=outcome_template,
filter_name='outcome_filter')
</pre>

(Note the additional filter_name keyword passed to add_column().)

Now if you visit http://localhost:8000/toastergui/minibuilds, the outcome column in the table should show the filter icon; clicking on this opens the popup which allows a user to select the filter to apply. Because we have no actions on the filter, the only option is the "All" one:

[[File:MiniBuildsTable-outcomefilter.png|750px|Adding an outcome filter to the MiniBuildsTable]]

outcome_filter is the string used to refer to the filter in the querystring; see the section [[#How filters are applied|How filters are applied]] for more details about how filters are applied.

=== Add table filter actions ===

The next step is to add actions to the filter.

TableFilterAction is the base class for an action associated with a filter. The following types of filter action (subclasses of TableFilterAction) are already implemented:

* TableFilterActionToggle: filter the records shown in the table by some arbitrary criteria; the action is either on or off.
* TableFilterActionDay: filter the records shown by day (yesterday or today).
* TableFilterActionDateRange: filter the records shown by a from/to date range.

In the case of TableFilterActionDay and TableFilterActionDateRange, you specify the field which is used for the filter action. In the case of TableFilterActionToggle, you can use arbitrary criteria for the action.

To add an action to a filter, create an instance of the desired action class and use the TableFilter.add_action() method to associate it with a filter. For example, here's how to add two actions to the outcome_filter defined earlier: one to show successful builds, and the other to show failed builds:

<pre>
# already imported in tables.py, but shown here for completeness
from toastergui.tablefilter import TableFilter
from toastergui.tablefilter import TableFilterActionToggle

class MiniBuildsTable(ToasterTable):
# ... other methods ...

def setup_filters(self, *args, **kwargs):
# filter by outcome (succeeded or failed)
outcome_filter = TableFilter(
'outcome_filter',
'Filter builds by outcome'
)

successful_builds_action = TableFilterActionToggle(
'successful_builds',
'Successful builds',
Q(outcome=Build.SUCCEEDED)
)

failed_builds_action = TableFilterActionToggle(
'failed_builds',
'Failed builds',
Q(outcome=Build.FAILED)
)

outcome_filter.add_action(successful_builds_action)
outcome_filter.add_action(failed_builds_action)

# add the filter to the ToasterTable
self.add_filter(outcome_filter)
</pre>

Here's how this will render:

[[File:MiniBuildsTable-outcomefilteractions.png|750px|Adding outcome filter actions to the MiniBuildsTable]]

I'll break this down further to give a bit more detail about how the filter action is created. Here's the code which creates the filter action to show successful builds only:

<pre>
successful_builds_action = TableFilterActionToggle(
'successful_builds',
'Successful builds',
Q(outcome=Build.SUCCEEDED)
)
</pre>

The arguments to TableFilterActionToggle have the following meanings:
* 'successful_builds' is the name of the action. This is used to map from the filter parameter in the querystring (see the next section).
* 'Successful builds' is the label shown next to the radio button which activates this action in the popup.
* Q(outcome=Build.SUCCEEDED) shows the criteria used to filter the records in the table's queryset when the filter action is applied. The [https://docs.djangoproject.com/en/1.9/topics/db/queries/#complex-lookups-with-q Q object] is part of the Django API; it should reference field names which are present in the queryset to be filtered and can use any criteria available to Q objects. In the case of the MiniBuildsTable, the queryset consists of Build objects; the Q(outcome=Build.SUCCEEDED) object is used to filter this queryset, so we effectively get the result of Build.objects.all().filter(Q(outcome=Build.SUCCEEDED)) when the action is applied.

For examples of how to add TableFilterActionDay and TableFilterActionDateRange filter actions, see the BuildsTable class in tables.py.

The next section explains how we go from clicking on a radio button in the filter popup to filtering the records shown in the table.

=== How filters are applied ===

The short version:

When a ToasterTable is rendered, a filter icon is shown on any column which has a filter_name defined for it. Clicking on this icon populates the filter dialog, then opens it so the user can select a filter to apply. When the user clicks on the "Apply" button in the dialog, new data for the table is requested, using the filter name, filter action, and filter value from the querystring. These parameters are used to filter the records which are shown in the table.

The long version:

* A filter icon for a column has a filter name associated with it.
* When a filter icon is clicked, the Toaster UI makes an Ajax request to the Toaster back-end for data about that filter name.
* When the filter data is received (in JSON format), the filter dialog is populated with radio buttons (one per action), labels (one per action) and any additional fields (e.g. date range fields for TableFilterActionDateRange actions). The count of records which will be returned by an action is part of the data returned by the back-end; if this is 0, the label and radio button are disabled. See static/js/table.js for the code which populates the filter dialog.
* The dialog is opened so the user can choose a filter to apply. The user clicks radio buttons, fills in fields etc.
* When the user clicks on the "Apply" button, the URL for the page is modified in place to reflect the filter criteria. The filter is represented in the URL by two querystring parameters:
** <code>filter=<filter name>:<filter action></code> : the filter name maps to the name used when creating the TableFilter object; and the filter action corresponds to one of the names of a TableFilterAction object added to the TableFilter. For example, ''filter=outcome_filter:successful_builds'' will map to the ''outcome_filter'' TableFilter and its ''successful_builds'' TableFilterAction.
** <code>filter_value=<filter value string></code> : for a TableFilterActionToggle filter, this is always "on", to show that the filter is applied; for a TableFilterActionDay this is either "today" or "yesterday"; for a TableFilterActionDateRange, this is a "from,to" date range in the format "2015-12-09,2015-12-11".
* The table data is fetched via Ajax, using the filter and filter_value parameters as part of the URL. These set the filter name, action and value to use for filtering.
* The back-end applies the requested filter action to the queryset (as well as any existing search string). Each filter action has a set of criteria which it applies to the queryset, as follows:
** TableFilterActionToggle: The recordset is filtered by the criteria specified when the action is created (see [[#Add table filter actions|Add table filter actions]] for an example).
** TableFilterActionDay: A date range clause is constructed at the time the filter action is applied. For example, if the field the filter action applies to is "completed_on", the day set for the TableFilterActionDay is "today", and today is 2016-03-05, the query clause (in pseudo-SQL) is "completed_on >= '2016-03-04 00:00:00' AND completed_on <= '2016-03-04 23:59:59'".
** TableFilterActionDateRange: This is similar to TableFilterActionDay, but the user specifies the start and end dates; these are in the filter_value variable in the querystring. For example, if the field the filter action applies to is "completed_on" and the date range is "2016-03-01,2016-03-04", the query clause (in pseudo-SQL) is "completed_on >= '2016-03-01 00:00:00' AND completed_on <= '2016-03-04 23:59:59'".
* The filtered queryset is returned as JSON.
* The Toaster UI code redraws the table with the filtered queryset.

ToasterTable

2016-05-17T17:15:04Z

Michael Wood:

[[Category:Toaster]]

== Introduction to ToasterTable ==

ToasterTable is a set of classes in the toastergui module of the Toaster Django application. It is used to present a set of database records in a style and with functionality consistent with the rest of Toaster.

The functionality provided by ToasterTable includes:
* Sorting
* Column hiding/showing
* Filtering
* Paging
* Search

When a ToasterTable is updated, a new set of records matching the filter criteria, search string, sort order and page number is fetched from the back-end using Ajax requests. The ToasterTable is then updated in place from the JSON in the response, using JavaScript to redraw the table rows.

As a developer, you don't need to worry (necessarily) about how this works: you just implement a subclass of ToasterTable, specify a template for rendering it, and supply a URL mapping.

The following sections explain how to use ToasterTable. The filenames in these sections refer to files in the toastergui/ directory.

Note that to be able to follow the example, you will need to set up Toaster for development work. This is described in [http://www.yoctoproject.org/docs/2.0.1/toaster-manual/toaster-manual.html the Toaster manual].

== How to create a ToasterTable ==

Most of the tables needed for Toaster are already implemented. Many of the key tables in Toaster already use ToasterTable, but some don't. A full list of unconverted tables is in https://bugzilla.yoctoproject.org/show_bug.cgi?id=8363.

This means that you will typically be porting an existing table to ToasterTable, rather than adding a new table from scratch. However, to keep things simple, this section explains how to create a ToasterTable from scratch. Knowing how to do this should make it easier to port an existing table to ToasterTable in future.

To provide a worked example, I'll create a new table which is a variant of the existing "all builds" table, using a reduced number of columns and filters.

=== Create the table class ===

A quick overview of where things are and where they go when creating a table:

* widgets.py contains the ToasterTable class. This is the base class for all ToasterTables.
* New tables are added to tables.py.
* A table in tables.py maps to a URL within the Toaster application; the mapping is defined in urls.py.
* Templates for the table are added to templates/.
* If you need [https://docs.djangoproject.com/en/1.9/howto/custom-template-tags/ custom template tags], these go in templatetags/.

To create a new table, first add the class definition to tables.py:

<pre>
# import any models you need for the table
from orm.models import Build

class MiniBuildsTable(ToasterTable):
def __init__(self, *args, **kwargs):
super(MiniBuildsTable, self).__init__(*args, **kwargs)
self.default_orderby = '-completed_on'
self.title = 'Mini Builds Table'

def setup_queryset(self, *args, **kwargs):
self.queryset = Build.objects.all()

def setup_columns(self, *args, **kwargs):
self.add_column(title='Completed on',
help_text='The date and time when the build finished',
hideable=False,
orderable=True,
static_data_name='completed_on',
static_data_template='{{data.completed_on | date:"d/m/y H:i"}}')
</pre>

The key to creating a ToasterTable is to decide which data is going to be shown in the table and define this in the ''setup_queryset()'' method. This will typically be a queryset derived from one model in the Toaster application. To see the available models, check the orm/models.py file.

Each row in the table corresponds to a record in the queryset. The ''setup_queryset()'' method specifies how to get this queryset for the table: it must set the self.queryset property to the Django queryset containing the data. In the MiniBuildsTable, we show all of the builds by default (using the standard Django model API).

Each column in the ToasterTable corresponds to a field in the queryset. Inside a table row, a cell corresponds to one or more fields from each record in the queryset. A cell can show various aspects of a record: a formatted version of a field's value, an amalgam of multiple fields, a computation based on a group of related records etc. For example, in a row of the MiniBuildsTable, a cell might contain the outcome of the build, the number of tasks which failed during the build, or the time since the build completed.

The ''setup_columns()'' method defines which columns should be shown in the table. In this initial version of MiniBuildsTable, only the "completed_on" column is shown. This column has the following properties:
* It cannot be hidden (hideable=False).
* It can be used to order the table (orderable=True).
* static_data_name sets the name of the column so that it can be married up with the sort order specified in the querystring, and with the default_orderby for the ToasterTable (see below).
* static_data_template specifies how to render a value from a row into the HTML for the page. The data.completed_on reference in the template demonstrates how to access field values from the record for the row: the object representing the current row can be reference via the 'data' object in the static_data_template string. The whole of Django's template machinery is available when specifying how to render properties of the record for a row: you can include other templates and use filters and template tags (your own or Django's). In this case, the date filter is used to format the date into a human-readable form.

The self.default_orderby member set in __init__() specifies the default column to use for ordering the table. In this case, the completed_on field is used to sort the builds in the table; the "-" specifies reverse order, so the newest build appears at the top of the table. Note that the value for self.default_orderby should match the name of a field in the model, and the name of a column added to the table.

See the [[#Column options|Column options]] section for full details of the column options available.

=== Add the template ===

This should go into templates/, as this is where Django looks for view templates.

In all cases, you will need to include the toastertable.html template from your template. You will also need to add links for the jQuery UI CSS and JS files. (In an ideal world, this would be in toastertable.html, but for historical reasons, it isn't yet.)

Finally, you should define an xhr_table_url variable in your template. This is used to request new data for the table when filters or search terms are applied, or if a new page is requested. ToasterTable will send a request for the table JSON to this URL, then automatically refresh the table display with the new data when it's received.

As an example, here's a minimal template for the Mini Builds page, which would go in templates/minibuilds.html:

<pre>
{% extends 'base.html' %}
{% load static %}

{% block extraheadcontent %}
<link rel="stylesheet" href="{% static 'css/jquery-ui.min.css' %}" type='text/css'>
<link rel="stylesheet" href="{% static 'css/jquery-ui.structure.min.css' %}" type='text/css'>
<link rel="stylesheet" href="{% static 'css/jquery-ui.theme.min.css' %}" type='text/css'>
<script src="{% static 'js/jquery-ui.min.js' %}"></script>
{% endblock %}

{% block title %}{{title}}{% endblock %}

{% block pagecontent %}
<h1 class="page-header top-air">{{title}}</h1>
{% url 'minibuilds' as xhr_table_url %}
{% include 'toastertable.html' %}
{% endblock %}
</pre>

Depending on which side menus or other wrapping you need around your table, you may need to create a template by adapting an existing one (particularly if you are porting a table to ToasterTable).

Note that the template above extends base.html, which provides the header/footer for Toaster itself but doesn't have a side menu. For an example of a more complex page which does have a side menu, see generic-toastertable-page.html and its parent template, baseprojectpage.html.

The important points to remember, though, are shown in the example template above; in particular, the need to import the jQuery UI assets, set xhr_table_url, and include the toastertable.html template.

=== Add a URL mapping for the table ===

To be able to view the Mini Builds page, it needs a mapping in the urls.py file:

<pre>
urlpatterns = patterns('toastergui.views',
// ... other mappings ...

url(r'^minibuilds/$',
tables.MiniBuildsTable.as_view(template_name='minibuilds.html'),
name='minibuilds')
)
</pre>

This is standard Django template mapping code. The only wrinkle is that because a ToasterTable is a Django TemplateView subclass, we call Django's as_view() method on our table class to render it, passing the name of the template file. (Usually, a view mapping would specify a function in the views.py file.)

Now, with Toaster running locally, you should be able to visit

http://localhost:8000/toastergui/minibuilds/

in a browser and see your Mini Builds table (note that it will be empty unless you've run some builds).

[[File:MiniBuildsTable.png|750px|Adding a project name column to MiniBuildsTable]]

=== Aside: turning off ToasterTable caching ===

By default, ToasterTable caches data to prevent the same data being fetched multiple times. However, during development, this may mean that you see stale data in the table (for example, I changed the setup_columns() method in MiniBuildsTable to remove a column, but when I refreshed the page, the column was still present).

To get around this, pass the nocache option in the querystring in your browser, e.g.

http://localhost:8000/toastergui/minibuilds?nocache=true

This ensures that ToasterTable doesn't cache data between page refreshes.

== Further ToasterTable configuration ==

The following sections flesh out the options for adding columns to a ToasterTable, adding more non-table data to the page, and using the table filter APIs.

=== Column options ===

As stated previously, columns are added to the table inside the setup_columns() method of your ToasterTable subclass. The following options can be set when adding a column to a table:

* '''title''' (string): The heading for the column.
* '''help_text''' (string; optional): Text which describes the column content of the column; this is shown in a popover when the question mark next to the column heading is hovered over. If not set, the column doesn't have a help icon.
* '''hideable''' (boolean; default=True): True if the user can hide the column (using the "Edit columns" drop-down).
* '''hidden''' (boolean; default=False): True if the column is hidden by default; the user can show the column using the "Edit columns" drop-down.
* '''field_name''' (string; optional): Name of the property to render into the field. Note that this can be a property or method on the model being rendered (e.g. for a Build object, completed_on) ; it could also be a property of a related object (e.g. for a Build, project__name).
* '''static_data_name''' (string; optional): This should ''not'' be set if field_name is set, but ''must'' be set if static_data_template is set.
* '''static_data_template''' (string; optional): The template to render for each row; the data for the row is interpolated into the template. This is more flexible than field_name, as you can add links, conditional statements and additional formatting each time the cell for this column is rendered for a record.
* '''orderable''' (boolean; default=False): True if the table can be ordered by this column. Note that if this is True, either field_name or static_data_name should match a name of one of the fields in the queryset for the ToasterTable. If not, Toaster will not be able to sort the table by that column.
* '''filter_name''' (string; optional): Name of the TableFilter associated with this column, which adds filtering behaviour for this column; see the [[#Filter API|Filter API]] section for full details.

As an example, I'll add two columns to the Mini Builds table.

First, a column to show the project name for the build:

<pre>
class MiniBuildsTable(ToasterTable):
# ... other code ...

def setup_columns(self, *args, **kwargs):
self.add_column(title='Completed on',
help_text='The date and time when the build finished',
hideable=False,
orderable=True,
static_data_name='completed_on',
static_data_template='{{data.completed_on | date:"d/m/y H:i"}}')

self.add_column(title='Project',
help_text='The project associated with this build',
hideable=True,
orderable=True,
field_name='project__name')
</pre>

The rest of the MiniBuildsTable remains the same: only setup_columns() changes. In this case, I'm showing the project name for the build using the field_name property for the column. This is simple, but doesn't allow me to modify how the value is formatted or add any HTML elements around it.

Try http://localhost:8000/toastergui/minibuilds/ again and you should see the new column. Note that this column can be hidden in the "Edit columns" drop-down, and can also be used to order the table.

[[File:MiniBuildsTable-projectname.png|750px|Adding a project name column to MiniBuildsTable]]

Next, a column to show the outcome for the build, with a link to that build's dashboard:

<pre>
class MiniBuildsTable(ToasterTable):
# ... other code ...

def setup_columns(self, *args, **kwargs):
outcome_template = '''
<a href="{% url 'builddashboard' data.id %}">
{% if data.outcome == 0 %}
succeeded
{% elif data.outcome == 1 %}
failed
{% endif %}
</a>
'''

self.add_column(title='Completed on',
help_text='The date and time when the build finished',
hideable=False,
orderable=True,
static_data_name='completed_on',
static_data_template='{{data.completed_on | date:"d/m/y H:i"}}')

self.add_column(title='Project',
help_text='The project associated with this build',
hideable=True,
orderable=True,
field_name='project__name')

self.add_column(title='Outcome',
help_text='The outcome of the build',
hideable=True,
orderable=True,
static_data_name='outcome',
static_data_template=outcome_template)
</pre>

In this case, static_data_template is used as we don't just want the value of the outcome field for the row: we also want to create a link. The template is slightly more complex than previous ones, so it's in a variable to make the code cleaner. Note the use of the built-in Django ''url'' template tag to get the URL for the build dashboard.

[[File:MiniBuildsTable-outcome.png|750px|Adding an outcome column to MiniBuildsTable]]

There is one issue with outcome_template: it compares the outcome field's value with integer values (0 for succeeded, 1 for failed) to determine what to show in the cell. This is a bit opaque for any developer coming to the project later, and also fragile if the integer values representing build outcomes change at a later date. It would be better to use the constants for Build outcomes (Build.SUCCEEDED, Build.FAILED) instead. However, the Build object is not accessible to the template, as it isn't passed in with the template context. The next section explains how to deal with this and similar missing context.

=== Additional context data ===

Sometimes a page will require data which is not in the queryset. This data may be shown on the page, or may influence how other data is rendered. For example, in the previous section, the outcome was displayed according to whether the build was a success or had some other outcome; but, rather than using hard-coded integer values, it would be better to use the Build.SUCCEEDED and Build.FAILED constants for this.

As another example, the Mini Builds page could show the time of the most recent build. While this could be done with some work in the template, it would be much cleaner to add a variable for "most_recent_build" to the template context instead.

In both cases, the required data must be added to the extra context data for the page. This can be done in one of two ways, depending on how you want to use it:

# If the data needs to be available to column templates, it should be added to the ''static_context_extra'' dictionary for the ToasterTable. For example, because we need to access constants on the Build class in the column templates, we should add Build to static_context_extra.
# If the data needs to be available to other templates (e.g. minibuilds.html in the case of Mini Builds), it should be set in the ''get_context_data()'' method.

==== Data for column templates ====

Data can be made available to column templates for a ToasterTable by adding it to the static_context_extra dictionary inside the __init__() method. This makes the data available for use in column templates (set using static_data_template). However, this data is ''only'' accessible from column templates, and is ''not'' available to the other templates used to render the page.

For example, to add the Build class to static_context_extra so that it can be used in a column template:

<pre>
class MiniBuildsTable(ToasterTable):
def __init__(self, *args, **kwargs):
super(MiniBuildsTable, self).__init__(*args, **kwargs)
self.default_orderby = '-completed_on'
self.title = 'Mini Builds Table'

# add the Build class to the static context
self.static_context_extra['Build'] = Build

def setup_queryset(self, *args, **kwargs):
self.queryset = Build.objects.all()

def setup_columns(self, *args, **kwargs):
# reference constants on Build from a column template
outcome_template = '''
<a href="{% url 'builddashboard' data.id %}">
{% if data.outcome == extra.Build.SUCCEEDED %}
succeeded
{% elif data.outcome == extra.Build.FAILED %}
failed
{% endif %}
</a>
'''

# ... the rest of setup_columns is the same ...
</pre>

To refer to data from static_context_extra in a column template, use the ''extra'' keyword (similar to how the ''data'' keyword works). In this case, I referenced the constants on the Build class with ''extra.Build.SUCCEEDED'' and ''extra.Build.FAILED''.

The page renders the same as before, but we no longer have hard-coded integer values in the column templates.

==== Data for other templates ====

Adding other data to the context for use in the page template is the same as for other Django TemplateView objects. See [https://docs.djangoproject.com/en/1.9/topics/class-based-views/generic-display/#adding-extra-context the Django documentation] for full details.

As a simple worked example, here's how we could show the most recent Toaster build in the Mini Builds page. First, we need to implement get_context_data() and add our own data to it; in this case, a "most_recent_build" property containing a Build object:

<pre>
class MiniBuildsTable(ToasterTable):
# ... other methods remain as they were ...

def get_context_data(self, **kwargs):
# invoke the super class' method, to include data like the page
# title in the context
context = super(MiniBuildsTable, self).get_context_data(**kwargs)

# add the most recent build to the context
all_builds = Build.objects.all().order_by('-completed_on')
context['most_recent_build'] = all_builds.first()

return context
</pre>

All of the properties on the context object returned by get_context_data() are available in the page template, as per the context for a standard Django template. To show the most recent build in the minibuilds.html template, we now use the most_recent_build property we added in get_context_data(), we modify the pagecontent block:

<pre>
{% block pagecontent %}
<h1 class="page-header top-air">{{title}}</h1>


{% if most_recent_build %}
<p>Most recent build completed: {{most_recent_build.completed_on}}</p>
{% endif %}

{% url 'minibuilds' as xhr_table_url %}
{% include 'toastertable.html' %}
{% endblock %}
</pre>

The result looks like this:

[[File:MiniBuildsTable-mostrecentbuild.png|750px|Showing the most recent build in the MiniBuildsTable]]

==== Dynamic data for other templates ====

If you need different data in the context depending on the URL (for example, you want an object retrieved using the ID in the URL), the pattern is similar to the above. The main difference is that you make use of the kwargs parameter passed to get_context_data(), which contains parameters derived from the page's URL.

As an example, the project builds table shows all of the builds for a project in a ToasterTable. It makes sense to add the project to the context so that its name can be shown at the top of the page. For a URL like http://localhost:8000/toastergui/project/X/builds, Toaster assigns the "X" in the URL to a parameter called ''pid''; this can be retrieved in get_context_data() and used to fetch the project for which we are showing builds:

<pre>
class ProjectBuildsTable(ToasterTable):
# ... other methods ...

def get_context_data(self, **kwargs):
context = super(ProjectBuildsTable, self).get_context_data(**kwargs)

# use the pid parameter extracted from the URL
context['project'] = Project.objects.get(pk=kwargs['pid'])

return context
</pre>

This can then be referenced the usual way in the template (e.g. <code>{{project}}</code>).

=== Search ===

Search is automatically enabled on a ToasterTable. However, you may find that you are unable to search on the fields you would like to. This section explains how to fix that.

By default, ToasterTable searches against the queryset's model, using an OR query with icontains (case insensitive, contains string) matching. The fields used for the search are defined by the search_allowed_fields property of the model.

For MiniBuildsTable, the queryset's model is orm.models.Build; search_allowed_fields (at the time of writing) for Build is set to:

<pre>
['machine', 'cooker_log_path', 'target__target', 'target__target_image_file__file_name']
</pre>

Doing a search for a string like "test" would therefore construct a query OR clause like the following (pseudo-SQL):

<pre>
... WHERE machine LIKE '%test%' OR cooker_log_path LIKE '%test%' OR target.target LIKE '%test%'
OR target.target_image_file.file_name LIKE '%test%'
</pre>

(The actual SQL clause is far more complicated, as Django would have to use various JOIN statements to combine rows from multiple database tables.)

Note that the search_allowed_fields don't have to be fields of the model: they can be fields in related models (here, the Target and TargetImageFile models related to a Build).

This means that it's not possible to search by project name in the MiniBuildsTable. To make this possible, we could modify search_allowed_fields for the Build model to:

<pre>
['machine', 'cooker_log_path', 'target__target', 'target__target_image_file__file_name', 'project__name']
</pre>

This would now allow the Mini Builds table to be searched by project name as well.

== Filter API ==

The classes for creating filters are defined in tablefilter.py.

A filter applies to a single column in the table. Any actions on a filter should filter records according to criteria which make sense with respect to that column. For example, you wouldn't add a filter to the "Completed on" column of the "All Builds" table which filters builds according to whether they failed or succeeded; but you would add a filter which filters the builds based on the time when the build was completed.

A filter adds an option to the column heading to filter in/out particular records or to remove filtering altogether.

Each filter has one or more actions associated with it. A filter also automatically gets a default action which turns off the filter and shows all the records again.

Each action changes the records shown in its associated ToasterTable, by applying a set of criteria to the query used to fetch records (e.g. show records for a single date, for a date range, or matching/not matching some other criteria related to the column).

To add a filter to a column, do the following:

# Create a TableFilter object for the column.
# Assign the filter to the column in the ToasterTable.
# Attach TableFilterAction objects to the TableFilter.

These steps are explained in detail below.

=== Add a table filter ===

The TableFilter class acts as a container for a group of TableFilterAction objects.

To create a TableFilter, instantiate it with a unique identifier (unique to the ToasterTable) and a text string which is displayed in the popup for the filter. The filters should be defined in the setup_filters() method for the ToasterTable.

Once the filter is defined, add it to the ToasterTable using the add_filter() method.

Here's an example of a TableFilter for the outcome column of MiniBuildsTable:

<pre>
# already imported in tables.py, but shown here for completeness
from toastergui.tablefilter import TableFilter

class MiniBuildsTable(ToasterTable):
# ... other methods ...

def setup_filters(self, *args, **kwargs):
# filter by outcome (succeeded or failed)
outcome_filter = TableFilter(
'outcome_filter',
'Filter builds by outcome'
)

# add the filter to the ToasterTable
self.add_filter(outcome_filter)
</pre>

Internally, add_filter() uses a TableFilterMap in the ToasterTable, which maps from column names to TableFilters.

Note that the filter has no actions, so won't actually filter the table yet.

=== Associate a filter with a column ===

To associate the filter with a column, pass the name of the filter in the filter_name argument for the column. For example, to associate the outcome_filter defined above with the outcome column, change setup_columns() like this:

<pre>
class MiniBuildsTable(ToasterTable):
# ... other methods ...

def setup_columns(self, *args, **kwargs):
# ... other column definitions ...

self.add_column(title='Outcome',
help_text='The outcome of the build',
hideable=True,
orderable=True,
static_data_name='outcome',
static_data_template=outcome_template,
filter_name='outcome_filter')
</pre>

(Note the additional filter_name keyword passed to add_column().)

Now if you visit http://localhost:8000/toastergui/minibuilds, the outcome column in the table should show the filter icon; clicking on this opens the popup which allows a user to select the filter to apply. Because we have no actions on the filter, the only option is the "All" one:

[[File:MiniBuildsTable-outcomefilter.png|750px|Adding an outcome filter to the MiniBuildsTable]]

outcome_filter is the string used to refer to the filter in the querystring; see the section [[#How filters are applied|How filters are applied]] for more details about how filters are applied.

=== Add table filter actions ===

The next step is to add actions to the filter.

TableFilterAction is the base class for an action associated with a filter. The following types of filter action (subclasses of TableFilterAction) are already implemented:

* TableFilterActionToggle: filter the records shown in the table by some arbitrary criteria; the action is either on or off.
* TableFilterActionDay: filter the records shown by day (yesterday or today).
* TableFilterActionDateRange: filter the records shown by a from/to date range.

In the case of TableFilterActionDay and TableFilterActionDateRange, you specify the field which is used for the filter action. In the case of TableFilterActionToggle, you can use arbitrary criteria for the action.

To add an action to a filter, create an instance of the desired action class and use the TableFilter.add_action() method to associate it with a filter. For example, here's how to add two actions to the outcome_filter defined earlier: one to show successful builds, and the other to show failed builds:

<pre>
# already imported in tables.py, but shown here for completeness
from toastergui.tablefilter import TableFilter
from toastergui.tablefilter import TableFilterActionToggle

class MiniBuildsTable(ToasterTable):
# ... other methods ...

def setup_filters(self, *args, **kwargs):
# filter by outcome (succeeded or failed)
outcome_filter = TableFilter(
'outcome_filter',
'Filter builds by outcome'
)

successful_builds_action = TableFilterActionToggle(
'successful_builds',
'Successful builds',
Q(outcome=Build.SUCCEEDED)
)

failed_builds_action = TableFilterActionToggle(
'failed_builds',
'Failed builds',
Q(outcome=Build.FAILED)
)

outcome_filter.add_action(successful_builds_action)
outcome_filter.add_action(failed_builds_action)

# add the filter to the ToasterTable
self.add_filter(outcome_filter)
</pre>

Here's how this will render:

[[File:MiniBuildsTable-outcomefilteractions.png|750px|Adding outcome filter actions to the MiniBuildsTable]]

I'll break this down further to give a bit more detail about how the filter action is created. Here's the code which creates the filter action to show successful builds only:

<pre>
successful_builds_action = TableFilterActionToggle(
'successful_builds',
'Successful builds',
Q(outcome=Build.SUCCEEDED)
)
</pre>

The arguments to TableFilterActionToggle have the following meanings:
* 'successful_builds' is the name of the action. This is used to map from the filter parameter in the querystring (see the next section).
* 'Successful builds' is the label shown next to the radio button which activates this action in the popup.
* Q(outcome=Build.SUCCEEDED) shows the criteria used to filter the records in the table's queryset when the filter action is applied. The [https://docs.djangoproject.com/en/1.9/topics/db/queries/#complex-lookups-with-q Q object] is part of the Django API; it should reference field names which are present in the queryset to be filtered and can use any criteria available to Q objects. In the case of the MiniBuildsTable, the queryset consists of Build objects; the Q(outcome=Build.SUCCEEDED) object is used to filter this queryset, so we effectively get the result of Build.objects.all().filter(Q(outcome=Build.SUCCEEDED)) when the action is applied.

For examples of how to add TableFilterActionDay and TableFilterActionDateRange filter actions, see the BuildsTable class in tables.py.

The next section explains how we go from clicking on a radio button in the filter popup to filtering the records shown in the table.

=== How filters are applied ===

The short version:

When a ToasterTable is rendered, a filter icon is shown on any column which has a filter_name defined for it. Clicking on this icon populates the filter dialog, then opens it so the user can select a filter to apply. When the user clicks on the "Apply" button in the dialog, new data for the table is requested, using the filter name, filter action, and filter value from the querystring. These parameters are used to filter the records which are shown in the table.

The long version:

* A filter icon for a column has a filter name associated with it.
* When a filter icon is clicked, the Toaster UI makes an Ajax request to the Toaster back-end for data about that filter name.
* When the filter data is received (in JSON format), the filter dialog is populated with radio buttons (one per action), labels (one per action) and any additional fields (e.g. date range fields for TableFilterActionDateRange actions). The count of records which will be returned by an action is part of the data returned by the back-end; if this is 0, the label and radio button are disabled. See static/js/table.js for the code which populates the filter dialog.
* The dialog is opened so the user can choose a filter to apply. The user clicks radio buttons, fills in fields etc.
* When the user clicks on the "Apply" button, the URL for the page is modified in place to reflect the filter criteria. The filter is represented in the URL by two querystring parameters:
** <code>filter=<filter name>:<filter action></code> : the filter name maps to the name used when creating the TableFilter object; and the filter action corresponds to one of the names of a TableFilterAction object added to the TableFilter. For example, ''filter=outcome_filter:successful_builds'' will map to the ''outcome_filter'' TableFilter and its ''successful_builds'' TableFilterAction.
** <code>filter_value=<filter value string></code> : for a TableFilterActionToggle filter, this is always "on", to show that the filter is applied; for a TableFilterActionDay this is either "today" or "yesterday"; for a TableFilterActionDateRange, this is a "from,to" date range in the format "2015-12-09,2015-12-11".
* The table data is fetched via Ajax, using the filter and filter_value parameters as part of the URL. These set the filter name, action and value to use for filtering.
* The back-end applies the requested filter action to the queryset (as well as any existing search string). Each filter action has a set of criteria which it applies to the queryset, as follows:
** TableFilterActionToggle: The recordset is filtered by the criteria specified when the action is created (see [[#Add table filter actions|Add table filter actions]] for an example).
** TableFilterActionDay: A date range clause is constructed at the time the filter action is applied. For example, if the field the filter action applies to is "completed_on", the day set for the TableFilterActionDay is "today", and today is 2016-03-05, the query clause (in pseudo-SQL) is "completed_on >= '2016-03-04 00:00:00' AND completed_on <= '2016-03-04 23:59:59'".
** TableFilterActionDateRange: This is similar to TableFilterActionDay, but the user specifies the start and end dates; these are in the filter_value variable in the querystring. For example, if the field the filter action applies to is "completed_on" and the date range is "2016-03-01,2016-03-04", the query clause (in pseudo-SQL) is "completed_on >= '2016-03-01 00:00:00' AND completed_on <= '2016-03-04 23:59:59'".
* The filtered queryset is returned as JSON.
* The Toaster UI code redraws the table with the filtered queryset.

Setting up a production instance of Toaster

2016-05-03T17:13:02Z

Michael Wood:

[[Category:Toaster]]
----
{| style="color:black; background-color:#ffffcc" width="100%" cellpadding="10" class="wikitable"
|This page is the development version of the documentation to provide the latest information, if you're using a release please refer to [https://www.yoctoproject.org/documentation/archived the published manual]'''
|}
----
A production instance of Toaster is one in which you wish to share the Toaster instance with remote and multiple users. It is also the setup which can cope with heavier loads on the web service. These instructions setup toaster in Build mode where builds and projects are run, viewed and defined by the Toaster web interface.

== Requirements ==
* [http://www.yoctoproject.org/docs/2.0/yocto-project-qs/yocto-project-qs.html#packages Build requirements]
* Apache webserver
* mod-wsgi for Apache webserver
* Mysql database server

Ubuntu 14.04.3:
<code>
$ sudo apt-get install apache2 libapache2-mod-wsgi mysql-server python-virtualenv libmysqlclient-dev python-dev python-mysqldb
</code>

Fedora 22/RH:
<code>
$ sudo dnf install httpd mod_wsgi python-virtualenv gcc mysql-devel
</code>

== Installation ==

'''1.''' Checkout a copy of Poky into the web server directory. We're going to be using /var/www/toaster.
<code>
$ mkdir -p /var/www/toaster
$ cd /var/www/toaster/
$ git clone git://git.yoctoproject.org/poky
$ cd poky
$ git checkout jethro # change for any release name required
</code>

'''2.''' Initialise a virtualenv and install Toaster dependencies. (Use virtualenv to keep the python packages isolated from your system provided packages - not required but recommended, alternative use your OS's package manager to install the packages)

<code>
$ cd /var/www/toaster/
$ virtualenv venv
$ source ./venv/bin/activate
$ pip install -r ./poky/bitbake/toaster-requirements.txt
$ pip install mysqlclient
</code>

'''3.''' Configure toaster edit /var/www/toaster/poky/bitbake/lib/toaster/toastermain/settings.py

Edit the DATABASE settings:
<code>
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.mysql',
'NAME': 'toaster_data',
'USER': 'toaster',
'PASSWORD': 'yourpasswordhere',
'HOST': 'localhost',
'PORT': '3306',
}
</code>

Edit the [https://docs.djangoproject.com/en/1.8/howto/deployment/checklist/#secret-key SECRET_KEY]:
<code>
SECRET_KEY = 'YOUR SECRET RANDOM KEY HERE'
</code>

Edit the STATIC_ROOT:
<code>
STATIC_ROOT = '/var/www/toaster/static_files/'
</code>

'''4.'''' Now add the database and user to your mysql server that we just defined
<code>
$ mysql -u root -p
mysql> CREATE DATABASE toaster_data;
mysql> CREATE USER 'toaster'@'localhost' identified by 'yourpasswordhere';
mysql> GRANT all on toaster_data.* to 'toaster'@'localhost';
mysql> quit
</code>
n.b. You may want to decide on fewer [https://dev.mysql.com/doc/refman/5.1/en/grant.html privileges] to the toaster user.

'''5.''' Get toaster to create the database schema, default data, update the TOASTER_DIR which is the build work dir and collect up the statically served files

<code>
$ cd /var/www/toaster/poky/
$ ./bitbake/lib/toaster/manage.py syncdb
$ ./bitbake/lib/toaster/manage.py migrate
$ ./bitbake/lib/toaster/manage.py loadconf ./meta-yocto/conf/toasterconf.json
$ TOASTER_DIR=/var/www/toaster/poky/ ./bitbake/lib/toaster/manage.py checksettings
$ ./bitbake/lib/toaster/manage.py lsupdates
$ ./bitbake/lib/toaster/manage.py collectstatic
</code>

'''6.''' Add a config file for Toaster to your Apache web server's configurations available directory.

Ubuntu/Debian put it here: /etc/apache2/conf-available/toaster.conf
Fedora/RH usually here: /etc/httpd/conf.d/toaster.conf
<code>

Alias /static /var/www/toaster/static_files
<Directory /var/www/toaster/static_files>
Order allow,deny
Allow from all
Require all granted
</Directory>
WSGIDaemonProcess toaster_wsgi python-path=/var/www/toaster/poky/bitbake/lib/toaster:/var/www/toaster/venv/lib/python2.7/site-packages
WSGIScriptAlias / "/var/www/toaster/poky/bitbake/lib/toaster/toastermain/wsgi.py"
<Location />
WSGIProcessGroup toaster_wsgi
</Location>

</code>

In Ubuntu/Debain you will need to enable the config and module in Apache webserver
<code>
$ sudo a2enmod wsgi
$ sudo a2enconf toaster
</code>

Restart Apache web server to make sure all new configuration is loaded

Ubuntu/Debian:
<code>
$ sudo service apache2 restart
</code>

Fedora/RH:
<code>
$ sudo service httpd restart
</code>

'''7.''' Install the build runner service

This service needs to be running in order to dispatch builds the command that needs to be run is:

<code>
/var/www/toaster/poky/bitbake/lib/toaster/manage.py runbuilds
</code>

Sample script:
<code>
#!/bin/sh
# toaster run builds dispatcher
cd /var/www/toaster/
source ./venv/bin/activate
while true; do ./poky/bitbake/lib/toaster/manage.py runbuilds; sleep 3; done
</code>

N.b. You may wish to add a service entry to your OS's init system so that it starts up on start up as well as adding a dedicated user.

Now open up a browser and you can start using Toaster!

Toaster future release planning

2016-04-22T12:35:03Z

Michael Wood: /* Michael's */

[[Category:Toaster]]

== 2.2 Planning ==

=== Wish lists ===

==== Michael's ====
* Remove pseudo API cruft / template context to json response stuff
* Remove bldcontrol (we only need localhostbecontroller.py)
* Look at schema and rework where needed
* Consolidate used API into it's own view
* Finish toaster tables porting
* Create a toasterclient.py to interact with toaster to avoid need for 'command line builds'
** This would use toaster's API as the entry point to using bitbake
** It could be called by CI systems
** It could be used by auto builders etc
** Means that there are no two modes anywhere in Toaster
** Allows project configuration to be changed in toaster from the command line
* Drop supporting old releases

==== Brian's ====
The Ordered Part
# Django 1.8 aka LTS
# Image Customization
# Testing and Stability
# One way to start/One code path
<br>
This is *not* ordered! This is more of a brain dump :)
* Collapse analysis and managed mode
* Working CI for each commit into toaster-next as well as for project peoples testME branches on poky-contrib:
** run django tests
** run selenium tests
** leverage sstate to make runs generally shorter
* remove git assumptions from code
* rationalize/simplify the configuration scripts
* if bitbake-memres comes out, interact with it gracefully (this is my preferred method of collapsing analysis and managed)
* fix the tmpdir/release build issue &/or discuss the need to support building old releases with a UI rather than limiting toaster to build the release it is part of. This is inherently fragile.
* ask bb to parse any layer added to a project and such as imported layers, and layers specified in the configuration file to give us recipe information.
* improve our build event following:
** note parse events
** update/retrieve ui so that we can tell something is happening before the first build event comes
** eliminate the remaining random tracebacks from try/except failures.
* allow configuration in addition to the addition/removal of layers/packages
** kernel config
* add user control/authentication
* update to Django 1.8. Always stay on Django LTS versions
* add layer update from layers.openembedded.org to the UI
* add Django tests for building e.g. we should be able to build core-image-sato from a django test and validate the db
* add mysql script to simplify setup with mysql/apache
* pull in new look and feel
* add django tests to validate all artifacts - if we say they can download it and it appears on a page anywhere, it should have a test
* improve docs of code on wiki. We have a nice start (ty E & M); I'd like to improve it.
** events - more info and list types supported
** what logic is in jscript , what is in python
** database schema definition and description esp interrelationships.
** list django tests, mostly so we can see what we are missing.
* build mode triggerred off of database entry, not autonomous process.
* add asynchronicity. I have noted a number of places where commands result in a sluggish ui. I'm nervous about this one.
** for example, starting a build that requires us to clone layers has a noticeable delay as compared to starting a local only project build.
* verify we can stop builds cleanly.
* get the rest of our logs into our local toaster dir
* allow us to run and stop 2 toasters on the same machine without them interfering with each other
* (2.2 or later for sure) discuss how to support multiple back end bb servers.
** probably will need to be able to add servers/remove servers/see server status/build state
* be able to see more details about build. For example, see the "jobs" being processed like you can in knotty
* revisit what variables we let people change. sstate_dir and dl_dir should certainly be allowed to change
* discuss way we do migrations. possibly change. thoughts:
** only support migrations from last release to current. (e.g. if you were running toaster jethro you could update to jethro++, but could not from fido to jethro)
** take migration from the startup script. it's only for releasetorelease.
*** allow it to be in for master and ease of development w/in a release and pull at end as we do with moving releases.
** whatever we decide, add a test set for it so we know it works
* Be able to delete things from the ui
** old builds
** old projects
** imported layers we no longer want
* Be able to set from the ui a policy to delete all builds/recipes/packages that are older than XXX &/or be able to set maximum size on db and delete everything older than XXX when we near that size
* update the builds on the web page so we dont have to hit refresh to see the build progress.

==== Elliot's ====

* Make sure all files lint properly
* Include lint in CI builds
* Improve test coverage of all areas of the UI and back-end
* Clean up toasterui.py so it's easier to see what's going on in there
* Fix localhostbecontroller so it doesn't rely on what's in the log file when figuring out whether the bitbake server has started
* Provide a way to modify settings in settings.py without editing that file (e.g. with a local overlay configuration file)
* Move the logic, environment variables, directory setup and script calls from bin/toaster into Django commands: this would allow you to run toaster correctly, with all the required setup, without having to use bin/toaster; it would also mean that we could provide cleaner production setup instructions, as a user could run a series of Django commands (with their own parameters) to setup their environment; bin/toaster would then just execute a series of Django commands

==== Belén's ====

* Implement new custom theme (already in progress, aiming for end of M1: Dec 7th). Related bugs: 8417
* Finish image customisation. Related bugs: 8070, 8081, 8082, 8103, 8104, 8117, 8128, 8132, 8091
* Review interaction with layers: as it is, imposes heavy restrictions on users. Some of the questions I would like to consider:
** Should the import layer action be global instead of per project, so that you can reuse imported layers across projects?
** Should we allow users to override the release layer branch, setting a particular layer index layer to a different branch or commit?
** Should we allow users to set a different git repo for layer index layers?
** Should we allow users to override the bitbake revision set by the release in a project?
** Related bugs: 6640, 6701, 7574, 8426, 8429
* Access control (a simple one to start, with very basic permissions). Related bugs: 6233, 6234
* WIC integration (getting a step closer to the holy grail of 'click button' > 'image deployed in board')
* Fix the variable history (this is one of the features users have highly praised, and it is a bit broken in 2.0). Related bugs: 5811, 8488, 8190, 8189, 8188
* Get Toaster to collect build data from Jenkins builds. Once again, repeatedly requested by users. Related bugs: 7527

==== Ed's ====

* Merge analysis and managed modes
* Fix as much pending bugs as possible.
* Finish image customisation.
* Support Django LTS
* Increase test coverage. It would be good to set realistic goals, e.g. cover specific modules or increase coverage to 10% and follow the plan.
* Include production configuration into testing
* Cover views.py with tests and split it into modules. Currently it's huge and hard to maintain. I suspect there are a lot of unused APIs in there.
* Finish refactoring of buildcontroller code. This work is started in ed/toaster/bec and sitting there for quite long time.
* Bring patchset support to toaster patchwork.
* Add 'submitted upstream' and 'merged upstream' statuses to patchwork. Make the statuses updated automatically when patch is accepted to toaster-next or bitbake master.
* Clean up the code. Set realistic goals for pylint scores and follow them.
* Enhance Jenkins CI jobs to run all possible django, tts and pylint tests

==== Sujith's ====

* Get toaster working with non-git repos. Related bugs: 8456
* minor, but would be good to have :- Once user launches toaster, it would be nice to see the progress bar updated without user refreshing the page. Related bugs: 8328
* Cancellation of build. Related bugs: 6787

Would love to take up more tasks.

==== David's ====

* Finish the effort around Toaster "extensions", started by Farrell. This will enable Toaster customization for commercial partners, as per branding and
alternate pages (for extended project creation).

== All bugs for 2.2 ==

{{#bugzilla:
|columns=id,milestone,summary,status,priority
|milestone=future,2.1,2.1 M1,2.1 M2,2.1 M3,2.1 M4,2.2,2.2 M1,2.2 M2,2.2 M3,2.2 M4
|sort=priority
|product=Toaster
|resolution=!fixed
|status=!IN PROGRESS REVIEW
}}

== Toaster Enhancements for 2.2 ==
{{#bugzilla:
|columns=id,to,estimated,summary,priority,from,qa,milestone,status,resolution,whiteboard
|milestone=2.2,2.2 M1,2.2 M2,2.2 M3,2.2 M4
|sort=priority
|severity=enhancement
|product=Toaster
|status=!verified
}}

Toaster future release planning

2016-04-20T16:54:52Z

Michael Wood:

[[Category:Toaster]]

== 2.2 Planning ==

=== Wish lists ===

==== Michael's ====
* Remove pseudo API cruft / template context to json response stuff
* Look at schema and rework where needed
* Consolidate used API into it's own view
* Finish toaster tables porting
* Create a toasterclient.py to interact with toaster to avoid need for 'command line builds'
** This would use toaster's API as the entry point to using bitbake
** It could be called by CI systems
** It could be used by auto builders etc
** Means that there are no two modes anywhere in Toaster
** Allows project configuration to be changed in toaster from the command line
* Drop supporting old releases

==== Brian's ====
The Ordered Part
# Django 1.8 aka LTS
# Image Customization
# Testing and Stability
# One way to start/One code path
<br>
This is *not* ordered! This is more of a brain dump :)
* Collapse analysis and managed mode
* Working CI for each commit into toaster-next as well as for project peoples testME branches on poky-contrib:
** run django tests
** run selenium tests
** leverage sstate to make runs generally shorter
* remove git assumptions from code
* rationalize/simplify the configuration scripts
* if bitbake-memres comes out, interact with it gracefully (this is my preferred method of collapsing analysis and managed)
* fix the tmpdir/release build issue &/or discuss the need to support building old releases with a UI rather than limiting toaster to build the release it is part of. This is inherently fragile.
* ask bb to parse any layer added to a project and such as imported layers, and layers specified in the configuration file to give us recipe information.
* improve our build event following:
** note parse events
** update/retrieve ui so that we can tell something is happening before the first build event comes
** eliminate the remaining random tracebacks from try/except failures.
* allow configuration in addition to the addition/removal of layers/packages
** kernel config
* add user control/authentication
* update to Django 1.8. Always stay on Django LTS versions
* add layer update from layers.openembedded.org to the UI
* add Django tests for building e.g. we should be able to build core-image-sato from a django test and validate the db
* add mysql script to simplify setup with mysql/apache
* pull in new look and feel
* add django tests to validate all artifacts - if we say they can download it and it appears on a page anywhere, it should have a test
* improve docs of code on wiki. We have a nice start (ty E & M); I'd like to improve it.
** events - more info and list types supported
** what logic is in jscript , what is in python
** database schema definition and description esp interrelationships.
** list django tests, mostly so we can see what we are missing.
* build mode triggerred off of database entry, not autonomous process.
* add asynchronicity. I have noted a number of places where commands result in a sluggish ui. I'm nervous about this one.
** for example, starting a build that requires us to clone layers has a noticeable delay as compared to starting a local only project build.
* verify we can stop builds cleanly.
* get the rest of our logs into our local toaster dir
* allow us to run and stop 2 toasters on the same machine without them interfering with each other
* (2.2 or later for sure) discuss how to support multiple back end bb servers.
** probably will need to be able to add servers/remove servers/see server status/build state
* be able to see more details about build. For example, see the "jobs" being processed like you can in knotty
* revisit what variables we let people change. sstate_dir and dl_dir should certainly be allowed to change
* discuss way we do migrations. possibly change. thoughts:
** only support migrations from last release to current. (e.g. if you were running toaster jethro you could update to jethro++, but could not from fido to jethro)
** take migration from the startup script. it's only for releasetorelease.
*** allow it to be in for master and ease of development w/in a release and pull at end as we do with moving releases.
** whatever we decide, add a test set for it so we know it works
* Be able to delete things from the ui
** old builds
** old projects
** imported layers we no longer want
* Be able to set from the ui a policy to delete all builds/recipes/packages that are older than XXX &/or be able to set maximum size on db and delete everything older than XXX when we near that size
* update the builds on the web page so we dont have to hit refresh to see the build progress.

==== Elliot's ====

* Make sure all files lint properly
* Include lint in CI builds
* Improve test coverage of all areas of the UI and back-end
* Clean up toasterui.py so it's easier to see what's going on in there
* Fix localhostbecontroller so it doesn't rely on what's in the log file when figuring out whether the bitbake server has started
* Provide a way to modify settings in settings.py without editing that file (e.g. with a local overlay configuration file)
* Move the logic, environment variables, directory setup and script calls from bin/toaster into Django commands: this would allow you to run toaster correctly, with all the required setup, without having to use bin/toaster; it would also mean that we could provide cleaner production setup instructions, as a user could run a series of Django commands (with their own parameters) to setup their environment; bin/toaster would then just execute a series of Django commands

==== Belén's ====

* Implement new custom theme (already in progress, aiming for end of M1: Dec 7th). Related bugs: 8417
* Finish image customisation. Related bugs: 8070, 8081, 8082, 8103, 8104, 8117, 8128, 8132, 8091
* Review interaction with layers: as it is, imposes heavy restrictions on users. Some of the questions I would like to consider:
** Should the import layer action be global instead of per project, so that you can reuse imported layers across projects?
** Should we allow users to override the release layer branch, setting a particular layer index layer to a different branch or commit?
** Should we allow users to set a different git repo for layer index layers?
** Should we allow users to override the bitbake revision set by the release in a project?
** Related bugs: 6640, 6701, 7574, 8426, 8429
* Access control (a simple one to start, with very basic permissions). Related bugs: 6233, 6234
* WIC integration (getting a step closer to the holy grail of 'click button' > 'image deployed in board')
* Fix the variable history (this is one of the features users have highly praised, and it is a bit broken in 2.0). Related bugs: 5811, 8488, 8190, 8189, 8188
* Get Toaster to collect build data from Jenkins builds. Once again, repeatedly requested by users. Related bugs: 7527

==== Ed's ====

* Merge analysis and managed modes
* Fix as much pending bugs as possible.
* Finish image customisation.
* Support Django LTS
* Increase test coverage. It would be good to set realistic goals, e.g. cover specific modules or increase coverage to 10% and follow the plan.
* Include production configuration into testing
* Cover views.py with tests and split it into modules. Currently it's huge and hard to maintain. I suspect there are a lot of unused APIs in there.
* Finish refactoring of buildcontroller code. This work is started in ed/toaster/bec and sitting there for quite long time.
* Bring patchset support to toaster patchwork.
* Add 'submitted upstream' and 'merged upstream' statuses to patchwork. Make the statuses updated automatically when patch is accepted to toaster-next or bitbake master.
* Clean up the code. Set realistic goals for pylint scores and follow them.
* Enhance Jenkins CI jobs to run all possible django, tts and pylint tests

==== Sujith's ====

* Get toaster working with non-git repos. Related bugs: 8456
* minor, but would be good to have :- Once user launches toaster, it would be nice to see the progress bar updated without user refreshing the page. Related bugs: 8328
* Cancellation of build. Related bugs: 6787

Would love to take up more tasks.

==== David's ====

* Finish the effort around Toaster "extensions", started by Farrell. This will enable Toaster customization for commercial partners, as per branding and
alternate pages (for extended project creation).

== All bugs for 2.2 ==

{{#bugzilla:
|columns=id,milestone,summary,status,priority
|milestone=future,2.1,2.1 M1,2.1 M2,2.1 M3,2.1 M4,2.2,2.2 M1,2.2 M2,2.2 M3,2.2 M4
|sort=priority
|product=Toaster
|resolution=!fixed
|status=!IN PROGRESS REVIEW
}}

== Toaster Enhancements for 2.2 ==
{{#bugzilla:
|columns=id,to,estimated,summary,priority,from,qa,milestone,status,resolution,whiteboard
|milestone=2.2,2.2 M1,2.2 M2,2.2 M3,2.2 M4
|sort=priority
|severity=enhancement
|product=Toaster
|status=!verified
}}

Setting up a production instance of Toaster

2016-04-20T13:22:08Z

Michael Wood: /* Installation */

[[Category:Toaster]]
----
'''This page is the development version of the documentation to provide the latest information, if you're using a release please refer to [https://www.yoctoproject.org/documentation/archived the published manual]'''
----
A production instance of Toaster is one in which you wish to share the Toaster instance with remote and multiple users. It is also the setup which can cope with heavier loads on the web service. These instructions setup toaster in Build mode where builds and projects are run, viewed and defined by the Toaster web interface.

== Requirements ==
* [http://www.yoctoproject.org/docs/2.0/yocto-project-qs/yocto-project-qs.html#packages Build requirements]
* Apache webserver
* mod-wsgi for Apache webserver
* Mysql database server

Ubuntu 14.04.3:
<code>
$ sudo apt-get install apache2 libapache2-mod-wsgi mysql-server python-virtualenv libmysqlclient-dev python-dev python-mysqldb
</code>

Fedora 22/RH:
<code>
$ sudo dnf install httpd mod_wsgi python-virtualenv gcc mysql-devel
</code>

== Installation ==

'''1.''' Checkout a copy of Poky into the web server directory. We're going to be using /var/www/toaster.
<code>
$ mkdir -p /var/www/toaster
$ cd /var/www/toaster/
$ git clone git://git.yoctoproject.org/poky
$ cd poky
$ git checkout jethro # change for any release name required
</code>

'''2.''' Initialise a virtualenv and install Toaster dependencies. (Use virtualenv to keep the python packages isolated from your system provided packages - not required but recommended, alternative use your OS's package manager to install the packages)

<code>
$ cd /var/www/toaster/
$ virtualenv venv
$ source ./venv/bin/activate
$ pip install -r ./poky/bitbake/toaster-requirements.txt
$ pip install mysqlclient
</code>

'''3.''' Configure toaster edit /var/www/toaster/poky/bitbake/lib/toaster/toastermain/settings.py

Edit the DATABASE settings:
<code>
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.mysql',
'NAME': 'toaster_data',
'USER': 'toaster',
'PASSWORD': 'yourpasswordhere',
'HOST': 'localhost',
'PORT': '3306',
}
</code>

Edit the [https://docs.djangoproject.com/en/1.8/howto/deployment/checklist/#secret-key SECRET_KEY]:
<code>
SECRET_KEY = 'YOUR SECRET RANDOM KEY HERE'
</code>

Edit the STATIC_ROOT:
<code>
STATIC_ROOT = '/var/www/toaster/static_files/'
</code>

'''4.'''' Now add the database and user to your mysql server that we just defined
<code>
$ mysql -u root -p
mysql> CREATE DATABASE toaster_data;
mysql> CREATE USER 'toaster'@'localhost' identified by 'yourpasswordhere';
mysql> GRANT all on toaster_data.* to 'toaster'@'localhost';
mysql> quit
</code>
n.b. You may want to decide on fewer [https://dev.mysql.com/doc/refman/5.1/en/grant.html privileges] to the toaster user.

'''5.''' Get toaster to create the database schema, default data, update the TOASTER_DIR which is the build work dir and collect up the statically served files

<code>
$ cd /var/www/toaster/poky/
$ ./bitbake/lib/toaster/manage.py syncdb
$ ./bitbake/lib/toaster/manage.py migrate
$ ./bitbake/lib/toaster/manage.py loadconf ./meta-yocto/conf/toasterconf.json
$ TOASTER_DIR=/var/www/toaster/poky/ ./bitbake/lib/toaster/manage.py checksettings
$ ./bitbake/lib/toaster/manage.py lsupdates
$ ./bitbake/lib/toaster/manage.py collectstatic
</code>

'''6.''' Add a config file for Toaster to your Apache web server's configurations available directory.

Ubuntu/Debian put it here: /etc/apache2/conf-available/toaster.conf
Fedora/RH usually here: /etc/httpd/conf.d/toaster.conf
<code>

Alias /static /var/www/toaster/static_files
<Directory /var/www/toaster/static_files>
Order allow,deny
Allow from all
Require all granted
</Directory>
WSGIDaemonProcess toaster_wsgi python-path=/var/www/toaster/poky/bitbake/lib/toaster:/var/www/toaster/venv/lib/python2.7/site-packages
WSGIScriptAlias / "/var/www/toaster/poky/bitbake/lib/toaster/toastermain/wsgi.py"
<Location />
WSGIProcessGroup toaster_wsgi
</Location>

</code>

In Ubuntu/Debain you will need to enable the config and module in Apache webserver
<code>
$ sudo a2enmod wsgi
$ sudo a2enconf toaster
</code>

Restart Apache web server to make sure all new configuration is loaded

Ubuntu/Debian:
<code>
$ sudo service apache2 restart
</code>

Fedora/RH:
<code>
$ sudo service httpd restart
</code>

'''7.''' Install the build runner service

This service needs to be running in order to dispatch builds the command that needs to be run is:

<code>
/var/www/toaster/poky/bitbake/lib/toaster/manage.py runbuilds
</code>

Sample script:
<code>
#!/bin/sh
# toaster run builds dispatcher
cd /var/www/toaster/
source ./venv/bin/activate
while true; do ./poky/bitbake/lib/toaster/manage.py runbuilds; sleep 3; done
</code>

N.b. You may wish to add a service entry to your OS's init system so that it starts up on start up as well as adding a dedicated user.

Now open up a browser and you can start using Toaster!

Toaster future release planning

2016-04-20T11:21:27Z

Michael Wood: /* All bugs for 2.1+ */

[[Category:Toaster]]

== 2.2 Planning ==

=== Wish lists ===

==== Michael's ====
* Remove pseudo API cruft / template context to json response stuff
* Look at schema and rework where needed
* Consolidate used API into it's own view
* Finish toaster tables porting
* Create a toasterclient.py to interact with toaster to avoid need for 'command line builds'
** This would use toaster's API as the entry point to using bitbake
** It could be called by CI systems
** It could be used by auto builders etc
** Means that there are no two modes anywhere in Toaster
** Allows project configuration to be changed in toaster from the command line

==== Brian's ====
The Ordered Part
# Django 1.8 aka LTS
# Image Customization
# Testing and Stability
# One way to start/One code path
<br>
This is *not* ordered! This is more of a brain dump :)
* Collapse analysis and managed mode
* Working CI for each commit into toaster-next as well as for project peoples testME branches on poky-contrib:
** run django tests
** run selenium tests
** leverage sstate to make runs generally shorter
* remove git assumptions from code
* rationalize/simplify the configuration scripts
* if bitbake-memres comes out, interact with it gracefully (this is my preferred method of collapsing analysis and managed)
* fix the tmpdir/release build issue &/or discuss the need to support building old releases with a UI rather than limiting toaster to build the release it is part of. This is inherently fragile.
* ask bb to parse any layer added to a project and such as imported layers, and layers specified in the configuration file to give us recipe information.
* improve our build event following:
** note parse events
** update/retrieve ui so that we can tell something is happening before the first build event comes
** eliminate the remaining random tracebacks from try/except failures.
* allow configuration in addition to the addition/removal of layers/packages
** kernel config
* add user control/authentication
* update to Django 1.8. Always stay on Django LTS versions
* add layer update from layers.openembedded.org to the UI
* add Django tests for building e.g. we should be able to build core-image-sato from a django test and validate the db
* add mysql script to simplify setup with mysql/apache
* pull in new look and feel
* add django tests to validate all artifacts - if we say they can download it and it appears on a page anywhere, it should have a test
* improve docs of code on wiki. We have a nice start (ty E & M); I'd like to improve it.
** events - more info and list types supported
** what logic is in jscript , what is in python
** database schema definition and description esp interrelationships.
** list django tests, mostly so we can see what we are missing.
* build mode triggerred off of database entry, not autonomous process.
* add asynchronicity. I have noted a number of places where commands result in a sluggish ui. I'm nervous about this one.
** for example, starting a build that requires us to clone layers has a noticeable delay as compared to starting a local only project build.
* verify we can stop builds cleanly.
* get the rest of our logs into our local toaster dir
* allow us to run and stop 2 toasters on the same machine without them interfering with each other
* (2.2 or later for sure) discuss how to support multiple back end bb servers.
** probably will need to be able to add servers/remove servers/see server status/build state
* be able to see more details about build. For example, see the "jobs" being processed like you can in knotty
* revisit what variables we let people change. sstate_dir and dl_dir should certainly be allowed to change
* discuss way we do migrations. possibly change. thoughts:
** only support migrations from last release to current. (e.g. if you were running toaster jethro you could update to jethro++, but could not from fido to jethro)
** take migration from the startup script. it's only for releasetorelease.
*** allow it to be in for master and ease of development w/in a release and pull at end as we do with moving releases.
** whatever we decide, add a test set for it so we know it works
* Be able to delete things from the ui
** old builds
** old projects
** imported layers we no longer want
* Be able to set from the ui a policy to delete all builds/recipes/packages that are older than XXX &/or be able to set maximum size on db and delete everything older than XXX when we near that size
* update the builds on the web page so we dont have to hit refresh to see the build progress.

==== Elliot's ====

* Make sure all files lint properly
* Include lint in CI builds
* Improve test coverage of all areas of the UI and back-end
* Clean up toasterui.py so it's easier to see what's going on in there
* Fix localhostbecontroller so it doesn't rely on what's in the log file when figuring out whether the bitbake server has started
* Provide a way to modify settings in settings.py without editing that file (e.g. with a local overlay configuration file)
* Move the logic, environment variables, directory setup and script calls from bin/toaster into Django commands: this would allow you to run toaster correctly, with all the required setup, without having to use bin/toaster; it would also mean that we could provide cleaner production setup instructions, as a user could run a series of Django commands (with their own parameters) to setup their environment; bin/toaster would then just execute a series of Django commands

==== Belén's ====

* Implement new custom theme (already in progress, aiming for end of M1: Dec 7th). Related bugs: 8417
* Finish image customisation. Related bugs: 8070, 8081, 8082, 8103, 8104, 8117, 8128, 8132, 8091
* Review interaction with layers: as it is, imposes heavy restrictions on users. Some of the questions I would like to consider:
** Should the import layer action be global instead of per project, so that you can reuse imported layers across projects?
** Should we allow users to override the release layer branch, setting a particular layer index layer to a different branch or commit?
** Should we allow users to set a different git repo for layer index layers?
** Should we allow users to override the bitbake revision set by the release in a project?
** Related bugs: 6640, 6701, 7574, 8426, 8429
* Access control (a simple one to start, with very basic permissions). Related bugs: 6233, 6234
* WIC integration (getting a step closer to the holy grail of 'click button' > 'image deployed in board')
* Fix the variable history (this is one of the features users have highly praised, and it is a bit broken in 2.0). Related bugs: 5811, 8488, 8190, 8189, 8188
* Get Toaster to collect build data from Jenkins builds. Once again, repeatedly requested by users. Related bugs: 7527

==== Ed's ====

* Merge analysis and managed modes
* Fix as much pending bugs as possible.
* Finish image customisation.
* Support Django LTS
* Increase test coverage. It would be good to set realistic goals, e.g. cover specific modules or increase coverage to 10% and follow the plan.
* Include production configuration into testing
* Cover views.py with tests and split it into modules. Currently it's huge and hard to maintain. I suspect there are a lot of unused APIs in there.
* Finish refactoring of buildcontroller code. This work is started in ed/toaster/bec and sitting there for quite long time.
* Bring patchset support to toaster patchwork.
* Add 'submitted upstream' and 'merged upstream' statuses to patchwork. Make the statuses updated automatically when patch is accepted to toaster-next or bitbake master.
* Clean up the code. Set realistic goals for pylint scores and follow them.
* Enhance Jenkins CI jobs to run all possible django, tts and pylint tests

==== Sujith's ====

* Get toaster working with non-git repos. Related bugs: 8456
* minor, but would be good to have :- Once user launches toaster, it would be nice to see the progress bar updated without user refreshing the page. Related bugs: 8328
* Cancellation of build. Related bugs: 6787

Would love to take up more tasks.

==== David's ====

* Finish the effort around Toaster "extensions", started by Farrell. This will enable Toaster customization for commercial partners, as per branding and
alternate pages (for extended project creation).

== All bugs for 2.2 ==

{{#bugzilla:
|columns=id,milestone,summary,status,priority
|milestone=future,2.1,2.1 M1,2.1 M2,2.1 M3,2.1 M4,2.2,2.2 M1,2.2 M2,2.2 M3,2.2 M4
|sort=priority
|product=Toaster
|resolution=!fixed
|status=!IN PROGRESS REVIEW
}}

== Toaster Enhancements for 2.2 ==
{{#bugzilla:
|columns=id,to,estimated,summary,priority,from,qa,milestone,status,resolution,whiteboard
|milestone=2.2,2.2 M1,2.2 M2,2.2 M3,2.2 M4
|sort=priority
|severity=enhancement
|product=Toaster
|status=!verified
}}

Toaster future release planning

2016-04-20T11:20:53Z

Michael Wood: /* Toaster Enhancements for 2.1 */

[[Category:Toaster]]

== 2.2 Planning ==

=== Wish lists ===

==== Michael's ====
* Remove pseudo API cruft / template context to json response stuff
* Look at schema and rework where needed
* Consolidate used API into it's own view
* Finish toaster tables porting
* Create a toasterclient.py to interact with toaster to avoid need for 'command line builds'
** This would use toaster's API as the entry point to using bitbake
** It could be called by CI systems
** It could be used by auto builders etc
** Means that there are no two modes anywhere in Toaster
** Allows project configuration to be changed in toaster from the command line

==== Brian's ====
The Ordered Part
# Django 1.8 aka LTS
# Image Customization
# Testing and Stability
# One way to start/One code path
<br>
This is *not* ordered! This is more of a brain dump :)
* Collapse analysis and managed mode
* Working CI for each commit into toaster-next as well as for project peoples testME branches on poky-contrib:
** run django tests
** run selenium tests
** leverage sstate to make runs generally shorter
* remove git assumptions from code
* rationalize/simplify the configuration scripts
* if bitbake-memres comes out, interact with it gracefully (this is my preferred method of collapsing analysis and managed)
* fix the tmpdir/release build issue &/or discuss the need to support building old releases with a UI rather than limiting toaster to build the release it is part of. This is inherently fragile.
* ask bb to parse any layer added to a project and such as imported layers, and layers specified in the configuration file to give us recipe information.
* improve our build event following:
** note parse events
** update/retrieve ui so that we can tell something is happening before the first build event comes
** eliminate the remaining random tracebacks from try/except failures.
* allow configuration in addition to the addition/removal of layers/packages
** kernel config
* add user control/authentication
* update to Django 1.8. Always stay on Django LTS versions
* add layer update from layers.openembedded.org to the UI
* add Django tests for building e.g. we should be able to build core-image-sato from a django test and validate the db
* add mysql script to simplify setup with mysql/apache
* pull in new look and feel
* add django tests to validate all artifacts - if we say they can download it and it appears on a page anywhere, it should have a test
* improve docs of code on wiki. We have a nice start (ty E & M); I'd like to improve it.
** events - more info and list types supported
** what logic is in jscript , what is in python
** database schema definition and description esp interrelationships.
** list django tests, mostly so we can see what we are missing.
* build mode triggerred off of database entry, not autonomous process.
* add asynchronicity. I have noted a number of places where commands result in a sluggish ui. I'm nervous about this one.
** for example, starting a build that requires us to clone layers has a noticeable delay as compared to starting a local only project build.
* verify we can stop builds cleanly.
* get the rest of our logs into our local toaster dir
* allow us to run and stop 2 toasters on the same machine without them interfering with each other
* (2.2 or later for sure) discuss how to support multiple back end bb servers.
** probably will need to be able to add servers/remove servers/see server status/build state
* be able to see more details about build. For example, see the "jobs" being processed like you can in knotty
* revisit what variables we let people change. sstate_dir and dl_dir should certainly be allowed to change
* discuss way we do migrations. possibly change. thoughts:
** only support migrations from last release to current. (e.g. if you were running toaster jethro you could update to jethro++, but could not from fido to jethro)
** take migration from the startup script. it's only for releasetorelease.
*** allow it to be in for master and ease of development w/in a release and pull at end as we do with moving releases.
** whatever we decide, add a test set for it so we know it works
* Be able to delete things from the ui
** old builds
** old projects
** imported layers we no longer want
* Be able to set from the ui a policy to delete all builds/recipes/packages that are older than XXX &/or be able to set maximum size on db and delete everything older than XXX when we near that size
* update the builds on the web page so we dont have to hit refresh to see the build progress.

==== Elliot's ====

* Make sure all files lint properly
* Include lint in CI builds
* Improve test coverage of all areas of the UI and back-end
* Clean up toasterui.py so it's easier to see what's going on in there
* Fix localhostbecontroller so it doesn't rely on what's in the log file when figuring out whether the bitbake server has started
* Provide a way to modify settings in settings.py without editing that file (e.g. with a local overlay configuration file)
* Move the logic, environment variables, directory setup and script calls from bin/toaster into Django commands: this would allow you to run toaster correctly, with all the required setup, without having to use bin/toaster; it would also mean that we could provide cleaner production setup instructions, as a user could run a series of Django commands (with their own parameters) to setup their environment; bin/toaster would then just execute a series of Django commands

==== Belén's ====

* Implement new custom theme (already in progress, aiming for end of M1: Dec 7th). Related bugs: 8417
* Finish image customisation. Related bugs: 8070, 8081, 8082, 8103, 8104, 8117, 8128, 8132, 8091
* Review interaction with layers: as it is, imposes heavy restrictions on users. Some of the questions I would like to consider:
** Should the import layer action be global instead of per project, so that you can reuse imported layers across projects?
** Should we allow users to override the release layer branch, setting a particular layer index layer to a different branch or commit?
** Should we allow users to set a different git repo for layer index layers?
** Should we allow users to override the bitbake revision set by the release in a project?
** Related bugs: 6640, 6701, 7574, 8426, 8429
* Access control (a simple one to start, with very basic permissions). Related bugs: 6233, 6234
* WIC integration (getting a step closer to the holy grail of 'click button' > 'image deployed in board')
* Fix the variable history (this is one of the features users have highly praised, and it is a bit broken in 2.0). Related bugs: 5811, 8488, 8190, 8189, 8188
* Get Toaster to collect build data from Jenkins builds. Once again, repeatedly requested by users. Related bugs: 7527

==== Ed's ====

* Merge analysis and managed modes
* Fix as much pending bugs as possible.
* Finish image customisation.
* Support Django LTS
* Increase test coverage. It would be good to set realistic goals, e.g. cover specific modules or increase coverage to 10% and follow the plan.
* Include production configuration into testing
* Cover views.py with tests and split it into modules. Currently it's huge and hard to maintain. I suspect there are a lot of unused APIs in there.
* Finish refactoring of buildcontroller code. This work is started in ed/toaster/bec and sitting there for quite long time.
* Bring patchset support to toaster patchwork.
* Add 'submitted upstream' and 'merged upstream' statuses to patchwork. Make the statuses updated automatically when patch is accepted to toaster-next or bitbake master.
* Clean up the code. Set realistic goals for pylint scores and follow them.
* Enhance Jenkins CI jobs to run all possible django, tts and pylint tests

==== Sujith's ====

* Get toaster working with non-git repos. Related bugs: 8456
* minor, but would be good to have :- Once user launches toaster, it would be nice to see the progress bar updated without user refreshing the page. Related bugs: 8328
* Cancellation of build. Related bugs: 6787

Would love to take up more tasks.

==== David's ====

* Finish the effort around Toaster "extensions", started by Farrell. This will enable Toaster customization for commercial partners, as per branding and
alternate pages (for extended project creation).

== All bugs for 2.1+ ==

{{#bugzilla:
|columns=id,milestone,summary,status,priority
|milestone=future,2.1,2.1 M1,2.1 M2,2.1 M3,2.1 M4,2.2
|sort=priority
|product=Toaster
|resolution=!fixed
|status=!IN PROGRESS REVIEW
}}

== Toaster Enhancements for 2.2 ==
{{#bugzilla:
|columns=id,to,estimated,summary,priority,from,qa,milestone,status,resolution,whiteboard
|milestone=2.2,2.2 M1,2.2 M2,2.2 M3,2.2 M4
|sort=priority
|severity=enhancement
|product=Toaster
|status=!verified
}}

Toaster future release planning

2016-04-20T11:11:55Z

Michael Wood: /* 2.1 Planning */

[[Category:Toaster]]

== 2.2 Planning ==

=== Wish lists ===

==== Michael's ====
* Remove pseudo API cruft / template context to json response stuff
* Look at schema and rework where needed
* Consolidate used API into it's own view
* Finish toaster tables porting
* Create a toasterclient.py to interact with toaster to avoid need for 'command line builds'
** This would use toaster's API as the entry point to using bitbake
** It could be called by CI systems
** It could be used by auto builders etc
** Means that there are no two modes anywhere in Toaster
** Allows project configuration to be changed in toaster from the command line

==== Brian's ====
The Ordered Part
# Django 1.8 aka LTS
# Image Customization
# Testing and Stability
# One way to start/One code path
<br>
This is *not* ordered! This is more of a brain dump :)
* Collapse analysis and managed mode
* Working CI for each commit into toaster-next as well as for project peoples testME branches on poky-contrib:
** run django tests
** run selenium tests
** leverage sstate to make runs generally shorter
* remove git assumptions from code
* rationalize/simplify the configuration scripts
* if bitbake-memres comes out, interact with it gracefully (this is my preferred method of collapsing analysis and managed)
* fix the tmpdir/release build issue &/or discuss the need to support building old releases with a UI rather than limiting toaster to build the release it is part of. This is inherently fragile.
* ask bb to parse any layer added to a project and such as imported layers, and layers specified in the configuration file to give us recipe information.
* improve our build event following:
** note parse events
** update/retrieve ui so that we can tell something is happening before the first build event comes
** eliminate the remaining random tracebacks from try/except failures.
* allow configuration in addition to the addition/removal of layers/packages
** kernel config
* add user control/authentication
* update to Django 1.8. Always stay on Django LTS versions
* add layer update from layers.openembedded.org to the UI
* add Django tests for building e.g. we should be able to build core-image-sato from a django test and validate the db
* add mysql script to simplify setup with mysql/apache
* pull in new look and feel
* add django tests to validate all artifacts - if we say they can download it and it appears on a page anywhere, it should have a test
* improve docs of code on wiki. We have a nice start (ty E & M); I'd like to improve it.
** events - more info and list types supported
** what logic is in jscript , what is in python
** database schema definition and description esp interrelationships.
** list django tests, mostly so we can see what we are missing.
* build mode triggerred off of database entry, not autonomous process.
* add asynchronicity. I have noted a number of places where commands result in a sluggish ui. I'm nervous about this one.
** for example, starting a build that requires us to clone layers has a noticeable delay as compared to starting a local only project build.
* verify we can stop builds cleanly.
* get the rest of our logs into our local toaster dir
* allow us to run and stop 2 toasters on the same machine without them interfering with each other
* (2.2 or later for sure) discuss how to support multiple back end bb servers.
** probably will need to be able to add servers/remove servers/see server status/build state
* be able to see more details about build. For example, see the "jobs" being processed like you can in knotty
* revisit what variables we let people change. sstate_dir and dl_dir should certainly be allowed to change
* discuss way we do migrations. possibly change. thoughts:
** only support migrations from last release to current. (e.g. if you were running toaster jethro you could update to jethro++, but could not from fido to jethro)
** take migration from the startup script. it's only for releasetorelease.
*** allow it to be in for master and ease of development w/in a release and pull at end as we do with moving releases.
** whatever we decide, add a test set for it so we know it works
* Be able to delete things from the ui
** old builds
** old projects
** imported layers we no longer want
* Be able to set from the ui a policy to delete all builds/recipes/packages that are older than XXX &/or be able to set maximum size on db and delete everything older than XXX when we near that size
* update the builds on the web page so we dont have to hit refresh to see the build progress.

==== Elliot's ====

* Make sure all files lint properly
* Include lint in CI builds
* Improve test coverage of all areas of the UI and back-end
* Clean up toasterui.py so it's easier to see what's going on in there
* Fix localhostbecontroller so it doesn't rely on what's in the log file when figuring out whether the bitbake server has started
* Provide a way to modify settings in settings.py without editing that file (e.g. with a local overlay configuration file)
* Move the logic, environment variables, directory setup and script calls from bin/toaster into Django commands: this would allow you to run toaster correctly, with all the required setup, without having to use bin/toaster; it would also mean that we could provide cleaner production setup instructions, as a user could run a series of Django commands (with their own parameters) to setup their environment; bin/toaster would then just execute a series of Django commands

==== Belén's ====

* Implement new custom theme (already in progress, aiming for end of M1: Dec 7th). Related bugs: 8417
* Finish image customisation. Related bugs: 8070, 8081, 8082, 8103, 8104, 8117, 8128, 8132, 8091
* Review interaction with layers: as it is, imposes heavy restrictions on users. Some of the questions I would like to consider:
** Should the import layer action be global instead of per project, so that you can reuse imported layers across projects?
** Should we allow users to override the release layer branch, setting a particular layer index layer to a different branch or commit?
** Should we allow users to set a different git repo for layer index layers?
** Should we allow users to override the bitbake revision set by the release in a project?
** Related bugs: 6640, 6701, 7574, 8426, 8429
* Access control (a simple one to start, with very basic permissions). Related bugs: 6233, 6234
* WIC integration (getting a step closer to the holy grail of 'click button' > 'image deployed in board')
* Fix the variable history (this is one of the features users have highly praised, and it is a bit broken in 2.0). Related bugs: 5811, 8488, 8190, 8189, 8188
* Get Toaster to collect build data from Jenkins builds. Once again, repeatedly requested by users. Related bugs: 7527

==== Ed's ====

* Merge analysis and managed modes
* Fix as much pending bugs as possible.
* Finish image customisation.
* Support Django LTS
* Increase test coverage. It would be good to set realistic goals, e.g. cover specific modules or increase coverage to 10% and follow the plan.
* Include production configuration into testing
* Cover views.py with tests and split it into modules. Currently it's huge and hard to maintain. I suspect there are a lot of unused APIs in there.
* Finish refactoring of buildcontroller code. This work is started in ed/toaster/bec and sitting there for quite long time.
* Bring patchset support to toaster patchwork.
* Add 'submitted upstream' and 'merged upstream' statuses to patchwork. Make the statuses updated automatically when patch is accepted to toaster-next or bitbake master.
* Clean up the code. Set realistic goals for pylint scores and follow them.
* Enhance Jenkins CI jobs to run all possible django, tts and pylint tests

==== Sujith's ====

* Get toaster working with non-git repos. Related bugs: 8456
* minor, but would be good to have :- Once user launches toaster, it would be nice to see the progress bar updated without user refreshing the page. Related bugs: 8328
* Cancellation of build. Related bugs: 6787

Would love to take up more tasks.

==== David's ====

* Finish the effort around Toaster "extensions", started by Farrell. This will enable Toaster customization for commercial partners, as per branding and
alternate pages (for extended project creation).

== All bugs for 2.1+ ==

{{#bugzilla:
|columns=id,milestone,summary,status,priority
|milestone=future,2.1,2.1 M1,2.1 M2,2.1 M3,2.1 M4,2.2
|sort=priority
|product=Toaster
|resolution=!fixed
|status=!IN PROGRESS REVIEW
}}

== Toaster Enhancements for 2.1 ==
{{#bugzilla:
|columns=id,to,estimated,summary,priority,from,qa,milestone,status,resolution,whiteboard
|milestone=2.1, 2.1 M1, 2.1 M2, 2.1 M3, 2.1 M4
|sort=priority
|severity=enhancement
|product=Toaster
|status=!verified
}}

Toaster future release planning

2016-04-20T11:10:26Z

Michael Wood: /* All bugs for 2.1+ */

[[Category:Toaster]]

== 2.1 Planning ==

=== Wish lists ===

==== Michael's ====
* Finish image-customisation 8070, 8081, 8082, 8103, 8104, 8117, 8128, 8132, 8091
* Remove pseudo API cruft / template context to json response stuff
* Consolidate used API into it's own view
* Finish toaster tables porting
* Get selenium test wrapper
* Create a toasterclient.py to interact with toaster to avoid need for 'command line builds'
** This would use toaster's API as the entry point to using bitbake
** It could be called by CI systems
** It could be used by auto builders etc
** Means that there are no two modes anywhere in Toaster
** Allows project configuration to be changed in toaster from the command line

==== Brian's ====
The Ordered Part
# Django 1.8 aka LTS
# Image Customization
# Testing and Stability
# One way to start/One code path
<br>
This is *not* ordered! This is more of a brain dump :)
* Collapse analysis and managed mode
* Working CI for each commit into toaster-next as well as for project peoples testME branches on poky-contrib:
** run django tests
** run selenium tests
** leverage sstate to make runs generally shorter
* remove git assumptions from code
* rationalize/simplify the configuration scripts
* if bitbake-memres comes out, interact with it gracefully (this is my preferred method of collapsing analysis and managed)
* fix the tmpdir/release build issue &/or discuss the need to support building old releases with a UI rather than limiting toaster to build the release it is part of. This is inherently fragile.
* ask bb to parse any layer added to a project and such as imported layers, and layers specified in the configuration file to give us recipe information.
* improve our build event following:
** note parse events
** update/retrieve ui so that we can tell something is happening before the first build event comes
** eliminate the remaining random tracebacks from try/except failures.
* allow configuration in addition to the addition/removal of layers/packages
** kernel config
* add user control/authentication
* update to Django 1.8. Always stay on Django LTS versions
* add layer update from layers.openembedded.org to the UI
* add Django tests for building e.g. we should be able to build core-image-sato from a django test and validate the db
* add mysql script to simplify setup with mysql/apache
* pull in new look and feel
* add django tests to validate all artifacts - if we say they can download it and it appears on a page anywhere, it should have a test
* improve docs of code on wiki. We have a nice start (ty E & M); I'd like to improve it.
** events - more info and list types supported
** what logic is in jscript , what is in python
** database schema definition and description esp interrelationships.
** list django tests, mostly so we can see what we are missing.
* build mode triggerred off of database entry, not autonomous process.
* add asynchronicity. I have noted a number of places where commands result in a sluggish ui. I'm nervous about this one.
** for example, starting a build that requires us to clone layers has a noticeable delay as compared to starting a local only project build.
* verify we can stop builds cleanly.
* get the rest of our logs into our local toaster dir
* allow us to run and stop 2 toasters on the same machine without them interfering with each other
* (2.2 or later for sure) discuss how to support multiple back end bb servers.
** probably will need to be able to add servers/remove servers/see server status/build state
* be able to see more details about build. For example, see the "jobs" being processed like you can in knotty
* revisit what variables we let people change. sstate_dir and dl_dir should certainly be allowed to change
* discuss way we do migrations. possibly change. thoughts:
** only support migrations from last release to current. (e.g. if you were running toaster jethro you could update to jethro++, but could not from fido to jethro)
** take migration from the startup script. it's only for releasetorelease.
*** allow it to be in for master and ease of development w/in a release and pull at end as we do with moving releases.
** whatever we decide, add a test set for it so we know it works
* Be able to delete things from the ui
** old builds
** old projects
** imported layers we no longer want
* Be able to set from the ui a policy to delete all builds/recipes/packages that are older than XXX &/or be able to set maximum size on db and delete everything older than XXX when we near that size
* update the builds on the web page so we dont have to hit refresh to see the build progress.

==== Elliot's ====

* Make sure all files lint properly
* Include lint in CI builds
* Improve test coverage of all areas of the UI and back-end
* Clean up toasterui.py so it's easier to see what's going on in there
* Fix localhostbecontroller so it doesn't rely on what's in the log file when figuring out whether the bitbake server has started
* Provide a way to modify settings in settings.py without editing that file (e.g. with a local overlay configuration file)
* Move the logic, environment variables, directory setup and script calls from bin/toaster into Django commands: this would allow you to run toaster correctly, with all the required setup, without having to use bin/toaster; it would also mean that we could provide cleaner production setup instructions, as a user could run a series of Django commands (with their own parameters) to setup their environment; bin/toaster would then just execute a series of Django commands

==== Belén's ====

* Implement new custom theme (already in progress, aiming for end of M1: Dec 7th). Related bugs: 8417
* Finish image customisation. Related bugs: 8070, 8081, 8082, 8103, 8104, 8117, 8128, 8132, 8091
* Review interaction with layers: as it is, imposes heavy restrictions on users. Some of the questions I would like to consider:
** Should the import layer action be global instead of per project, so that you can reuse imported layers across projects?
** Should we allow users to override the release layer branch, setting a particular layer index layer to a different branch or commit?
** Should we allow users to set a different git repo for layer index layers?
** Should we allow users to override the bitbake revision set by the release in a project?
** Related bugs: 6640, 6701, 7574, 8426, 8429
* Access control (a simple one to start, with very basic permissions). Related bugs: 6233, 6234
* WIC integration (getting a step closer to the holy grail of 'click button' > 'image deployed in board')
* Fix the variable history (this is one of the features users have highly praised, and it is a bit broken in 2.0). Related bugs: 5811, 8488, 8190, 8189, 8188
* Get Toaster to collect build data from Jenkins builds. Once again, repeatedly requested by users. Related bugs: 7527

==== Ed's ====

* Merge analysis and managed modes
* Fix as much pending bugs as possible.
* Finish image customisation.
* Support Django LTS
* Increase test coverage. It would be good to set realistic goals, e.g. cover specific modules or increase coverage to 10% and follow the plan.
* Include production configuration into testing
* Cover views.py with tests and split it into modules. Currently it's huge and hard to maintain. I suspect there are a lot of unused APIs in there.
* Finish refactoring of buildcontroller code. This work is started in ed/toaster/bec and sitting there for quite long time.
* Bring patchset support to toaster patchwork.
* Add 'submitted upstream' and 'merged upstream' statuses to patchwork. Make the statuses updated automatically when patch is accepted to toaster-next or bitbake master.
* Clean up the code. Set realistic goals for pylint scores and follow them.
* Enhance Jenkins CI jobs to run all possible django, tts and pylint tests

==== Sujith's ====

* Get toaster working with non-git repos. Related bugs: 8456
* minor, but would be good to have :- Once user launches toaster, it would be nice to see the progress bar updated without user refreshing the page. Related bugs: 8328
* Cancellation of build. Related bugs: 6787

Would love to take up more tasks.

==== David's ====

* Finish the effort around Toaster "extensions", started by Farrell. This will enable Toaster customization for commercial partners, as per branding and
alternate pages (for extended project creation).

== All bugs for 2.1+ ==

{{#bugzilla:
|columns=id,milestone,summary,status,priority
|milestone=future,2.1,2.1 M1,2.1 M2,2.1 M3,2.1 M4,2.2
|sort=priority
|product=Toaster
|resolution=!fixed
|status=!IN PROGRESS REVIEW
}}

== Toaster Enhancements for 2.1 ==
{{#bugzilla:
|columns=id,to,estimated,summary,priority,from,qa,milestone,status,resolution,whiteboard
|milestone=2.1, 2.1 M1, 2.1 M2, 2.1 M3, 2.1 M4
|sort=priority
|severity=enhancement
|product=Toaster
|status=!verified
}}

Setting up a production instance of Toaster

2016-04-18T11:08:10Z

Michael Wood:

[[Category:Toaster]]
----
'''This page is the development version of the documentation to provide the latest information, if you're using a release please refer to [https://www.yoctoproject.org/documentation/archived the published manual]'''
----
A production instance of Toaster is one in which you wish to share the Toaster instance with remote and multiple users. It is also the setup which can cope with heavier loads on the web service. These instructions setup toaster in Build mode where builds and projects are run, viewed and defined by the Toaster web interface.

== Requirements ==
* [http://www.yoctoproject.org/docs/2.0/yocto-project-qs/yocto-project-qs.html#packages Build requirements]
* Apache webserver
* mod-wsgi for Apache webserver
* Mysql database server

Ubuntu 14.04.3:
<code>
$ sudo apt-get install apache2 libapache2-mod-wsgi mysql-server python-virtualenv libmysqlclient-dev python-dev python-mysqldb
</code>

Fedora 22/RH:
<code>
$ sudo dnf install httpd mod_wsgi python-virtualenv gcc mysql-devel
</code>

== Installation ==

'''1.''' Checkout a copy of Poky into the web server directory. We're going to be using /var/www/toaster.
<code>
$ mkdir -p /var/www/toaster
$ cd /var/www/toaster/
$ git clone git://git.yoctoproject.org/poky
$ cd poky
$ git checkout jethro # change for any release name required
</code>

'''2.''' Initialise a virtualenv and install Toaster dependencies. (Use virtualenv to keep the python packages isolated from your system provided packages - not required but recommended, alternative use your OS's package manager to install the packages)

<code>
$ cd /var/www/toaster/
$ virtualenv venv
$ source ./venv/bin/activate
$ pip install -r ./poky/bitbake/toaster-requirements.txt
$ pip install mysqlclient
</code>

'''3.''' Configure toaster edit /var/www/toaster/poky/bitbake/lib/toaster/toastermain/settings.py

Edit the DATABASE settings:
<code>
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.mysql',
'NAME': 'toaster_data',
'USER': 'toaster',
'PASSWORD': 'yourpasswordhere',
'HOST': 'localhost',
'PORT': '3306',
}
</code>

Edit the [https://docs.djangoproject.com/en/1.8/howto/deployment/checklist/#secret-key SECRET_KEY]:
<code>
SECRET_KEY = 'YOUR SECRET RANDOM KEY HERE'
</code>

Edit the STATIC_ROOT:
<code>
STATIC_ROOT = '/var/www/toaster/static_files/'
</code>

Edit BUILD_MODE:
<code>
BUILD_MODE = True
</code>

'''4.'''' Now add the database and user to your mysql server that we just defined
<code>
$ mysql -u root -p
mysql> CREATE DATABASE toaster_data;
mysql> CREATE USER 'toaster'@'localhost' identified by 'yourpasswordhere';
mysql> GRANT all on toaster_data.* to 'toaster'@'localhost';
mysql> quit
</code>
n.b. You may want to decide on fewer [https://dev.mysql.com/doc/refman/5.1/en/grant.html privileges] to the toaster user.

'''5.''' Get toaster to create the database schema, default data, update the TOASTER_DIR which is the build work dir and collect up the statically served files

<code>
$ cd /var/www/toaster/poky/
$ ./bitbake/lib/toaster/manage.py syncdb
$ ./bitbake/lib/toaster/manage.py migrate
$ ./bitbake/lib/toaster/manage.py loadconf ./meta-yocto/conf/toasterconf.json
$ TOASTER_DIR=/var/www/toaster/poky/ ./bitbake/lib/toaster/manage.py checksettings
$ ./bitbake/lib/toaster/manage.py lsupdates
$ ./bitbake/lib/toaster/manage.py collectstatic
</code>

'''6.''' Add a config file for Toaster to your Apache web server's configurations available directory.

Ubuntu/Debian put it here: /etc/apache2/conf-available/toaster.conf
Fedora/RH usually here: /etc/httpd/conf.d/toaster.conf
<code>

Alias /static /var/www/toaster/static_files
<Directory /var/www/toaster/static_files>
Order allow,deny
Allow from all
Require all granted
</Directory>
WSGIDaemonProcess toaster_wsgi python-path=/var/www/toaster/poky/bitbake/lib/toaster:/var/www/toaster/venv/lib/python2.7/site-packages
WSGIScriptAlias / "/var/www/toaster/poky/bitbake/lib/toaster/toastermain/wsgi.py"
<Location />
WSGIProcessGroup toaster_wsgi
</Location>

</code>

In Ubuntu/Debain you will need to enable the config and module in Apache webserver
<code>
$ sudo a2enmod wsgi
$ sudo a2enconf toaster
</code>

Restart Apache web server to make sure all new configuration is loaded

Ubuntu/Debian:
<code>
$ sudo service apache2 restart
</code>

Fedora/RH:
<code>
$ sudo service httpd restart
</code>

'''7.''' Install the build runner service

This service needs to be running in order to dispatch builds the command that needs to be run is:

<code>
/var/www/toaster/poky/bitbake/lib/toaster/manage.py runbuilds
</code>

Sample script:
<code>
#!/bin/sh
# toaster run builds dispatcher
cd /var/www/toaster/
source ./venv/bin/activate
while true; do ./poky/bitbake/lib/toaster/manage.py runbuilds; sleep 3; done
</code>

N.b. You may wish to add a service entry to your OS's init system so that it starts up on start up as well as adding a dedicated user.

Now open up a browser and you can start using Toaster!

Setting up a production instance of Toaster

2016-04-18T11:07:48Z

Michael Wood:

[[Category:Toaster]]

----
'''This page is the development version of the documentation to provide the latest information, if you're using a release please refer to [https://www.yoctoproject.org/documentation/archived the published manual]'''
----

A production instance of Toaster is one in which you wish to share the Toaster instance with remote and multiple users. It is also the setup which can cope with heavier loads on the web service. These instructions setup toaster in Build mode where builds and projects are run, viewed and defined by the Toaster web interface.

== Requirements ==
* [http://www.yoctoproject.org/docs/2.0/yocto-project-qs/yocto-project-qs.html#packages Build requirements]
* Apache webserver
* mod-wsgi for Apache webserver
* Mysql database server

Ubuntu 14.04.3:
<code>
$ sudo apt-get install apache2 libapache2-mod-wsgi mysql-server python-virtualenv libmysqlclient-dev python-dev python-mysqldb
</code>

Fedora 22/RH:
<code>
$ sudo dnf install httpd mod_wsgi python-virtualenv gcc mysql-devel
</code>

== Installation ==

'''1.''' Checkout a copy of Poky into the web server directory. We're going to be using /var/www/toaster.
<code>
$ mkdir -p /var/www/toaster
$ cd /var/www/toaster/
$ git clone git://git.yoctoproject.org/poky
$ cd poky
$ git checkout jethro # change for any release name required
</code>

'''2.''' Initialise a virtualenv and install Toaster dependencies. (Use virtualenv to keep the python packages isolated from your system provided packages - not required but recommended, alternative use your OS's package manager to install the packages)

<code>
$ cd /var/www/toaster/
$ virtualenv venv
$ source ./venv/bin/activate
$ pip install -r ./poky/bitbake/toaster-requirements.txt
$ pip install mysqlclient
</code>

'''3.''' Configure toaster edit /var/www/toaster/poky/bitbake/lib/toaster/toastermain/settings.py

Edit the DATABASE settings:
<code>
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.mysql',
'NAME': 'toaster_data',
'USER': 'toaster',
'PASSWORD': 'yourpasswordhere',
'HOST': 'localhost',
'PORT': '3306',
}
</code>

Edit the [https://docs.djangoproject.com/en/1.8/howto/deployment/checklist/#secret-key SECRET_KEY]:
<code>
SECRET_KEY = 'YOUR SECRET RANDOM KEY HERE'
</code>

Edit the STATIC_ROOT:
<code>
STATIC_ROOT = '/var/www/toaster/static_files/'
</code>

Edit BUILD_MODE:
<code>
BUILD_MODE = True
</code>

'''4.'''' Now add the database and user to your mysql server that we just defined
<code>
$ mysql -u root -p
mysql> CREATE DATABASE toaster_data;
mysql> CREATE USER 'toaster'@'localhost' identified by 'yourpasswordhere';
mysql> GRANT all on toaster_data.* to 'toaster'@'localhost';
mysql> quit
</code>
n.b. You may want to decide on fewer [https://dev.mysql.com/doc/refman/5.1/en/grant.html privileges] to the toaster user.

'''5.''' Get toaster to create the database schema, default data, update the TOASTER_DIR which is the build work dir and collect up the statically served files

<code>
$ cd /var/www/toaster/poky/
$ ./bitbake/lib/toaster/manage.py syncdb
$ ./bitbake/lib/toaster/manage.py migrate
$ ./bitbake/lib/toaster/manage.py loadconf ./meta-yocto/conf/toasterconf.json
$ TOASTER_DIR=/var/www/toaster/poky/ ./bitbake/lib/toaster/manage.py checksettings
$ ./bitbake/lib/toaster/manage.py lsupdates
$ ./bitbake/lib/toaster/manage.py collectstatic
</code>

'''6.''' Add a config file for Toaster to your Apache web server's configurations available directory.

Ubuntu/Debian put it here: /etc/apache2/conf-available/toaster.conf
Fedora/RH usually here: /etc/httpd/conf.d/toaster.conf
<code>

Alias /static /var/www/toaster/static_files
<Directory /var/www/toaster/static_files>
Order allow,deny
Allow from all
Require all granted
</Directory>
WSGIDaemonProcess toaster_wsgi python-path=/var/www/toaster/poky/bitbake/lib/toaster:/var/www/toaster/venv/lib/python2.7/site-packages
WSGIScriptAlias / "/var/www/toaster/poky/bitbake/lib/toaster/toastermain/wsgi.py"
<Location />
WSGIProcessGroup toaster_wsgi
</Location>

</code>

In Ubuntu/Debain you will need to enable the config and module in Apache webserver
<code>
$ sudo a2enmod wsgi
$ sudo a2enconf toaster
</code>

Restart Apache web server to make sure all new configuration is loaded

Ubuntu/Debian:
<code>
$ sudo service apache2 restart
</code>

Fedora/RH:
<code>
$ sudo service httpd restart
</code>

'''7.''' Install the build runner service

This service needs to be running in order to dispatch builds the command that needs to be run is:

<code>
/var/www/toaster/poky/bitbake/lib/toaster/manage.py runbuilds
</code>

Sample script:
<code>
#!/bin/sh
# toaster run builds dispatcher
cd /var/www/toaster/
source ./venv/bin/activate
while true; do ./poky/bitbake/lib/toaster/manage.py runbuilds; sleep 3; done
</code>

N.b. You may wish to add a service entry to your OS's init system so that it starts up on start up as well as adding a dedicated user.

Now open up a browser and you can start using Toaster!

Toaster

2016-03-31T14:13:58Z

Michael Wood:

[[Category:Toaster]]
[[Toaster]] is a web-based interface to OpenEmbedded and BitBake.
[[File:Screenshot toaster.png|thumb|Screenshot of Toaster 2.1]]
General discussion about '''Toaster''' happens on a dedicated mailing list: [https://lists.yoctoproject.org/listinfo/toaster https://lists.yoctoproject.org/listinfo/toaster]

=== Using Toaster ===

Toaster can run in various modes and setups.

* '''Local mode''' - in this mode Toaster is setup for use as a local development tool. It can be used to configure builds or just as a receiver for builds done on the command line with bitbake. You can launch it like this:

$ source oe-init-build-env
$ source toaster start

You then navigate to the link in your browser (e.g. http://localhost:8000) and configure a project. Or start building in the normal way with bitbake via the command line. Toaster will automatically pick up the builds and you will be able to see them on the build dashboard in your browser.

* '''Production mode''' - All the same functionality as the local mode but with the web server setup as a shared service for multiple developers to use, this sets up Toaster as a wsgi application and [[Setting up a production instance of Toaster|requires additional configuration]].

=== Toaster How-to's ===

Specific pages with Toaster how-tos are available below.

* [[Contribute to Toaster]]
* [[Testing Toaster]]
* [[Setting up a local instance of Toaster]]
* [[Setting up a production instance of Toaster]] - documentation for Interactive mode
* [https://www.yoctoproject.org/documentation/toaster-manual-18 How to use the Toaster web interface]
* [[How to delete information from the Toaster database]]
* [[How to support permission management in Build Mode for Toaster]]

=== About Toaster ===
[[File:Screenshot toaster analyis.png|thumb|Analysis of builds using Toaster]]

* [[File:Working_with_design.pdf]]
* [https://bugzilla.yoctoproject.org/buglist.cgi?list_id=213820&columnlist=status_whiteboard%2Cassigned_to%2Ctarget_milestone%2Cbug_status%2Cshort_desc%2Cbug_severity%2Cpriority&classification=Build%20System%20%26%20Metadata&query_based_on=Toaster-Opens&query_format=advanced&bug_status=NEW&bug_status=ACCEPTED&bug_status=IN%20PROGRESS%20DESIGN&bug_status=IN%20PROGRESS%20DESIGN%20COMPLETE&bug_status=IN%20PROGRESS%20IMPLEMENTATION&bug_status=REOPENED&bug_status=NEEDINFO&bug_status=WaitForUpstream&component=toaster&product=Toaster&known_name=Toaster-Opens Bug list]
* [[Toaster architecture design]]
* [[Toaster and bitbake communications]]
* [[Toaster testing plan]]

=== In progress documentation ===

We are currently preparing the documentation for the Toaster build functionality. The content here is just a brain dump of what we need to cover (in no particular order). Feel free to add and create content as you see fit:

*[[Using virtualenv]]
*[[Setting up a production instance of Toaster]]
*[[manage.py commands]] - this should include an explanation of lsupdates
*[[Start Toaster in managed mode]]

Toaster

2016-03-31T12:33:02Z

Michael Wood:

[[Category:Toaster]]
[[Toaster]] is a web-based interface to OpenEmbedded and BitBake.
[[File:Screenshot toaster.png|thumb|Screenshot of Toaster 2.1]]
General discussion about '''Toaster''' happens on a dedicated mailing list: [https://lists.yoctoproject.org/listinfo/toaster https://lists.yoctoproject.org/listinfo/toaster]

=== Using Toaster ===

Toaster can run in various modes and setups.

* '''Local mode''' - in this mode Toaster is setup for use as a local development tool. It can be used to configure builds or just as a receiver for builds done on the command line with bitbake. You can launch it like this:

$ source oe-init-build-env
$ source toaster start

You then navigate to the link in your browser (e.g. [[http://localhost:8000]]) and configure a project. Or start building in the normal way with bitbake via the command line. Toaster will automatically pick up the builds and you will be able to see them on the build dashboard in your browser.

* '''Production mode''' - All the same functionality as the local mode but with the web server setup as a shared service for multiple developers to use, this sets up Toaster as a wsgi application and [[Setting up a production instance of Toaster|requires additional configuration]].

=== Toaster How-to's ===

Specific pages with Toaster how-tos are available below.

* [[Contribute to Toaster]]
* [[Testing Toaster]]
* [[Setting up a local instance of Toaster]]
* [[Setting up a production instance of Toaster]] - documentation for Interactive mode
* [https://www.yoctoproject.org/documentation/toaster-manual-18 How to use the Toaster web interface]
* [[How to delete information from the Toaster database]]
* [[How to support permission management in Build Mode for Toaster]]

=== About Toaster ===
[[File:Screenshot toaster analyis.png|thumb|Analysis of builds using Toaster]]

* [[File:Working_with_design.pdf]]
* [https://bugzilla.yoctoproject.org/buglist.cgi?list_id=213820&columnlist=status_whiteboard%2Cassigned_to%2Ctarget_milestone%2Cbug_status%2Cshort_desc%2Cbug_severity%2Cpriority&classification=Build%20System%20%26%20Metadata&query_based_on=Toaster-Opens&query_format=advanced&bug_status=NEW&bug_status=ACCEPTED&bug_status=IN%20PROGRESS%20DESIGN&bug_status=IN%20PROGRESS%20DESIGN%20COMPLETE&bug_status=IN%20PROGRESS%20IMPLEMENTATION&bug_status=REOPENED&bug_status=NEEDINFO&bug_status=WaitForUpstream&component=toaster&product=Toaster&known_name=Toaster-Opens Bug list]
* [[Toaster architecture design]]
* [[Toaster and bitbake communications]]
* [[Toaster testing plan]]

=== In progress documentation ===

We are currently preparing the documentation for the Toaster build functionality. The content here is just a brain dump of what we need to cover (in no particular order). Feel free to add and create content as you see fit:

*[[Using virtualenv]]
*[[Setting up a production instance of Toaster]]
*[[manage.py commands]] - this should include an explanation of lsupdates
*[[Start Toaster in managed mode]]

Toaster

2016-03-31T11:36:18Z

Michael Wood:

[[Category:Toaster]]
[[Toaster]] is a web-based interface to OpenEmbedded and BitBake.
[[File:Screenshot toaster.png|thumb|Screenshot of Toaster 2.1]]
General discussion about '''Toaster''' happens on a dedicated mailing list: [https://lists.yoctoproject.org/listinfo/toaster https://lists.yoctoproject.org/listinfo/toaster]

=== Using Toaster ===
[[File:Screenshot toaster analyis.png|thumb|Analysis of builds using Toaster]]
Toaster can run in various modes and setups.

* '''Local mode''' - in this mode Toaster is setup for use as a local development tool. It can be used to configure builds or just as a receiver for builds done on the command line with bitbake. You can launch it like this:

$ source oe-init-build-env
$ source toaster start

You then navigate to the link in your browser (e.g. [[http://localhost:8000]]) and configure a project. Or start building in the normal way with bitbake via the command line. Toaster will automatically pick up the builds and you will be able to see them on the build dashboard in your browser.

* '''Production mode''' - All the same functionality as the local mode but with the web server setup as a shared service for multiple developers to use, this sets up Toaster as a wsgi application and [[Setting up a production instance of Toaster|requires additional configuration]].

=== Toaster How-to's ===

Specific pages with Toaster how-tos are available below.

* [[Contribute to Toaster]]
* [[Testing Toaster]]
* [[Setting up a local instance of Toaster]]
* [[Setting up a production instance of Toaster]] - documentation for Interactive mode
* [https://www.yoctoproject.org/documentation/toaster-manual-18 How to use the Toaster web interface]
* [[How to delete information from the Toaster database]]
* [[How to support permission management in Build Mode for Toaster]]

=== About Toaster ===

* [[File:Working_with_design.pdf]]
* [https://bugzilla.yoctoproject.org/buglist.cgi?list_id=213820&columnlist=status_whiteboard%2Cassigned_to%2Ctarget_milestone%2Cbug_status%2Cshort_desc%2Cbug_severity%2Cpriority&classification=Build%20System%20%26%20Metadata&query_based_on=Toaster-Opens&query_format=advanced&bug_status=NEW&bug_status=ACCEPTED&bug_status=IN%20PROGRESS%20DESIGN&bug_status=IN%20PROGRESS%20DESIGN%20COMPLETE&bug_status=IN%20PROGRESS%20IMPLEMENTATION&bug_status=REOPENED&bug_status=NEEDINFO&bug_status=WaitForUpstream&component=toaster&product=Toaster&known_name=Toaster-Opens Bug list]
* [[Toaster architecture design]]
* [[Toaster and bitbake communications]]
* [[Toaster testing plan]]

=== In progress documentation ===

We are currently preparing the documentation for the Toaster build functionality. The content here is just a brain dump of what we need to cover (in no particular order). Feel free to add and create content as you see fit:

*[[Using virtualenv]]
*[[Setting up a production instance of Toaster]]
*[[manage.py commands]] - this should include an explanation of lsupdates
*[[Start Toaster in managed mode]]

File:Screenshot toaster analyis.png

2016-03-31T11:33:55Z

Michael Wood: Build analysis using Toaster

Build analysis using Toaster

Toaster

2016-03-31T11:32:27Z

Michael Wood:

[[Category:Toaster]]
[[Toaster]] is a web-based interface to OpenEmbedded and BitBake.
[[File:Screenshot toaster.png|thumb|Screenshot of Toaster 2.1]]
General discussion about '''Toaster''' happens on a dedicated mailing list: [https://lists.yoctoproject.org/listinfo/toaster https://lists.yoctoproject.org/listinfo/toaster]

=== Using Toaster ===

Toaster can run in various modes and setups.

* '''Local mode''' - in this mode Toaster is setup for use as a local development tool. It can be used to configure builds or just as a receiver for builds done on the command line with bitbake. You can launch it like this:

$ source oe-init-build-env
$ source toaster start

You then navigate to the link in your browser (e.g. [[http://localhost:8000]]) and configure a project. Or start building in the normal way with bitbake via the command line. Toaster will automatically pick up the builds and you will be able to see them on the build dashboard in your browser.

* '''Production mode''' - All the same functionality as the local mode but with the web server setup as a shared service for multiple developers to use, this sets up Toaster as a wsgi application and [[Setting up a production instance of Toaster|requires additional configuration]].

=== Toaster How-to's ===

Specific pages with Toaster how-tos are available below.

* [[Contribute to Toaster]]
* [[Testing Toaster]]
* [[Setting up a local instance of Toaster]]
* [[Setting up a production instance of Toaster]] - documentation for Interactive mode
* [https://www.yoctoproject.org/documentation/toaster-manual-18 How to use the Toaster web interface]
* [[How to delete information from the Toaster database]]
* [[How to support permission management in Build Mode for Toaster]]

=== About Toaster ===

* [[File:Working_with_design.pdf]]
* [https://bugzilla.yoctoproject.org/buglist.cgi?list_id=213820&columnlist=status_whiteboard%2Cassigned_to%2Ctarget_milestone%2Cbug_status%2Cshort_desc%2Cbug_severity%2Cpriority&classification=Build%20System%20%26%20Metadata&query_based_on=Toaster-Opens&query_format=advanced&bug_status=NEW&bug_status=ACCEPTED&bug_status=IN%20PROGRESS%20DESIGN&bug_status=IN%20PROGRESS%20DESIGN%20COMPLETE&bug_status=IN%20PROGRESS%20IMPLEMENTATION&bug_status=REOPENED&bug_status=NEEDINFO&bug_status=WaitForUpstream&component=toaster&product=Toaster&known_name=Toaster-Opens Bug list]
* [[Toaster architecture design]]
* [[Toaster and bitbake communications]]
* [[Toaster testing plan]]

=== In progress documentation ===

We are currently preparing the documentation for the Toaster build functionality. The content here is just a brain dump of what we need to cover (in no particular order). Feel free to add and create content as you see fit:

*[[Using virtualenv]]
*[[Setting up a production instance of Toaster]]
*[[manage.py commands]] - this should include an explanation of lsupdates
*[[Start Toaster in managed mode]]

Toaster

2016-03-31T11:02:57Z

Michael Wood:

[[Category:Toaster]]
[[Toaster]] is a web-based interface to OpenEmbedded and BitBake.
[[File:Screenshot toaster.png|thumb|Screenshot of Toaster 2.1]]
General discussion about '''Toaster''' happens on a dedicated mailing list: [https://lists.yoctoproject.org/listinfo/toaster https://lists.yoctoproject.org/listinfo/toaster]

=== Using Toaster ===

Toaster can run in various modes and setups. To ease out the understanding of documentation, we review here the terminology used throughout the documentation.

* Interactive mode - this is the mode released with Yocto Project Release 1.6. Functional components consist of a build recording component, 'toasterui', and a build stats and inspection user interface, the 'toastergui'. It is started with the command sequence listed below, and the builds are started using normal bitbake command line

$ source oe-init-build-env
$ source toaster start

* Managed mode - in this mode Toaster handles the build configuration GUI (through Project pages), and build scheduling and execution, in addition to the features launched with Yocto Project Release 1.6. The builds are triggered through the web interface,
the user as no direct access to the bitbake command line.

** Local managed mode, in short _local_, is the out-of-box mode available after a poky/ checkout. It allows the user to perform build on his local machine source code, with a local build directory, and is intended to help the user discover the interface, and configure and run local builds. You can launch it with

$ ./bitbake/bin/toaster

** Remote managed mode, or [[hosted Toaster]], is intended to be the production setup for running Toaster in organizations supporting multiple users and using customized Toaster installations.

=== Toaster How-to's ===

Specific pages with Toaster how-tos are available below.

* [[Contribute to Toaster]]
* [[Testing Toaster]]
* [[Setting up a local instance of Toaster]]
* [[Setting up a production instance of Toaster]] - documentation for Interactive mode
* [https://www.yoctoproject.org/documentation/toaster-manual-18 How to use the Toaster web interface]
* [[How to delete information from the Toaster database]]
* [[How to support permission management in Build Mode for Toaster]]

=== About Toaster ===

* [[File:Working_with_design.pdf]]
* [https://bugzilla.yoctoproject.org/buglist.cgi?list_id=213820&columnlist=status_whiteboard%2Cassigned_to%2Ctarget_milestone%2Cbug_status%2Cshort_desc%2Cbug_severity%2Cpriority&classification=Build%20System%20%26%20Metadata&query_based_on=Toaster-Opens&query_format=advanced&bug_status=NEW&bug_status=ACCEPTED&bug_status=IN%20PROGRESS%20DESIGN&bug_status=IN%20PROGRESS%20DESIGN%20COMPLETE&bug_status=IN%20PROGRESS%20IMPLEMENTATION&bug_status=REOPENED&bug_status=NEEDINFO&bug_status=WaitForUpstream&component=toaster&product=Toaster&known_name=Toaster-Opens Bug list]
* [[Toaster architecture design]]
* [[Toaster and bitbake communications]]
* [[Toaster testing plan]]
* [[Toaster 1.9 design | Scope and design (1.9)]]
* [[Toaster 1.7 design | Scope and design (1.7 - 1.8)]]
* [[Toaster archive | Archive]]

=== In progress documentation ===

We are currently preparing the documentation for the Toaster build functionality. The content here is just a brain dump of what we need to cover (in no particular order). Feel free to add and create content as you see fit:

*[[Using virtualenv]]
*[[Toaster configuration]] - explain releases, layer sources, BitBake versions and default project configurations
*[[Set up Toaster locally]] - explain the set up process and the local version of the localconf.json file
*[[Set up production instance]] - explain the Django admin interface and the remove version of the localconf.json file
*[[manage.py commands]] - this should include an explanation of lsupdates
*[[Custom layer index]] - explain how to create your own layer index
*[[Start Toaster in managed mode]]

File:Screenshot toaster.png

2016-03-31T10:58:50Z

Michael Wood:

Toaster architecture design

2016-03-30T17:34:53Z

Michael Wood: /* bldcontrol */

[[Category:Toaster]]

= Introduction =

Toaster project is located in [http://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/bitbake/lib/toaster/ bitbake/lib/toaster].

Toaster is built on the Django framework, if you're not familiar with Django framework, it's very [https://docs.djangoproject.com/en/1.6/intro/tutorial01/ well documented]

The basic layout of the framework is that for each Django application A URL defined in urls.py, maps to a view defined in views.py, the urls and views have a 1:1 mapping to a function or View class in the views.py. These functions/Views runs the server side logic and return the data either to render a [https://docs.djangoproject.com/en/1.8/topics/templates/#the-django-template-language Django template] which can be found in the templates directory or alternatively they can return whatever http response is needed (e.g. a JSON document).

Toaster stores a lot of information about the configuration of projects, layers and recipes, this is stored in [[Toaster database]] which is fully abstracted via [https://docs.djangoproject.com/en/1.8/ref/models/querysets/ Django's object relation mapping (orm)]. This means that we don't have to get our hands dirty with making database connections, caring which kind of database is chosen or mapping data in and out of the database. Toaster's database schema is defined in the ''models.py'' of each of the Django applications that makes up Toaster.

<nowiki>

├── bldcollector
│   ├── admin.py
│   ├── __init__.py
│   ├── urls.py
│   └── views.py
├── bldcontrol
│   ├── admin.py
│   ├── bbcontroller.py
│   ├── __init__.py
│   ├── localhostbecontroller.py
│   ├── management
│   ├── migrations
│   ├── models.py
│   ├── tests.py
│   └── views.py
├── __init__.py
├── manage.py
├── orm
│   ├── __init__.py
│   ├── management
│   ├── migrations
│   ├── models.py
│   └── tests.py
├── toastergui
│   ├── fixtures
│   ├── __init__.py
│   ├── static
│   ├── tablefilter.py
│   ├── tables.py
│   ├── templates
│   ├── templatetags
│   ├── tests.py
│   ├── typeaheads.py
│   ├── urls.py
│   ├── views.py
│   └── widgets.py
└── toastermain
├── __init__.py
├── management
├── settings.py
├── urls.py
└── wsgi.py
</nowiki>

== Toaster's Django applications ==

=== toastergui ===
This is where Toaster's UI related processing happens, the urls.py ,views.py are where the bulk of this happens. All the web pages you see in toaster are processed via this application. As Toaster utilises tables to display information we have a special view class that deals directly with this, cunningly called ToasterTables, this and a few others of our common 'widgets' can be found in widgets.py.

=== orm ===
This is the main database structure which holds the configuration data and the data that is collected from running a build. It also includes some utility and convenience functions on data objects returned. It is just used to define and share the ORM between the other Django applications and doesn't serve any web pages.

=== bldcontrol ===
This is another Django application that is used just to utilise the Django framework rather than actually serve web pages. This module is concerned with the task of polling to see if there is a build scheduled via the runbuilds.py command and launching the build process, which includes commanding bitbake and doing any Git cloning / fetching layers if needed via the localhostbecontroller (we used to have multiple controller implementations), there are a number of additional tables defined to support build creation and build control specific tasks such as a BuildRequest and BuildEnvironment.

=== bldcollector ===
This provides an interface for the build data to be manually uploaded to the orm, it is currently not in development.

=== toastermain ===
This is the overarching/main Django application which loads the 4 above. It is where the default [https://docs.djangoproject.com/en/1.8/ref/settings/ settings] for Toaster are.

== Other ==
=== manage.py ===
Django framework script to manage the framework, includes commands to run unit tests, do database setup, migrations as well as running the development server process.

=== bitbake/bin/toaster ===
Shell script to run if running Toaster in standalone mode on your local system this sets up Toaster's environment, loads the default config, runs the database migrations if needed and grabs the data from the [http://layers.openemebdded.org layerindex] to bootstrap Toaster, it starts the webserver and the runbuilds poller process.

=== toastermain/wsgi.py ===
Django framework script that is the entry point to running the webserver as a wsgi application or as we call it [[Setting up a production instance of Toaster|production setup]]

== Integration with bitbake ==

=== bitbake/lib/bb/ui/toasterui.py and buildinfohelper.py ===
These run in the bitbake observer process and are responsible for converting the events generated by bitbake by a build into objects in Toaster's database.

=== openembedded-core meta/classes/toaster.bbclass ===
The class that extends bitbake in order to generate build events that toaster needs (loaded via INHERIT += "toaster" )

Toaster architecture design

2016-03-30T17:30:13Z

Michael Wood: /* Introduction */

[[Category:Toaster]]

= Introduction =

Toaster project is located in [http://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/bitbake/lib/toaster/ bitbake/lib/toaster].

Toaster is built on the Django framework, if you're not familiar with Django framework, it's very [https://docs.djangoproject.com/en/1.6/intro/tutorial01/ well documented]

The basic layout of the framework is that for each Django application A URL defined in urls.py, maps to a view defined in views.py, the urls and views have a 1:1 mapping to a function or View class in the views.py. These functions/Views runs the server side logic and return the data either to render a [https://docs.djangoproject.com/en/1.8/topics/templates/#the-django-template-language Django template] which can be found in the templates directory or alternatively they can return whatever http response is needed (e.g. a JSON document).

Toaster stores a lot of information about the configuration of projects, layers and recipes, this is stored in [[Toaster database]] which is fully abstracted via [https://docs.djangoproject.com/en/1.8/ref/models/querysets/ Django's object relation mapping (orm)]. This means that we don't have to get our hands dirty with making database connections, caring which kind of database is chosen or mapping data in and out of the database. Toaster's database schema is defined in the ''models.py'' of each of the Django applications that makes up Toaster.

<nowiki>

├── bldcollector
│   ├── admin.py
│   ├── __init__.py
│   ├── urls.py
│   └── views.py
├── bldcontrol
│   ├── admin.py
│   ├── bbcontroller.py
│   ├── __init__.py
│   ├── localhostbecontroller.py
│   ├── management
│   ├── migrations
│   ├── models.py
│   ├── tests.py
│   └── views.py
├── __init__.py
├── manage.py
├── orm
│   ├── __init__.py
│   ├── management
│   ├── migrations
│   ├── models.py
│   └── tests.py
├── toastergui
│   ├── fixtures
│   ├── __init__.py
│   ├── static
│   ├── tablefilter.py
│   ├── tables.py
│   ├── templates
│   ├── templatetags
│   ├── tests.py
│   ├── typeaheads.py
│   ├── urls.py
│   ├── views.py
│   └── widgets.py
└── toastermain
├── __init__.py
├── management
├── settings.py
├── urls.py
└── wsgi.py
</nowiki>

== Toaster's Django applications ==

=== toastergui ===
This is where Toaster's UI related processing happens, the urls.py ,views.py are where the bulk of this happens. All the web pages you see in toaster are processed via this application. As Toaster utilises tables to display information we have a special view class that deals directly with this, cunningly called ToasterTables, this and a few others of our common 'widgets' can be found in widgets.py.

=== orm ===
This is the main database structure which holds the configuration data and the data that is collected from running a build. It also includes some utility and convenience functions on data objects returned. It is just used to define and share the ORM between the other Django applications and doesn't serve any web pages.

=== bldcontrol ===
This is another Django application that is used just to utilise the Django framework rather than actually serve web pages. This module is concerned with the task of polling if there is a build scheduled via the runbuilds.py command and launching the build process, which includes commanding bitbake and doing any Git cloning / fetching layers if needed via the localhostbecontroller (we used to have multiple controller implementations), there are a number of additional tables defined to support build creation and build control specific tasks such as a BuildRequest and BuildEnvironment.

=== bldcollector ===
This provides an interface for the build data to be manually uploaded to the orm, it is currently not in development.

=== toastermain ===
This is the overarching/main Django application which loads the 4 above. It is where the default [https://docs.djangoproject.com/en/1.8/ref/settings/ settings] for Toaster are.

== Other ==
=== manage.py ===
Django framework script to manage the framework, includes commands to run unit tests, do database setup, migrations as well as running the development server process.

=== bitbake/bin/toaster ===
Shell script to run if running Toaster in standalone mode on your local system this sets up Toaster's environment, loads the default config, runs the database migrations if needed and grabs the data from the [http://layers.openemebdded.org layerindex] to bootstrap Toaster, it starts the webserver and the runbuilds poller process.

=== toastermain/wsgi.py ===
Django framework script that is the entry point to running the webserver as a wsgi application or as we call it [[Setting up a production instance of Toaster|production setup]]

== Integration with bitbake ==

=== bitbake/lib/bb/ui/toasterui.py and buildinfohelper.py ===
These run in the bitbake observer process and are responsible for converting the events generated by bitbake by a build into objects in Toaster's database.

=== openembedded-core meta/classes/toaster.bbclass ===
The class that extends bitbake in order to generate build events that toaster needs (loaded via INHERIT += "toaster" )

Toaster architecture design

2016-03-30T17:26:02Z

Michael Wood:

[[Category:Toaster]]

= Introduction =

Toaster project is located in [http://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/bitbake/lib/toaster/ bitbake/lib/toaster].

Toaster is built on the Django framework, if you're not familiar with Django framework, it's very [https://docs.djangoproject.com/en/1.6/intro/tutorial01/ well documented]

The basic layout of the framework is that for each Django application A URL defined in urls.py, this maps to a view defined in views.py, the urls and views have a 1:1 mapping to a function or View class in the views.py. These functions/Views runs the server side logic and returns data either to render a [https://docs.djangoproject.com/en/1.8/topics/templates/#the-django-template-language Django template] which can be found in the templates directory, alternatively they can return whatever http response is needed (e.g. a JSON document).

Toaster stores a lot of information about the configuration of projects, layers and recipes, this is stored in [Toaster database] which is fully abstracted via [https://docs.djangoproject.com/en/1.8/ref/models/querysets/ Django's object relation mapping (orm)]. This means that we don't have to get our hands dirty with making database connections, caring which kind of database is chosen or mapping data in and out of the database. Toaster's database schema is defined in the ''models.py'' of each of the Django applications that makes up Toaster.

<nowiki>

├── bldcollector
│   ├── admin.py
│   ├── __init__.py
│   ├── urls.py
│   └── views.py
├── bldcontrol
│   ├── admin.py
│   ├── bbcontroller.py
│   ├── __init__.py
│   ├── localhostbecontroller.py
│   ├── management
│   ├── migrations
│   ├── models.py
│   ├── tests.py
│   └── views.py
├── __init__.py
├── manage.py
├── orm
│   ├── __init__.py
│   ├── management
│   ├── migrations
│   ├── models.py
│   └── tests.py
├── toastergui
│   ├── fixtures
│   ├── __init__.py
│   ├── static
│   ├── tablefilter.py
│   ├── tables.py
│   ├── templates
│   ├── templatetags
│   ├── tests.py
│   ├── typeaheads.py
│   ├── urls.py
│   ├── views.py
│   └── widgets.py
└── toastermain
├── __init__.py
├── management
├── settings.py
├── urls.py
└── wsgi.py
</nowiki>

== Toaster's Django applications ==

=== toastergui ===
This is where Toaster's UI related processing happens, the urls.py ,views.py are where the bulk of this happens. All the web pages you see in toaster are processed via this application. As Toaster utilises tables to display information we have a special view class that deals directly with this, cunningly called ToasterTables, this and a few others of our common 'widgets' can be found in widgets.py.

=== orm ===
This is the main database structure which holds the configuration data and the data that is collected from running a build. It also includes some utility and convenience functions on data objects returned. It is just used to define and share the ORM between the other Django applications and doesn't serve any web pages.

=== bldcontrol ===
This is another Django application that is used just to utilise the Django framework rather than actually serve web pages. This module is concerned with the task of polling if there is a build scheduled via the runbuilds.py command and launching the build process, which includes commanding bitbake and doing any Git cloning / fetching layers if needed via the localhostbecontroller (we used to have multiple controller implementations), there are a number of additional tables defined to support build creation and build control specific tasks such as a BuildRequest and BuildEnvironment.

=== bldcollector ===
This provides an interface for the build data to be manually uploaded to the orm, it is currently not in development.

=== toastermain ===
This is the overarching/main Django application which loads the 4 above. It is where the default [https://docs.djangoproject.com/en/1.8/ref/settings/ settings] for Toaster are.

== Other ==
=== manage.py ===
Django framework script to manage the framework, includes commands to run unit tests, do database setup, migrations as well as running the development server process.

=== bitbake/bin/toaster ===
Shell script to run if running Toaster in standalone mode on your local system this sets up Toaster's environment, loads the default config, runs the database migrations if needed and grabs the data from the [http://layers.openemebdded.org layerindex] to bootstrap Toaster, it starts the webserver and the runbuilds poller process.

=== toastermain/wsgi.py ===
Django framework script that is the entry point to running the webserver as a wsgi application or as we call it [[Setting up a production instance of Toaster|production setup]]

== Integration with bitbake ==

=== bitbake/lib/bb/ui/toasterui.py and buildinfohelper.py ===
These run in the bitbake observer process and are responsible for converting the events generated by bitbake by a build into objects in Toaster's database.

=== openembedded-core meta/classes/toaster.bbclass ===
The class that extends bitbake in order to generate build events that toaster needs (loaded via INHERIT += "toaster" )

Toaster architecture design

2016-03-30T17:22:04Z

Michael Wood: /* Other */

[[Category:Toaster]]

= Introduction =

Toaster project is located in [http://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/bitbake/lib/toaster/ bitbake/lib/toaster].

Toaster is built on the Django framework, if you're not familiar with Django framework, it's very [https://docs.djangoproject.com/en/1.6/intro/tutorial01/ well documented]

The basic layout of the framework is that for each Django application A URL defined in urls.py, this maps to a view defined in views.py, the urls and views have a 1:1 mapping to a function or View class in the views.py. These functions/Views runs the server side logic and returns data either to render a [https://docs.djangoproject.com/en/1.8/topics/templates/#the-django-template-language Django template] which can be found in the templates directory, alternatively they can return whatever http response is needed (e.g. a JSON document).

Toaster stores a lot of information about the configuration of projects, layers and recipes, this is stored in [Toaster database] which is fully abstracted via [https://docs.djangoproject.com/en/1.8/ref/models/querysets/ Django's object relation mapping (orm)]. This means that we don't have to get our hands dirty with making database connections, caring which kind of database is chosen or mapping data in and out of the database. Toaster's database schema is defined in the ''models.py'' of each of the Django applications that makes up Toaster.

<nowiki>

├── bldcollector
│   ├── admin.py
│   ├── __init__.py
│   ├── urls.py
│   └── views.py
├── bldcontrol
│   ├── admin.py
│   ├── bbcontroller.py
│   ├── __init__.py
│   ├── localhostbecontroller.py
│   ├── management
│   ├── migrations
│   ├── models.py
│   ├── tests.py
│   └── views.py
├── __init__.py
├── manage.py
├── orm
│   ├── __init__.py
│   ├── management
│   ├── migrations
│   ├── models.py
│   └── tests.py
├── toastergui
│   ├── fixtures
│   ├── __init__.py
│   ├── static
│   ├── tablefilter.py
│   ├── tables.py
│   ├── templates
│   ├── templatetags
│   ├── tests.py
│   ├── typeaheads.py
│   ├── urls.py
│   ├── views.py
│   └── widgets.py
└── toastermain
├── __init__.py
├── management
├── settings.py
├── urls.py
└── wsgi.py
</nowiki>

== Toaster's Django applications ==

=== toastergui ===
This is where Toaster's UI and related processing happens, the urls.py ,views.py are where the bulk of this is. All the web pages you see in toaster are processed via this application. As toaster utilises Tables to display information we have a special view class that deals directly with this cunningly called ToasterTables, this and a few others of our common 'widgets' can be found in widgets.py.

=== orm ===
This is the main database structure which holds the configuration data and the data that is collected from running a build. It also includes some utility and convenience functions on data objects returned. It is just used to define and share the ORM between the other Django applications and doesn't serve any web pages.

=== bldcontrol ===
This is another Django application that is used just to utilise the Django framework rather than actually serve web pages. This module is concerned with the task of polling if there is a build scheduled via the runbuilds.py command and launching the build process, which includes commanding bitbake and doing any Git cloning / fetching layers if needed via the localhostbecontroller (we used to have multiple controller implementations), there are a number of additional tables defined to support build creation and build control specific tasks such as a BuildRequest and BuildEnvironment.

=== bldcollector ===
This provides an interface for the build data to be manually uploaded to the orm, it is currently not in development.

=== toastermain ===
This is the overarching/main Django application which loads the 4 above. It is where the default [https://docs.djangoproject.com/en/1.8/ref/settings/ settings] for Toaster are.

== Other ==
=== manage.py ===
Django framework script to manage the framework, includes commands to run unit tests, do database setup, migrations as well as running the development server process.

=== bitbake/bin/toaster ===
Shell script to run if running Toaster in standalone mode on your local system this sets up Toaster's environment, loads the default config, runs the database migrations if needed and grabs the data from the [http://layers.openemebdded.org layerindex] to bootstrap Toaster, it starts the webserver and the runbuilds poller process.

=== toastermain/wsgi.py ===
Django framework script that is the entry point to running the webserver as a wsgi application or as we call it [[Setting up a production instance of Toaster|production setup]]

=== bitbake/lib/bb/ui/toasterui.py and buildinfohelper.py ===
These run in the bitbake observer process and are responsible for converting the events generated by bitbake by a build into objects in Toaster's database.

=== openembedded-core meta/classes/toaster.bbclass ===
The class that extends bitbake in order to generate build events that toaster needs (loaded via INHERIT += "toaster" )

Toaster architecture design

2016-03-30T17:21:03Z

Michael Wood: /* Other */

[[Category:Toaster]]

= Introduction =

Toaster project is located in [http://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/bitbake/lib/toaster/ bitbake/lib/toaster].

Toaster is built on the Django framework, if you're not familiar with Django framework, it's very [https://docs.djangoproject.com/en/1.6/intro/tutorial01/ well documented]

The basic layout of the framework is that for each Django application A URL defined in urls.py, this maps to a view defined in views.py, the urls and views have a 1:1 mapping to a function or View class in the views.py. These functions/Views runs the server side logic and returns data either to render a [https://docs.djangoproject.com/en/1.8/topics/templates/#the-django-template-language Django template] which can be found in the templates directory, alternatively they can return whatever http response is needed (e.g. a JSON document).

Toaster stores a lot of information about the configuration of projects, layers and recipes, this is stored in [Toaster database] which is fully abstracted via [https://docs.djangoproject.com/en/1.8/ref/models/querysets/ Django's object relation mapping (orm)]. This means that we don't have to get our hands dirty with making database connections, caring which kind of database is chosen or mapping data in and out of the database. Toaster's database schema is defined in the ''models.py'' of each of the Django applications that makes up Toaster.

<nowiki>

├── bldcollector
│   ├── admin.py
│   ├── __init__.py
│   ├── urls.py
│   └── views.py
├── bldcontrol
│   ├── admin.py
│   ├── bbcontroller.py
│   ├── __init__.py
│   ├── localhostbecontroller.py
│   ├── management
│   ├── migrations
│   ├── models.py
│   ├── tests.py
│   └── views.py
├── __init__.py
├── manage.py
├── orm
│   ├── __init__.py
│   ├── management
│   ├── migrations
│   ├── models.py
│   └── tests.py
├── toastergui
│   ├── fixtures
│   ├── __init__.py
│   ├── static
│   ├── tablefilter.py
│   ├── tables.py
│   ├── templates
│   ├── templatetags
│   ├── tests.py
│   ├── typeaheads.py
│   ├── urls.py
│   ├── views.py
│   └── widgets.py
└── toastermain
├── __init__.py
├── management
├── settings.py
├── urls.py
└── wsgi.py
</nowiki>

== Toaster's Django applications ==

=== toastergui ===
This is where Toaster's UI and related processing happens, the urls.py ,views.py are where the bulk of this is. All the web pages you see in toaster are processed via this application. As toaster utilises Tables to display information we have a special view class that deals directly with this cunningly called ToasterTables, this and a few others of our common 'widgets' can be found in widgets.py.

=== orm ===
This is the main database structure which holds the configuration data and the data that is collected from running a build. It also includes some utility and convenience functions on data objects returned. It is just used to define and share the ORM between the other Django applications and doesn't serve any web pages.

=== bldcontrol ===
This is another Django application that is used just to utilise the Django framework rather than actually serve web pages. This module is concerned with the task of polling if there is a build scheduled via the runbuilds.py command and launching the build process, which includes commanding bitbake and doing any Git cloning / fetching layers if needed via the localhostbecontroller (we used to have multiple controller implementations), there are a number of additional tables defined to support build creation and build control specific tasks such as a BuildRequest and BuildEnvironment.

=== bldcollector ===
This provides an interface for the build data to be manually uploaded to the orm, it is currently not in development.

=== toastermain ===
This is the overarching/main Django application which loads the 4 above. It is where the default [https://docs.djangoproject.com/en/1.8/ref/settings/ settings] for Toaster are.

== Other ==
- manage.py - Django framework script to manage the framework, includes commands to run unit tests, do database setup, migrations as well as running the development server process.
- bitbake/bin/toaster - Shell script to run if running Toaster in standalone mode on your local system this sets up Toaster's environment, loads the default config, runs the database migrations if needed and grabs the data from the [http://layers.openemebdded.org layerindex] to bootstrap Toaster, it starts the webserver and the runbuilds poller process.
- toastermain/wsgi.py - Django framework script that is the entry point to running the webserver as a wsgi application or as we call it [[Setting up a production instance of Toaster|production setup]]
- bitbake/lib/bb/ui/toasterui.py and buildinfohelper.py - These run in the bitbake observer process and are responsible for converting the events generated by bitbake by a build into objects in Toaster's database.
- openembedded-core meta/classes/toaster.bbclass - The class that extends bitbake in order to generate build events that toaster needs (loaded via INHERIT += "toaster" )

Toaster architecture design

2016-03-30T17:12:23Z

Michael Wood:

[[Category:Toaster]]

= Introduction =

Toaster project is located in [http://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/bitbake/lib/toaster/ bitbake/lib/toaster].

Toaster is built on the Django framework, if you're not familiar with Django framework, it's very [https://docs.djangoproject.com/en/1.6/intro/tutorial01/ well documented]

The basic layout of the framework is that for each Django application A URL defined in urls.py, this maps to a view defined in views.py, the urls and views have a 1:1 mapping to a function or View class in the views.py. These functions/Views runs the server side logic and returns data either to render a [https://docs.djangoproject.com/en/1.8/topics/templates/#the-django-template-language Django template] which can be found in the templates directory, alternatively they can return whatever http response is needed (e.g. a JSON document).

Toaster stores a lot of information about the configuration of projects, layers and recipes, this is stored in [Toaster database] which is fully abstracted via [https://docs.djangoproject.com/en/1.8/ref/models/querysets/ Django's object relation mapping (orm)]. This means that we don't have to get our hands dirty with making database connections, caring which kind of database is chosen or mapping data in and out of the database. Toaster's database schema is defined in the ''models.py'' of each of the Django applications that makes up Toaster.

<nowiki>

├── bldcollector
│   ├── admin.py
│   ├── __init__.py
│   ├── urls.py
│   └── views.py
├── bldcontrol
│   ├── admin.py
│   ├── bbcontroller.py
│   ├── __init__.py
│   ├── localhostbecontroller.py
│   ├── management
│   ├── migrations
│   ├── models.py
│   ├── tests.py
│   └── views.py
├── __init__.py
├── manage.py
├── orm
│   ├── __init__.py
│   ├── management
│   ├── migrations
│   ├── models.py
│   └── tests.py
├── toastergui
│   ├── fixtures
│   ├── __init__.py
│   ├── static
│   ├── tablefilter.py
│   ├── tables.py
│   ├── templates
│   ├── templatetags
│   ├── tests.py
│   ├── typeaheads.py
│   ├── urls.py
│   ├── views.py
│   └── widgets.py
└── toastermain
├── __init__.py
├── management
├── settings.py
├── urls.py
└── wsgi.py
</nowiki>

== Toaster's Django applications ==

=== toastergui ===
This is where Toaster's UI and related processing happens, the urls.py ,views.py are where the bulk of this is. All the web pages you see in toaster are processed via this application. As toaster utilises Tables to display information we have a special view class that deals directly with this cunningly called ToasterTables, this and a few others of our common 'widgets' can be found in widgets.py.

=== orm ===
This is the main database structure which holds the configuration data and the data that is collected from running a build. It also includes some utility and convenience functions on data objects returned. It is just used to define and share the ORM between the other Django applications and doesn't serve any web pages.

=== bldcontrol ===
This is another Django application that is used just to utilise the Django framework rather than actually serve web pages. This module is concerned with the task of polling if there is a build scheduled via the runbuilds.py command and launching the build process, which includes commanding bitbake and doing any Git cloning / fetching layers if needed via the localhostbecontroller (we used to have multiple controller implementations), there are a number of additional tables defined to support build creation and build control specific tasks such as a BuildRequest and BuildEnvironment.

=== bldcollector ===
This provides an interface for the build data to be manually uploaded to the orm, it is currently not in development.

=== toastermain ===
This is the overarching/main Django application which loads the 4 above. It is where the default [https://docs.djangoproject.com/en/1.8/ref/settings/ settings] for Toaster are.

== Other ==
- manage.py - Django framework script to manage the framework, includes commands to run unit tests, do database setup, migrations as well as running the development server process.
- bitbake/bin/toaster - Shell script to run if running Toaster in standalone mode on your local system this sets up Toaster's environment, loads the default config, runs the database migrations if needed and grabs the data from the [http://layers.openemebdded.org layerindex] to bootstrap Toaster, it starts the webserver and the runbuilds poller process.
- toastermain/wsgi.py - Django framework script that is the entry point to running the webserver as a wsgi application or as we call it [[Setting up a production instance of Toaster|production setup]]

Toaster architecture design

2016-03-30T17:01:32Z

Michael Wood:

[[Category:Toaster]]

Toaster project is located in [http://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/bitbake/lib/toaster/ bitbake/lib/toaster].

Toaster is built on the Django framework, if you're not familiar with Django framework, it's very [https://docs.djangoproject.com/en/1.6/intro/tutorial01/ well documented]

The basic layout of the framework is that for each Django application A URL defined in urls.py, this maps to a view defined in views.py, the urls and views have a 1:1 mapping to a function or View class in the views.py. These functions/Views runs the server side logic and returns data either to render a [https://docs.djangoproject.com/en/1.8/topics/templates/#the-django-template-language Django template] which can be found in the templates directory, alternatively they can return whatever http response is needed (e.g. a JSON document).

Toaster stores a lot of information about the configuration of projects, layers and recipes, this is stored in [Toaster database] which is fully abstracted via [https://docs.djangoproject.com/en/1.8/ref/models/querysets/ Django's object relation mapping (orm)]. This means that we don't have to get our hands dirty with making database connections, caring which kind of database is chosen or mapping data in and out of the database. Toaster's database schema is defined in the ''models.py'' of each of the Django applications that makes up Toaster.

== Toaster's Django applications ==

=== toastergui ===
This is where Toaster's UI and related processing happens, the urls.py ,views.py are where the bulk of this is. All the web pages you see in toaster are processed via this application. As toaster utilises Tables to display information we have a special view class that deals directly with this cunningly called ToasterTables, this and a few others of our common 'widgets' can be found in widgets.py.

=== orm ===
This is the main database structure which holds the configuration data and the data that is collected from running a build. It also includes some utility and convenience functions on data objects returned. It is just used to define and share the ORM between the other Django applications and doesn't serve any web pages.

=== bldcontrol ===
This is another Django application that is used just to utilise the Django framework rather than actually serve web pages. This module is concerned with the task of polling if there is a build scheduled via the runbuilds.py command and launching the build process, which includes commanding bitbake and doing any Git cloning / fetching layers if needed via the localhostbecontroller (we used to have multiple controller implementations), there are a number of additional tables defined to support build creation and build control specific tasks such as a BuildRequest and BuildEnvironment.

=== bldcollector ===
This provides an interface for the build data to be manually uploaded to the orm, it is currently not in development.

=== toastermain ===
This is the overarching/main Django application which loads the 4 above. It is where the default [https://docs.djangoproject.com/en/1.8/ref/settings/ settings] for Toaster are.

=== Other ===
- manage.py - Django framework script to manage the framework, includes commands to run unit tests, do database setup, migrations as well as running the development server process.
- bitbake/bin/toaster - Shell script to run if running Toaster in standalone mode on your local system this sets up Toaster's environment, loads the default config, runs the database migrations if needed and grabs the data from the [http://layers.openemebdded.org layerindex] to bootstrap Toaster, it starts the webserver and the runbuilds poller process.
- toastermain/wsgi.py - Django framework script that is the entry point to running the webserver as a wsgi application or as we call it [[Setting up a production instance of Toaster|production setup]]

Toaster and bitbake communications

2016-03-30T15:47:12Z

Michael Wood:

[[Category:Toaster]]

This is a write up of some investigation I did into how toaster asks bitbake to do stuff, and how it hears about what bitbake is doing. It's not definitive and not guaranteed to be correct, but it might provide some pointers if you are working on toaster. If anyone knows better, please feel free to correct this.

Any paths given below are relative to the root of the poky/poky-contrib source tree.

== toaster asking bitbake to do stuff ==

First off I wanted to figure out how clicking on the "Build" button in the toaster interface triggers a bitbake build.

toaster only asks bitbake to perform builds when in managed mode. This is the point I started from when doing this investigation. I think analysis mode is very similar, except toaster doesn't ask bitbake to do anything, it just listens to stuff which is already happening.

=== runbuilds ===

The conversation with bitbake is handled via the runbuilds command (bitbake/lib/toaster/bldcontrol/management/commands/runbuilds.py). This command is run in a loop once every second from the toaster start script (bitbake/bin/toaster).

The runbuilds.schedule() function looks for BuildRequests in the database. A '''BuildRequest''' is created each time you press a "Build" button in the toaster web interface. Those build requests are created in orm/model.py schedule_build on the project object. When creating this BuildRequest also created are BRLayer for each of the layers in your project, BRTarget for the target to be built and BRBitbake for the version of bitbake to use, these are all defined in bldcontrol/models.py, and contain copies of the information from the project.

The localhostbecontroller takes this BuildRequest

* schedule() gets a localhostbecontroller (be = build environment) instance (there is a becontroller for remote bitbakes, but I don't think the implementation is complete) and assigns it to a variable bec.

* schedule() then calls bec.triggerBuild() for a build request which is in the BuildRequest.REQ_QUEUED state. Only one build request gets picked up each time the runbuilds script runs: the one with the largest ID.

* schedule() also creates a build identification variable for each build request, combining the primary key of the build request with the primary key of the build environment controller's build environment (!)).

=== localhostbecontroller ===

This is the build environment controller, which handles instances of build environments. It is also responsible for cloning any layers that are in the build request.

It is a subclass of BuildEnvironmentController.

* triggerBuild() calls getBBController(), which returns a BitBakeController instance (bbctrl). getBBController() actually instantiates the controller if it isn't already instantiated, passing it a "server" object. This server object is an instance of bb.server.xmlrpc.BitBakeXMLRPCClient().

* triggerBuild() then calls the BitBakeController build() method: bbctrl.build()

=== BitBakeController (bbctrl) ===

This is constructed with the server object, which is an XMLRPC connection to a BitBake server.

* build() invokes the "buildTargets" command on the connection using runCommand().

=== BitBakeXMLRPCClient (bitbake/lib/bb/server/xmlrpc.py) ===

This is a subclass of BitBakeServer.

It has a "connection" member, which has runCommand() invoked on it.

When constructed by getBBController(), initServer() is called as part of its construction. This is a method from BitBakeServer which sets the interface address (default: (localhost,0)) and creates an XMLRPCServer() using this interface address.

Once initServer() is done, establishConnection() is invoked, which creates the "real" socket.

The established connection (which runCommand() is invoked on) is a BitBakeXMLRPCServerConnection.

=== BitBakeServer ===

BitbakeServer is a subclass of BitBakeBaseServer (declared in bitbake/lib/bb/server/__init__.py); this actually has the addcooker() method which associates a bitbake cooker instance with the connection.

BitBakeBaseServer acts like a decorator around a server implementation; in our case, it's a BitBakeXMLRPCServerConnection.

Calling addcooker() on the BitBakeBaseServer also calls it on any server implementation (serverImpl) which is wrapped by BitBakeServer.

=== BitBakeXMLRPCServerConnection ===

This is a subclass of BitBakeBaseServerConnection.

When runCommand() is invoked on this, it's passed on to the cooker object associated with it, i.e. it actually invokes self.cooker.command.runCommand().

The cooker object is associated with the BitBakeXMLRPCServerConnection when the bitbake/lib/bb/main.py script runs.

=== main.py ===

The main script which starts a bitbake server and ui. This is what runs when you use "bitbake" from the command line.

toaster starts the bitbake server with the --server-only switch, which calls bitbake_main() in this file; this in turn calls the main() function.

This instantiates a bb.cooker.BBCooker and adds it to the server implementation via addcooker(). The cooker is what actually enables commands to be sent to the bitbake server.

=== BBCooker (bitbake/lib/bb/cooker.py) ===

This has a "command" property which is an instance of BBCommand; this is what runCommand() finally gets invoked on.

=== BBCommand ===

runCommand() calls a method from an instance of CommandSync (all the synchronous commands bitbake understands) or CommandAsync (the asynchronous ones), depending on the type of command passed to runCommand(). Both the CommandSync and CommandAsync instances are added to the BBCooker when it is created.

For example, runCommand("getVariable", ...) is invoked via CommandsSync.getVariable().

The commands which go through BBCommand.runCommand() now make their way to the bitbake server over XMLRPC.

== toaster listening to what bitbake is doing ==

At this point, I realised I probably understood enough to see how bitbake was being invoked from toaster: asking toaster to start a build sends a "buildTargets" command to the bitbake XMLRPC server, via a rather indirect series of objects and method calls.

What I wanted to know now was how toaster listens to the result of that command. At a high level, I understood that it gathered events from the XMLRPC connection and converted them into database objects. However, I didn't really get the code path.

I started from BuildInfoHelper, which is where bitbake events are converted into toaster db objects.

=== BuildInfoHelper ===

This is passed build events from the toasterui.py which routes them off to the buildinfohelper based on the event types.

BuildInfoHelper is responsible for constructing toaster ORM objects from events. The BuildInfoHelper is constructed with a server (instance of BitBakeXMLRPCServerConnection in the case of toaster), so it can also interrogate the bitbake server for extra environmental data via getVariable().

BuildInfoHelper also tries to match the data from the build recipe events from toasterui.py to data in toaster's database, so that toaster can "learn" from the build, this means that we get recipe and package information from the build which otherwise we are unable to determine.

The task of matching the recipes and associated data is done when we get the recipe information from the store task event, this contains a path of the recipe e.g. "/home/yocto/cloned_layers/_mylayer_master.toaster_cloned/mydir/example.bb... the buildinfohelper then tries to find a layer in toaster that has a checkout directory location that starts with the one provided by the event.

It does this by asking the "bc" BuildController (which can only be localhostbecontroller.py at the moment as this is the only controller which contains the implementation) to return the git checkout path via 'getGitCloneDirectory' this returns what it believes was the git checkout base location, in our example hopefully something like "/home/yocto/cloned_layers/_mylayer_master.toaster_cloned" then the buildinfohelper adds the directory name (brl.dirpath) from the information in the build request layer, e.g. "mydir" after all this reconstruction we hope that the recipe path we got from the event starts with the reconstructed path. e.g. Does "/home/yocto/cloned_layers/_mylayer_master.toaster_cloned/mydir/example.bb" start with "/home/yocto/cloned_layers/_mylayer_master.toaster_cloned/mydir/" if yes then we have a layer in toaster to which we can associate the information from the build to.

One problem here is that the getGitCloneDirectory function a) doesn't exist in the other hostcontrollers and b) doesn't always return the correct value due to some special case

=== toasterui.py ===

This has a main() function which sets up an event listener loop.

main() is passed a server, eventHandler and params. The eventHandler is the object which listens to bitbake events.

toasterui is called from main.py (see below).

=== main.py (bitbake/lib/bb/main.py) ===

The toasterui.py main() method is invoked from main.py.

It's passed two parameters:

# server_connection.connection
# server_connection.events

The second of these is the eventHandler which is set up in a loop in toasterui.py.

server_connection comes from a call to establishConnection() in main.py. The server is constructed via a servermodule, which is dynamically chosen in main.py according to the parameters used to invoke it (-t). It will either be "process" or "xmlrpc".

The UI type is set in the main.py script via the -u option.

toaster invokes bitbake with: -t xmlrpc -u toasterui

which means that we get the toasterui.main() called, and we get an xmlrpc bitbake server.

Back to server_connection.events...

This refers to an xmlrpc bitbake server connection's events object.

So toasterui.main() gets:

# server_connection.connection => BitBakeXMLRPCServerConnection.connection
# server_connection.events => BitBakeXMLRPCServerConnection.events

The events object is an instance of uievent.BBUIEventQueue in our case, as we're using toasterui.

=== uievent.BBUIEventQueue (bitbake/lib/bb/ui/uievent.py) ===

This is our event handler ("events") in toasterui.main(), which is receiving events on the XMLRPC connection (see later).

toasterui.main() calls events.waitEvent(0.25), which looks for events on the queue every 0.25s. If the queue has events, one is popped off.

Each event popped from the queue is passed to uihelper.BBUIHelper.eventHandler(), which adds build tracking information (how many packages built, tasks completed etc.)

The event then goes to the BuildInfoHelper, where it gets stored in toaster's database.

==== How do events get on BBUIEventQueue? ====

BBUIEventQueue is instantiated on the BitBakeXMLRPCServerConnection object.

It is passed a BBServer, which is an xmlrpclib.ServerProxy (from the Python standard library).

The ServerProxy has a method corresponding to each method on the server it is proxying for; in this case, the proxied server is a BBServer, so the proxy has the same methods as BBServer.

Event handling is set up by calling self.BBServer.registerEventHandler() from BBUIEventQueue.

registerEventHandler() is defined in bitbake/lib/bb/server/xmlrpc.py, BitBakeServerCommands. This in turn calls bb.event.register_UIHandler, defined in bitbake/lib/bb/event.py.

When an event is fired, each handler registered for it is invoked with that event (in event.py).

The main function for firing events in event.py is fire_from_worker(), which is called from bitbake/lib/bb/runqueue.py.

Events are constructed from xmlrpc messages coming from the bitbake server (see runqueue.py, runQueuePipe.read()).

==== How do toasterui + buildinfohelper manage events? ====

toasterui runs as part of the bitbake instance: bitbake is invoked with a -u toasterui option, which means that toasterui is instantiated as the event handler for the bitbake instance.

As events occur in bitbake, they are passed to toasterui; toasterui then hands those events off to buildinfohelper, which uses the event data to create records in the Toaster database. Note that buildinfohelper has to start up the Django database machinery manually for this to be possible, and that this is happening outside the main Django instance, in a separate process. This might be part of the reason why we get database locking issues with Django 1.8.

As buildinfohelper receives bitbake events, it sets variables in its internal_state dictionary. These variables are used to represent events which are "partial" from the perspective of Toaster: that is, events which can't create a complete, useful record in Toaster's database.

The best example of this comes from bitbake's Task* events. We receive two events for a task in buildinfohelper:

# The event notifying that the task started, e.g. TaskStarted, runQueueTaskStarted, sceneQueueTaskStarted
# The end state of the task, e.g. TaskFailedSilent, TaskCompleted, runQueueTaskFailed, sceneQueueTaskFailed, runQueueTaskCompleted, sceneQueueTaskCompleted, runQueueTaskSkipped

Because Toaster represents a Task as a single entity, with an outcome state, we can't add a complete Task record when a task starts: we add a partial Task record, then update that when the task end event is received.

To get this to work, buildinfohelper saves the partial Task to the database when the task start event is received; then retrieves and updates that record when the task end event arrives. In between these two points, buildinfohelper keeps some internal state about tasks which have started but which don't have a "done" outcome. However, the identifiers used in this internal state are composed of the task file + name, which is a fairly arbitrary algorithm (I quote: "we do a bit of guessing"). When the task end event is received, it is matched up to the internal state (where we have a list of tasks which haven't ended) using this fairly arbitrary identifier. This seems like it might be prone to error.

The reason this approach has been used, though, is because (as far as I can tell) bitbake doesn't provide any identifiers on its events which would enable them to be tied together. A TaskStarted event doesn't specify which task started (tasks don't have unique IDs), just the name of that task and its .bb file; there is also nothing to tie a task to the build it's associated with. This means that any event handler waiting for bitbake events has to manually tie together events and maintain local state to be able to do that.

The following internal state is maintained in buildinfohelper for this purpose:

* lvs (layer versions): layer versions known to Toaster
* recipes: recipes known to Toaster
* backlog: list of log events which haven't been saved for the current build yet
* brbe: the build request and the build environment primary keys, concatenated together around a colon (e.g. "1:2")
* build: the ongoing build (there's an implicit assumption that bitbake can only run one build at a time)
* taskdata: a list of ongoing tasks (i.e. task events for which we have received a task started event, but haven't yet received a task ended event); as task end events are received, the task goes into the database and is removed from taskdata
* targets: targets for the current build
* task_order: a counter which is added to each task as it is saved to the database, which marks the order in which the task events were received; it gets incremented each time a certain type of task event is received

Note that lvs and recipes are set before the build started event, using metadata events fired by bitbake.

While the build is ongoing, the internal state affects how data is added to the database: it is used to implicitly associate events with the build which is assumed to occur between a BuildStarted event and the next BuildCompleted/BuildFailed event.

Here's an example of the workflow to make it clearer:

# toasterui receives a BuildStarted event and passes it to buildinfohelper.
# buildinfohelper creates a database record B for the new build, and stores it as a property on itself.
# toasterui receives a TaskStarted event with name "foo" and file "/bar/bar/humbug"; it passes the event to buildinfohelper.
# buildinfohelper stores the partial task in the database as T, associating it with the build record B; it also stores T under the key "/bar/bar/humbug:foo" in taskdata.
# toasterui receives a TaskCompleted event with name "foo" and file "/bar/bar/humbug"; it passes the event to buildinfohelper.
# buildinfohelper marries up the new TaskCompleted event with the partial record T, by matching the new event's key "/bar/bar/humbug:foo" with the key for the TaskStarted event which is already in taskdata (see 4); T is updated in the database.
# toasterui receives a BuildCompleted event and passes it to buildinfohelper.
# buildinfohelper assumes that the BuildCompleted event applies to the build B already stored in its internal state. It updates the database record for B with data about when the build ended, build artifacts etc.
# toasterui creates a new buildinfohelper, ready to deal with the next build.

backlog is a list of log events which haven't yet been attached to a build; if a log event occurs before the build has been saved, it is added to the list; then, once a BuildStarted event has occurred, the list of events in the backlog is added to the database and associated with that build.

== Conclusion ==

At this point, I felt I understood enough about how events are processed for my purpose, so I didn't dig any further. I knew where I could amend an event to add/remove properties on it, which is what I was after.

Contribute to Toaster

2016-02-22T17:03:31Z

Michael Wood: /* Submitting patch sets for integration into Bitbake */

File:Toaster orm graph.png

2016-02-10T18:09:07Z

Michael Wood: Michael Wood uploaded a new version of "File:Toaster orm graph.png"

Toaster ORM graph

File:Toaster orm graph.png

2016-02-10T18:01:13Z

Michael Wood: Michael Wood uploaded a new version of "File:Toaster orm graph.png"

Toaster ORM graph

File:Toaster orm graph.png

2016-02-10T17:57:46Z

Michael Wood: Michael Wood uploaded a new version of "File:Toaster orm graph.png": Updated tables for image customisation

Toaster ORM graph

Toaster database

2016-02-10T17:52:14Z

Michael Wood: /* What running a Build affects */

[[Category:Toaster]]
[[File:Toaster_orm_graph.png|thumb|ORM diagram]]
Toaster Uses [https://docs.djangoproject.com/en/dev/topics/db/models/ Django] and it's [https://en.wikipedia.org/wiki/Object-relational_mapping ORM] for database interaction.

The database table/model definitions can be found in the [http://cgit.openembedded.org/bitbake/tree/lib/toaster/orm/models.py orm/models.py] file.
We also have additional models defined for the build controller, these are generally the 'behind the scenes' way in which we translate toaster information into the build controller process. This page is a '''high level''' look at the database design and how it is used.

== Categories of data in Toaster ==

=== Build data ===
Build data is Layers, Layer_Versions, Recipes, Machines, Packages, Dependencies and all other objects that are output by running a build of a recipe. '''Once it is created it is never edited or updated'''. Unless deleted by the user.

=== Configuration data ===
Configuration data is data which is provided by Toaster to configure an openembedded based project in Toaster, like Build data this is stored as Layers, Layer_Versions, Recipes and Machines etc. The configuration data can be produced by fetching it from the [http://layers.openembedded.org Layer Index] (the toaster start up script attempts to do this for you at first run) from importing information from the toasterconf.json file, importing Layer definitions into Projects via the Import layer page and also from running a build.

=== Project data ===

This is the container for Configuration data and Build data. Additionally it contains project specific data such as user imported Layers and CustomImageRecipe.

== Top level models ==
There are many intermediate tables in Toaster's database, see diagram. Below are the most top level models on which everything else depends.
These models map to the metadata [http://www.yoctoproject.org/docs/1.8/bitbake-user-manual/bitbake-user-manual.html#Concepts concepts] that make up openembedded/Bitbake.

=== Layer and Layer_Version ===
A Layer_Version is the model in which most other models relate to in Toaster. Every Layer_Version object belongs to a Layer object which contains the common metadata for each 'version' or revision of the Layer.

An example would be that we have a Layer called 'meta-openembedded' which has 3 Layer_Versions; one Layer_Version which is set to the release 'dizzy' one on 'fido' and one on 'master' (in most cases these correspond to git branch names)

'''Layer 1 - N Layer_Version 1 - N Recipe 1 - N Packages'''

=== Recipe ===

Recipes contain the information about a collection or particular piece of software. This can be the ''name'', ''version'' and ''description'' and any distribution ''package''s are created by the Recipe.

An example would be a recipe called myrecipe version 1.2 which produces 3 Packages myrecipe-dev, myrecipe-dbg and myrecipe. Recipes are the objects which get built by BitBake.

Recipe N - 1 Layer_Version

Recipe 1 - N Package

==== CustomImageRecipe ====

This is a special kind of Recipe which has three additional fields; base_recipe, last_updated and project. base_recipe is the Recipe which the Custom recipe has been based on and is the recipe which will be copied in when the Custom recipe is generated. The last_updated field is used internally to keep track of the last time the package list was last updated from a build.

=== Package ===

A Package is the representation of distribution packages produced by a Recipe when it has been built. Package data is almost always Build data since we don't know about packages until they have been produced. The exception is when the package data is used as configuration data to Customise a recipe by adding and removing known packages.

Package N - 1 Recipe

==== CustomImagePackage ====
This is a subclass of Package which contains 3 extra fields. ''recipe_includes'' are for keeping track of the packages which are included as part of the base recipe when it is built. e.g. the ''base_recipe'' has package busybox. ''recipe_excludes'' are packages which are normally included as defined by the base_recipe which have been explicitly asked to be excluded by the user. ''recipe_appends'' are the packages which are being appended to this recipe (think IMAGE_INSTALL +=)

These objects are configuration data and are generated by running builds. When a build runs we create or update CustomImagePackages meaning that each build helps to create a pool of packages which are available to the user to add or remove based on the available recipes when creating a CustomImageRecipe

CustomImagePackage N - N CustomImageRecipe

=== Build ===

The Build model contains the metadata for reporting information about a Build. It is the top level object for many of the Build related models which contain output from a Build. The Build model also includes when the Build started and stopped, and the status of the build. Any data in toaster that has a build object related to it is considered Build data.

Build 1 - 1 Target 1 - N Target_File

Build 1 - N Task

Build 1 - N BuildArtifact

Build 1 - N LogMessage 1 - 1 Task

Build 1 - N Variable

Build 1 - N Layer_Version

=== Target ===

A target is the instruction on what to build that is sent to Bitbake. When a build is triggered a Target object is created which records the one or more Recipe names to be built and the [http://www.yoctoproject.org/docs/latest/bitbake-user-manual/bitbake-user-manual.html#tasks Task] that you're asking BitBake to do (no task supplied executes the default which is build the recipe).

Target 1 - 1 Build

=== Project ===

The project model is the container for the configuration data of your current project, this includes the release, the Layer_Versions that you're using in your project and any project wide variables that are set when you trigger a build. Builds that you trigger on your project are also associated with the project.

Project 1 - N Build N - N Layer_Version 1 - N Recipe 1 - N Package

Project 1 - N ProjectLayers 1 - 1 Layer_Version

Project 1 - N Layer_Versions (ImportedLayer)

Project 1 - N CustomImageRecipe

== What running a Build affects ==
When you've configured your project in Toaster (using configuration data) and hit build on a Recipe two main things happen.

'''1.''' Update or create the ''Recipe'' and ''Layer_Version'' in Toaster with any new information that we gathered by building it

When a recipe is built we can update the configuration information in Toaster to be more accurate. e.g. if we have newer information than provided by the Layer index or if we're building a recipe in an imported layer for the first time we can update that ''Layer_Version'' to say what ''Recipes'' are available in the layer or other metadata items which we have discovered by building (Summary, Description, Version, Licence, etc) we also create and update the packages which are provided by the recipe. These are created as ''CustomImagePackage'' objects as their use is for customising the packages installed in an image recipe ''(CustomImageRecipe)''

'''2.''' Make a copy of the Recipe(s) built and the Layer_Version(s) in the project at build time

Copying creates a snapshot of that configuration data for the purposes of the build history. It also allows us to compartmentalise the build specific data e.g. what version the recipe was on, what the git checkout was at the time of build, the Machine set which may affect the recipes and packages etc

The build property for these copied Layer_Versions is set to the current Build object. When this is set the objects down the relational chain (Layer_Version 1 - N Recipe 1 - N Packages) are considered part of the "build data" and should never be changed or edited after this point.

== What running lsupdates affects ==
When lsupdates (layer source updates) is run all the LayerSources run their update function. Layer sources provide the configuration data for Toaster from pre-generated configuration data. This allows us to to bootstrap Toaster to a state where you can select Recipes, Machines and Layers for your project without having had the cost of downloading all of the Layers in openembedded and then parsing or building them. Toaster by default uses the [http://layers.openembedded.org Layer index] as a LayerSource to download this pre-generated configuration data.

=== Layer index layer source ===
The [http://layers.openembedded.org Layer index] [http://git.yoctoproject.org/cgit/cgit.cgi/layerindex-web/ source] has a ResT API which is called by the LayerIndex LayerSource which creates or updates (if they exist) Layers, Layer_Versions, Recipes and Machines.

Toaster database

2016-02-10T17:46:30Z

Michael Wood: /* Package */

[[Category:Toaster]]
[[File:Toaster_orm_graph.png|thumb|ORM diagram]]
Toaster Uses [https://docs.djangoproject.com/en/dev/topics/db/models/ Django] and it's [https://en.wikipedia.org/wiki/Object-relational_mapping ORM] for database interaction.

The database table/model definitions can be found in the [http://cgit.openembedded.org/bitbake/tree/lib/toaster/orm/models.py orm/models.py] file.
We also have additional models defined for the build controller, these are generally the 'behind the scenes' way in which we translate toaster information into the build controller process. This page is a '''high level''' look at the database design and how it is used.

== Categories of data in Toaster ==

=== Build data ===
Build data is Layers, Layer_Versions, Recipes, Machines, Packages, Dependencies and all other objects that are output by running a build of a recipe. '''Once it is created it is never edited or updated'''. Unless deleted by the user.

=== Configuration data ===
Configuration data is data which is provided by Toaster to configure an openembedded based project in Toaster, like Build data this is stored as Layers, Layer_Versions, Recipes and Machines etc. The configuration data can be produced by fetching it from the [http://layers.openembedded.org Layer Index] (the toaster start up script attempts to do this for you at first run) from importing information from the toasterconf.json file, importing Layer definitions into Projects via the Import layer page and also from running a build.

=== Project data ===

This is the container for Configuration data and Build data. Additionally it contains project specific data such as user imported Layers and CustomImageRecipe.

== Top level models ==
There are many intermediate tables in Toaster's database, see diagram. Below are the most top level models on which everything else depends.
These models map to the metadata [http://www.yoctoproject.org/docs/1.8/bitbake-user-manual/bitbake-user-manual.html#Concepts concepts] that make up openembedded/Bitbake.

=== Layer and Layer_Version ===
A Layer_Version is the model in which most other models relate to in Toaster. Every Layer_Version object belongs to a Layer object which contains the common metadata for each 'version' or revision of the Layer.

An example would be that we have a Layer called 'meta-openembedded' which has 3 Layer_Versions; one Layer_Version which is set to the release 'dizzy' one on 'fido' and one on 'master' (in most cases these correspond to git branch names)

'''Layer 1 - N Layer_Version 1 - N Recipe 1 - N Packages'''

=== Recipe ===

Recipes contain the information about a collection or particular piece of software. This can be the ''name'', ''version'' and ''description'' and any distribution ''package''s are created by the Recipe.

An example would be a recipe called myrecipe version 1.2 which produces 3 Packages myrecipe-dev, myrecipe-dbg and myrecipe. Recipes are the objects which get built by BitBake.

Recipe N - 1 Layer_Version

Recipe 1 - N Package

==== CustomImageRecipe ====

This is a special kind of Recipe which has three additional fields; base_recipe, last_updated and project. base_recipe is the Recipe which the Custom recipe has been based on and is the recipe which will be copied in when the Custom recipe is generated. The last_updated field is used internally to keep track of the last time the package list was last updated from a build.

=== Package ===

A Package is the representation of distribution packages produced by a Recipe when it has been built. Package data is almost always Build data since we don't know about packages until they have been produced. The exception is when the package data is used as configuration data to Customise a recipe by adding and removing known packages.

Package N - 1 Recipe

==== CustomImagePackage ====
This is a subclass of Package which contains 3 extra fields. ''recipe_includes'' are for keeping track of the packages which are included as part of the base recipe when it is built. e.g. the ''base_recipe'' has package busybox. ''recipe_excludes'' are packages which are normally included as defined by the base_recipe which have been explicitly asked to be excluded by the user. ''recipe_appends'' are the packages which are being appended to this recipe (think IMAGE_INSTALL +=)

These objects are configuration data and are generated by running builds. When a build runs we create or update CustomImagePackages meaning that each build helps to create a pool of packages which are available to the user to add or remove based on the available recipes when creating a CustomImageRecipe

CustomImagePackage N - N CustomImageRecipe

=== Build ===

The Build model contains the metadata for reporting information about a Build. It is the top level object for many of the Build related models which contain output from a Build. The Build model also includes when the Build started and stopped, and the status of the build. Any data in toaster that has a build object related to it is considered Build data.

Build 1 - 1 Target 1 - N Target_File

Build 1 - N Task

Build 1 - N BuildArtifact

Build 1 - N LogMessage 1 - 1 Task

Build 1 - N Variable

Build 1 - N Layer_Version

=== Target ===

A target is the instruction on what to build that is sent to Bitbake. When a build is triggered a Target object is created which records the one or more Recipe names to be built and the [http://www.yoctoproject.org/docs/latest/bitbake-user-manual/bitbake-user-manual.html#tasks Task] that you're asking BitBake to do (no task supplied executes the default which is build the recipe).

Target 1 - 1 Build

=== Project ===

The project model is the container for the configuration data of your current project, this includes the release, the Layer_Versions that you're using in your project and any project wide variables that are set when you trigger a build. Builds that you trigger on your project are also associated with the project.

Project 1 - N Build N - N Layer_Version 1 - N Recipe 1 - N Package

Project 1 - N ProjectLayers 1 - 1 Layer_Version

Project 1 - N Layer_Versions (ImportedLayer)

Project 1 - N CustomImageRecipe

== What running a Build affects ==
When you've configured your project in Toaster (using configuration data) and hit build on a Recipe two main things happen.

'''1.''' Update or create the Recipe and Layer_Version in Toaster with any new information that we gathered by building it

When a recipe is built we can update the configuration information in Toaster to be more accurate. e.g. if we have newer information than provided by the Layer index or if we're building a recipe in an imported layer for the first time we can update that Layer_Version to say what Recipes are in it.

'''2.''' Make a copy of the Recipe(s) built and the Layer_Version(s) in the project at build time

Copying creates a snapshot of that configuration data for the purposes of the build history. It also allows us to compartmentalise the build specific data e.g. what version the recipe was on, what the git checkout was at the time of build, the Machine set which may affect the recipes and packages etc

The build property for these copied Layer_Versions is set to the current Build object. When this is set the objects down the relational chain (Layer_Version 1 - N Recipe 1 - N Packages) are considered part of the "build data" and should never be changed or edited after this point.

== What running lsupdates affects ==
When lsupdates (layer source updates) is run all the LayerSources run their update function. Layer sources provide the configuration data for Toaster from pre-generated configuration data. This allows us to to bootstrap Toaster to a state where you can select Recipes, Machines and Layers for your project without having had the cost of downloading all of the Layers in openembedded and then parsing or building them. Toaster by default uses the [http://layers.openembedded.org Layer index] as a LayerSource to download this pre-generated configuration data.

=== Layer index layer source ===
The [http://layers.openembedded.org Layer index] [http://git.yoctoproject.org/cgit/cgit.cgi/layerindex-web/ source] has a ResT API which is called by the LayerIndex LayerSource which creates or updates (if they exist) Layers, Layer_Versions, Recipes and Machines.

Toaster database

2016-02-10T17:17:22Z

Michael Wood: /* CustomImageRecipe */

[[Category:Toaster]]
[[File:Toaster_orm_graph.png|thumb|ORM diagram]]
Toaster Uses [https://docs.djangoproject.com/en/dev/topics/db/models/ Django] and it's [https://en.wikipedia.org/wiki/Object-relational_mapping ORM] for database interaction.

The database table/model definitions can be found in the [http://cgit.openembedded.org/bitbake/tree/lib/toaster/orm/models.py orm/models.py] file.
We also have additional models defined for the build controller, these are generally the 'behind the scenes' way in which we translate toaster information into the build controller process. This page is a '''high level''' look at the database design and how it is used.

== Categories of data in Toaster ==

=== Build data ===
Build data is Layers, Layer_Versions, Recipes, Machines, Packages, Dependencies and all other objects that are output by running a build of a recipe. '''Once it is created it is never edited or updated'''. Unless deleted by the user.

=== Configuration data ===
Configuration data is data which is provided by Toaster to configure an openembedded based project in Toaster, like Build data this is stored as Layers, Layer_Versions, Recipes and Machines etc. The configuration data can be produced by fetching it from the [http://layers.openembedded.org Layer Index] (the toaster start up script attempts to do this for you at first run) from importing information from the toasterconf.json file, importing Layer definitions into Projects via the Import layer page and also from running a build.

=== Project data ===

This is the container for Configuration data and Build data. Additionally it contains project specific data such as user imported Layers and CustomImageRecipe.

== Top level models ==
There are many intermediate tables in Toaster's database, see diagram. Below are the most top level models on which everything else depends.
These models map to the metadata [http://www.yoctoproject.org/docs/1.8/bitbake-user-manual/bitbake-user-manual.html#Concepts concepts] that make up openembedded/Bitbake.

=== Layer and Layer_Version ===
A Layer_Version is the model in which most other models relate to in Toaster. Every Layer_Version object belongs to a Layer object which contains the common metadata for each 'version' or revision of the Layer.

An example would be that we have a Layer called 'meta-openembedded' which has 3 Layer_Versions; one Layer_Version which is set to the release 'dizzy' one on 'fido' and one on 'master' (in most cases these correspond to git branch names)

'''Layer 1 - N Layer_Version 1 - N Recipe 1 - N Packages'''

=== Recipe ===

Recipes contain the information about a collection or particular piece of software. This can be the ''name'', ''version'' and ''description'' and any distribution ''package''s are created by the Recipe.

An example would be a recipe called myrecipe version 1.2 which produces 3 Packages myrecipe-dev, myrecipe-dbg and myrecipe. Recipes are the objects which get built by BitBake.

Recipe N - 1 Layer_Version

Recipe 1 - N Package

==== CustomImageRecipe ====

This is a special kind of Recipe which has three additional fields; base_recipe, last_updated and project. base_recipe is the Recipe which the Custom recipe has been based on and is the recipe which will be copied in when the Custom recipe is generated. The last_updated field is used internally to keep track of the last time the package list was last updated from a build.

=== Package ===

A Package is the representation of distribution packages produced by a Recipe when it has been built. Package data is almost always Build data since we don't know about packages until they have been produced. The exception is when the package data is used as configuration data to Customise a recipe by adding and removing known packages.

Package N - 1 Recipe

=== Build ===

The Build model contains the metadata for reporting information about a Build. It is the top level object for many of the Build related models which contain output from a Build. The Build model also includes when the Build started and stopped, and the status of the build. Any data in toaster that has a build object related to it is considered Build data.

Build 1 - 1 Target 1 - N Target_File

Build 1 - N Task

Build 1 - N BuildArtifact

Build 1 - N LogMessage 1 - 1 Task

Build 1 - N Variable

Build 1 - N Layer_Version

=== Target ===

A target is the instruction on what to build that is sent to Bitbake. When a build is triggered a Target object is created which records the one or more Recipe names to be built and the [http://www.yoctoproject.org/docs/latest/bitbake-user-manual/bitbake-user-manual.html#tasks Task] that you're asking BitBake to do (no task supplied executes the default which is build the recipe).

Target 1 - 1 Build

=== Project ===

The project model is the container for the configuration data of your current project, this includes the release, the Layer_Versions that you're using in your project and any project wide variables that are set when you trigger a build. Builds that you trigger on your project are also associated with the project.

Project 1 - N Build N - N Layer_Version 1 - N Recipe 1 - N Package

Project 1 - N ProjectLayers 1 - 1 Layer_Version

Project 1 - N Layer_Versions (ImportedLayer)

Project 1 - N CustomImageRecipe

== What running a Build affects ==
When you've configured your project in Toaster (using configuration data) and hit build on a Recipe two main things happen.

'''1.''' Update or create the Recipe and Layer_Version in Toaster with any new information that we gathered by building it

When a recipe is built we can update the configuration information in Toaster to be more accurate. e.g. if we have newer information than provided by the Layer index or if we're building a recipe in an imported layer for the first time we can update that Layer_Version to say what Recipes are in it.

'''2.''' Make a copy of the Recipe(s) built and the Layer_Version(s) in the project at build time

Copying creates a snapshot of that configuration data for the purposes of the build history. It also allows us to compartmentalise the build specific data e.g. what version the recipe was on, what the git checkout was at the time of build, the Machine set which may affect the recipes and packages etc

The build property for these copied Layer_Versions is set to the current Build object. When this is set the objects down the relational chain (Layer_Version 1 - N Recipe 1 - N Packages) are considered part of the "build data" and should never be changed or edited after this point.

== What running lsupdates affects ==
When lsupdates (layer source updates) is run all the LayerSources run their update function. Layer sources provide the configuration data for Toaster from pre-generated configuration data. This allows us to to bootstrap Toaster to a state where you can select Recipes, Machines and Layers for your project without having had the cost of downloading all of the Layers in openembedded and then parsing or building them. Toaster by default uses the [http://layers.openembedded.org Layer index] as a LayerSource to download this pre-generated configuration data.

=== Layer index layer source ===
The [http://layers.openembedded.org Layer index] [http://git.yoctoproject.org/cgit/cgit.cgi/layerindex-web/ source] has a ResT API which is called by the LayerIndex LayerSource which creates or updates (if they exist) Layers, Layer_Versions, Recipes and Machines.

Contribute to Toaster

2016-02-08T15:15:49Z

Michael Wood:

Contribute to Toaster

2016-02-08T15:14:58Z

Michael Wood: /* Sending patches to Toaster Project */

Setting up a production instance of Toaster

2015-12-10T16:40:57Z

Michael Wood:

[[Category:Toaster]]

'''This page is the development version of the documentation to provide the latest information, if you're using a release please refer to [https://www.yoctoproject.org/documentation/archived the published manual]'''

A production instance of Toaster is one in which you wish to share the Toaster instance with remote and multiple users. It is also the setup which can cope with heavier loads on the web service. These instructions setup toaster in Build mode where builds and projects are run, viewed and defined by the Toaster web interface.

== Requirements ==
* [http://www.yoctoproject.org/docs/2.0/yocto-project-qs/yocto-project-qs.html#packages Build requirements]
* Apache webserver
* mod-wsgi for Apache webserver
* Mysql database server

Ubuntu 14.04.3:
<code>
$ sudo apt-get install apache2 libapache2-mod-wsgi mysql-server python-virtualenv libmysqlclient-dev python-dev
</code>

Fedora 22/RH:
<code>
$ sudo dnf install httpd mod_wsgi python-virtualenv gcc mysql-devel
</code>

== Installation ==

'''1.''' Checkout a copy of Poky into the web server directory. We're going to be using /var/www/toaster.
<code>
$ mkdir -p /var/www/toaster
$ cd /var/www/toaster/
$ git clone git://git.yoctoproject.org/poky
$ cd poky
$ git checkout jethro # change for any release name required
</code>

'''2.''' Initialise a virtualenv and install Toaster dependencies. (Use virtualenv to keep the python packages isolated from your system provided packages - not required but recommended, alternative use your OS's package manager to install the packages)

<code>
$ cd /var/www/toaster/
$ virtualenv venv
$ source ./venv/bin/activate
$ pip install -r ./poky/bitbake/toaster-requirements.txt
$ pip install mysqlclient
</code>

'''3.''' Configure toaster edit /var/www/toaster/poky/bitbake/lib/toaster/toastermain/settings.py

Edit the DATABASE settings:
<code>
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.mysql',
'NAME': 'toaster_data',
'USER': 'toaster',
'PASSWORD': 'yourpasswordhere',
'HOST': 'localhost',
'PORT': '3306',
}
</code>

Edit the [https://docs.djangoproject.com/en/1.8/howto/deployment/checklist/#secret-key SECRET_KEY]:
<code>
SECRET_KEY = 'YOUR SECRET RANDOM KEY HERE'
</code>

Edit the STATIC_ROOT:
<code>
STATIC_ROOT = '/var/www/toaster/static_files/'
</code>

Edit BUILD_MODE:
<code>
BUILD_MODE = True
</code>

'''4.'''' Now add the database and user to your mysql server that we just defined
<code>
$ mysql -u root -p
mysql> CREATE DATABASE toaster_data;
mysql> CREATE USER 'toaster'@'localhost' identified by 'yourpasswordhere';
mysql> GRANT all on toaster_data.* to 'toaster'@'localhost';
mysql> quit
</code>
n.b. You may want to decide on fewer [https://dev.mysql.com/doc/refman/5.1/en/grant.html privileges] to the toaster user.

'''5.''' Get toaster to create the database schema, default data, update the TOASTER_DIR which is the build work dir and collect up the statically served files

<code>
$ cd /var/www/toaster/poky/
$ ./bitbake/lib/toaster/manage.py migrate
$ ./bitbake/lib/toaster/manage.py loadconf ./meta-yocto/conf/toasterconf.json
$ TOASTER_DIR=/var/www/toaster/poky/ ./bitbake/lib/toaster/manage.py checksettings
$ ./bitbake/lib/toaster/manage.py lsupdates
$ ./bitbake/lib/toaster/manage.py collectstatic
</code>

'''6.''' Add a config file for Toaster to your Apache web server's configurations available directory.

Ubuntu/Debian put it here: /etc/apache2/conf-available/toaster.conf
Fedora/RH usually here: /etc/httpd/conf.d/toaster.conf
<code>

Alias /static /var/www/toaster/static_files
<Directory /var/www/toaster/static_files>
Order allow,deny
Allow from all
Require all granted
</Directory>
WSGIDaemonProcess toaster_wsgi python-path=/var/www/toaster/poky/bitbake/lib/toaster:/var/www/toaster/venv/lib/python2.7/site-packages
WSGIScriptAlias / "/var/www/toaster/poky/bitbake/lib/toaster/toastermain/wsgi.py"
<Location />
WSGIProcessGroup toaster_wsgi
</Location>

</code>

In Ubuntu/Debain you will need to enable the config and module in Apache webserver
<code>
$ sudo a2enmod wsgi
$ sudo a2enconf toaster
</code>

Restart Apache web server to make sure all new configuration is loaded

Ubuntu/Debian:
<code>
$ sudo service apache2 restart
</code>

Fedora/RH:
<code>
$ sudo service httpd restart
</code>

'''7.''' Install the build runner service

This service needs to be running in order to dispatch builds the command that needs to be run is:

<code>
/var/www/toaster/poky/bitbake/lib/toaster/manage.py runbuilds
</code>

Sample script:
<code>
#!/bin/sh
# toaster run builds dispatcher
cd /var/www/toaster/
source ./venv/bin/activate
while true; do ./poky/bitbake/lib/toaster/manage.py runbuilds; sleep 3; done
</code>

N.b. You may wish to add a service entry to your OS's init system so that it starts up on start up as well as adding a dedicated user.

Now open up a browser and you can start using Toaster!

Setting up a production instance of Toaster

2015-12-08T14:45:19Z

Michael Wood:

[[Category:Toaster]]

'''This page is the development version of the documentation to provide the latest information, if you're using a release please refer to [https://www.yoctoproject.org/documentation/archived the published manual]'''

A production instance of Toaster is one in which you wish to share the Toaster instance with remote and multiple users. It is also the setup which can cope with heavier loads on the web service. These instructions setup toaster in Build mode where builds and projects are run, viewed and defined by the Toaster web interface.

== Requirements ==
* [http://www.yoctoproject.org/docs/2.0/yocto-project-qs/yocto-project-qs.html#packages Build requirements]
* Apache webserver
* mod-wsgi for Apache webserver
* Mysql database server

Ubuntu 14.04.3:
<code>
$ sudo apt-get install apache2 libapache2-mod-wsgi mysql-server python-virtualenv libmysqlclient-dev python-dev
</code>

Fedora 22/RH:
<code>
$ sudo dnf install httpd mod_wsgi python-virtualenv gcc mysql-devel
</code>

== Installation ==

'''1.''' Checkout a copy of Poky into the web server directory. We're going to be using /var/www/toaster.
<code>
$ mkdir -p /var/www/toaster
$ cd /var/www/toaster/
$ git clone git://git.yoctoproject.org/poky
$ cd poky
$ git checkout jethro # change for any release name required
</code>

'''2.''' Initialise a virtualenv and install Toaster dependencies. (Use virtualenv to keep the python packages isolated from your system provided packages - not required but recommended, alternative use your OS's package manager to install the packages)

<code>
$ cd /var/www/toaster/
$ virtualenv venv
$ source ./venv/bin/activate
$ pip install -r ./poky/bitbake/toaster-requirements.txt
$ pip install mysqlclient
</code>

'''3.''' Configure toaster edit /var/www/toaster/poky/bitbake/lib/toaster/toastermain/settings.py

Edit the DATABASE settings:
<code>
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.mysql',
'NAME': 'toaster_data',
'USER': 'toaster',
'PASSWORD': 'yourpasswordhere',
'HOST': 'localhost',
'PORT': '3306',
}
</code>

Edit the [https://docs.djangoproject.com/en/1.8/howto/deployment/checklist/#secret-key SECRET_KEY]:
<code>
SECRET_KEY = 'YOUR SECRET RANDOM KEY HERE'
</code>

Edit the STATIC_ROOT:
<code>
STATIC_ROOT = '/var/www/toaster/static_files/'
</code>

Edit BUILD_MODE:
<code>
BUILD_MODE = True
</code>

'''4.'''' Now add the database and user to your mysql server that we just defined
<code>
$ mysql -u root -p
mysql> CREATE DATABASE toaster_data;
mysql> CREATE USER 'toaster'@'localhost' identified by 'yourpasswordhere';
mysql> GRANT all on toaster_data.* to 'toaster'@'localhost';
mysql> quit
</code>
n.b. You may want to decide on fewer [https://dev.mysql.com/doc/refman/5.1/en/grant.html privileges] to the toaster user.

'''5.''' Get toaster to create the database schema, default data and collect up the statically service files

<code>
$ cd /var/www/toaster/poky/
$ ./bitbake/lib/toaster/manage.py syncdb --migrate
$ ./bitbake/lib/toaster/manage.py loadconf ./meta-yocto/conf/toasterconf.json
$ ./bitbake/lib/toaster/manage.py lsupdates
$ ./bitbake/lib/toaster/manage.py collectstatic
</code>

'''6.''' Add a config file for Toaster to your Apache web server's configurations available directory.

Ubuntu/Debian put it here: /etc/apache2/conf-available/toaster.conf
Fedora/RH usually here: /etc/httpd/conf.d/toaster.conf
<code>

Alias /static /var/www/toaster/static_files
<Directory /var/www/toaster/static_files>
Order allow,deny
Allow from all
Require all granted
</Directory>
WSGIDaemonProcess toaster_wsgi python-path=/var/www/toaster/poky/bitbake/lib/toaster:/var/www/toaster/venv/lib/python2.7/site-packages
WSGIScriptAlias / "/var/www/toaster/poky/bitbake/lib/toaster/toastermain/wsgi.py"
<Location />
WSGIProcessGroup toaster_wsgi
</Location>

</code>

In Ubuntu/Debain you will need to enable the config and module in Apache webserver
<code>
$ sudo a2enmod wsgi
$ sudo a2enconf toaster
</code>

Restart Apache web server to make sure all new configuration is loaded

Ubuntu/Debian:
<code>
$ sudo service apache2 restart
</code>

Fedora/RH:
<code>
$ sudo service httpd restart
</code>

'''7.''' Install the build runner service

This service needs to be running in order to dispatch builds the command that needs to be run is:

<code>
/var/www/toaster/poky/bitbake/lib/toaster/manage.py runbuilds
</code>

Sample script:
<code>
#!/bin/sh
# toaster run builds dispatcher
cd /var/www/toaster/
source ./venv/bin/activate
while true; do ./poky/bitbake/lib/toaster/manage.py runbuilds; sleep 3; done
</code>

N.b. You may wish to add a service entry to your OS's init system so that it starts up on start up as well as adding a dedicated user.

Now open up a browser and you can start using Toaster!