Setting up a production instance of Toaster: Difference between revisions

From Yocto Project
Jump to navigationJump to search
No edit summary
 
(26 intermediate revisions by 3 users not shown)
Line 1: Line 1:
[[Category:Toaster]]
[[Category:Toaster]]
This page explains how to set up Toaster separately from the hardware running your Yocto Project builds. Do it this way if you want to share a single instance of Toaster across multiple users and build servers.
----
{| style="color:black; background-color:#ffffcc" width="100%" cellpadding="10" class="wikitable"
|This page is the development version of the documentation to provide the latest information, if you're using a release please refer to [https://docs.yoctoproject.org/toaster-manual/index.html the published manual]'''
|}
----
A production instance of Toaster is one in which you wish to share the Toaster instance with remote and multiple users. It is also the setup which can cope with heavier loads on the web service. These instructions setup toaster in Build mode where builds and projects are run, viewed and defined by the Toaster web interface.


== Setting up a Toaster Instance on a Remote Host ==
== Requirements ==
* [http://www.yoctoproject.org/docs/2.0/yocto-project-qs/yocto-project-qs.html#packages Build requirements]
* Apache webserver
* mod-wsgi for Apache webserver
* Mysql database server


Under normal circumstances, starting Toaster causes three things happen:
Ubuntu 14.04.3:
<code>
    $ sudo apt-get install apache2 libapache2-mod-wsgi mysql-server python-virtualenv libmysqlclient-dev python-dev python-mysqldb
</code>


* A BitBake server starts
Fedora 22/RH:
<code>
    $ sudo dnf install httpd mod_wsgi python-virtualenv gcc mysql-devel
</code>


* The Toaster UI starts, which connects to the BitBake server on one side and to the SQL database on the other side
== Installation ==


* The web server starts, which reads the database and displays the web user interface
'''1.''' Checkout a copy of Poky into the web server directory. We're going to be using /var/www/toaster.
<code>
  $ mkdir -p /var/www/toaster
  $ cd /var/www/toaster/
  $ git clone git://git.yoctoproject.org/poky
  $ cd poky
  $ git checkout jethro # change for any release name required
</code>


Situations exist, however, where you might want to have multiple instances of Toaster running on various remote machines. You can create this situation by basically modifying how Toaster starts and where the common SQL database resides. You are able to do this because it is not required that Toaster starts the above set of components in order to run. Minimally, an instance of Toaster requires just one of the components to run. Consequently, you are free to manually start as many or few of the components as you need rather than using the Toaster script to cause all three things to happen.


The concepts for setting up multiple instances of Toaster revolve around maintaining a common SQL database and Web server that show data from that common database and then setting up separate instances of BitBake servers and Toaster user interfaces for each separate BitBake build directory. Note that the common SQL database and the Web server show data from all the various BitBake builds. Setting the SQL database outside of any BitBake build directories maintains a
'''2.''' Initialise a virtualenv and install Toaster dependencies. (Use virtualenv to keep the python packages isolated from your system provided packages - not required but recommended, alternative use your OS's package manager to install the packages)
separation layer between the various builds.


The database is persistent because the logging database is set up external to the database server (e.g. MySQL). It is not even necessary to run the BitBake servers, the SQL server, and the Web server on the same machine. Each component can be run on its own machine.
<code>
  $ cd /var/www/toaster/
  $ virtualenv venv
  $ source ./venv/bin/activate
  $ pip install -r ./poky/bitbake/toaster-requirements.txt
  $ pip install mysqlclient
</code>


== Steps to get set up ==


=== 1. Set up the SQL Logging Server and the Web Server ===
'''3.''' Configure toaster edit /var/www/toaster/poky/bitbake/lib/toaster/toastermain/settings.py


You can use any SQL server out of the box (e.g. <code>apt-get install mysgl-server</code> works for an Ubuntu system). If you are concerned about performance, you might want to hand-tune the server. You must set up proper username and password access for the server. You need administration rights for the mysql root account. Realize that this is not the same thing as root access on the machine.
Edit the DATABASE settings:
<code>
DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.mysql',
        'NAME': 'toaster_data',                   
        'USER': 'toaster',
        'PASSWORD': 'yourpasswordhere',
        'HOST': 'localhost',               
        'PORT': '3306',                     
}
</code>


Clone a separate, local Git repository of the Toaster master branch to use for running the Web server. You do not perform builds on this tree. You need to create this local repository away from any build areas.
Edit the [https://docs.djangoproject.com/en/1.8/howto/deployment/checklist/#secret-key SECRET_KEY]:
<code>
SECRET_KEY = 'YOUR SECRET RANDOM KEY HERE'
</code>


In the separately cloned tree for the Web server, edit the <code>bitbake/lib/toaster/toastermain/settings.py</code> file so that the <code>DATABASES</code> value points to the previously created database server. Use the username and password you established earlier.
Edit the STATIC_ROOT:
<code>
STATIC_ROOT = '/var/www/toaster/static_files/'
</code>


Run the database sync scripts to create the needed tables as follows:


    $ python bitbake/lib/toaster/manage.py syncdb
'''4.'''' Now add the database and user to your mysql server that we just defined
    $ python bitbake/lib/toaster/manage.py migrate orm
<code>
$ mysql -u root -p
mysql> CREATE DATABASE toaster_data;
mysql> CREATE USER 'toaster'@'localhost' identified by 'yourpasswordhere';
mysql> GRANT all on toaster_data.* to 'toaster'@'localhost';
mysql> quit
</code>
n.b. You may want to decide on fewer [https://dev.mysql.com/doc/refman/5.1/en/grant.html privileges] to the toaster user.  


Note: It is important, for toaster running in 1.6 mode, to not sync bldcontrol (only used in 1.7 mode).


'''5.''' Get toaster to create the database schema, default data, update the TOASTER_DIR which is the build work dir and collect up the statically served files


Start the Web server using the following command:
<code>
$ cd  /var/www/toaster/poky/
$ ./bitbake/lib/toaster/manage.py migrate
$ ./bitbake/lib/toaster/manage.py loadconf ./meta-yocto/conf/toasterconf.json
$ TOASTER_DIR=/var/www/toaster/poky/ ./bitbake/lib/toaster/manage.py checksettings
$ ./bitbake/lib/toaster/manage.py lsupdates
$ ./bitbake/lib/toaster/manage.py collectstatic
</code>


    $ python bitbake/lib/toaster/manage.py runserver


... or in the background:
'''6.''' Add a config file for Toaster to your Apache web server's configurations available directory.


    $ nohup python bitbake/lib/toaster/manage.py runserver 2>toaster_web.log >toaster_web.log &
Ubuntu/Debian put it here: /etc/apache2/conf-available/toaster.conf
Fedora/RH usually here: /etc/httpd/conf.d/toaster.conf
<code>


=== 2. Enable build logging to the common SQL server for each build directory you are using ===
Alias /static /var/www/toaster/static_files
<Directory /var/www/toaster/static_files>
Order allow,deny
Allow from all
Require all granted
</Directory>
WSGIDaemonProcess toaster_wsgi python-path=/var/www/toaster/poky/bitbake/lib/toaster:/var/www/toaster/venv/lib/python2.7/site-packages
WSGIScriptAlias / "/var/www/toaster/poky/bitbake/lib/toaster/toastermain/wsgi.py"
<Location />
    WSGIProcessGroup toaster_wsgi
</Location>


Edit <code>_build local_ bitbake/lib/toaster/toastermain/settings.py</code> to alter the <code>DATABASES</code> value to point to the common SQL logging server.
</code>


Create the required conf/toaster.conf file as per Bitbake extra options on "https://wiki.yoctoproject.org/wiki/Setting_up_a_local_instance_of_Toaster".
In Ubuntu/Debain you will need to enable the config and module in Apache webserver
<code>
  $ sudo a2enmod wsgi
  $ sudo a2enconf toaster
</code>


Start the BitBake server using the following command:
Restart Apache web server to make sure all new configuration is loaded


    $ bitbake --postread conf/toaster.conf --server-only -t xmlrpc -B localhost:0 && export BBSERVER=localhost:-1
Ubuntu/Debian:
<code>
  $ sudo service apache2 restart
</code>


Start the logging user interface using the following command:
Fedora/RH:
<code>
  $ sudo service httpd restart
</code>


    $ nohup bitbake --observe-only -u toasterui >toaster_ui.log &


<strong>NOTE:</strong> No hard-coded ports are used as there is enough code to run autodiscovery for ports to prevent collisions.
'''7.''' Install the build runner service


At this point, you are ready to run your builds using commands such as:
This service needs to be running in order to dispatch builds the command that needs to be run is:


    $ bitbake core-image-minimal
<code>
$ source oe-init-build-env
$ source toaster start noweb
</code>


When you are finished, you need to kill the BitBake server for that particular build area:


    $ bitbake -m
Example upstart service /etc/init/toaster-buildservice.conf :
<pre>
author "Michael W"
description "start and stop toaster build service"
version "1.0"


=== 3. Verify that it all works ===
start on started networking
stop on runlevel [!2345]


You should examine the logs and be sure that the logging worked, that data is persistent, and that data from multiple builds from different areas was supported.
respawn
 
script
  exec su toasterbuilder -c "cd /var/www/toaster/poky && source oe-init-build-env && source toaster start noweb"
end script
</pre>
 
 
Example systemd service usually in /lib/systemd/system/toaster.service:
<pre>
[Unit]
Description=Toaster runbuilds
[Service]
Type=forking
User=toaster
ExecStart=/var/www/toaster/toaster-service.sh start
ExecStop=/var/www/toaster/toaster-service.sh stop
[Install]
WantedBy=multi-user.target
</pre>
script /var/www/toaster/toaster-service.sh:
<pre>
 
#!/bin/bash
cd /var/www/toaster/poky/
source oe-init-build-env
if [ "$1" == 'start' ]; then
  source ../bitbake/bin/toaster start noweb
fi
if [ "$1" == 'stop' ]; then
  source ../bitbake/bin/toaster stop
fi
</pre>
 
 
N.b. You may wish to add a service entry to your OS's init system so that it starts up on start up as well as adding a dedicated user.
 
 
Now open up a browser and you can start using Toaster!

Latest revision as of 00:42, 18 February 2024


This page is the development version of the documentation to provide the latest information, if you're using a release please refer to the published manual

A production instance of Toaster is one in which you wish to share the Toaster instance with remote and multiple users. It is also the setup which can cope with heavier loads on the web service. These instructions setup toaster in Build mode where builds and projects are run, viewed and defined by the Toaster web interface.

Requirements

Ubuntu 14.04.3:

   $ sudo apt-get install apache2 libapache2-mod-wsgi mysql-server python-virtualenv libmysqlclient-dev python-dev python-mysqldb

Fedora 22/RH:

   $ sudo dnf install httpd mod_wsgi python-virtualenv gcc mysql-devel

Installation

1. Checkout a copy of Poky into the web server directory. We're going to be using /var/www/toaster.

 $ mkdir -p /var/www/toaster
 $ cd /var/www/toaster/
 $ git clone git://git.yoctoproject.org/poky
 $ cd poky
 $ git checkout jethro # change for any release name required


2. Initialise a virtualenv and install Toaster dependencies. (Use virtualenv to keep the python packages isolated from your system provided packages - not required but recommended, alternative use your OS's package manager to install the packages)

  $ cd /var/www/toaster/
  $ virtualenv venv
  $ source ./venv/bin/activate
  $ pip install -r ./poky/bitbake/toaster-requirements.txt
  $ pip install mysqlclient


3. Configure toaster edit /var/www/toaster/poky/bitbake/lib/toaster/toastermain/settings.py

Edit the DATABASE settings:

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.mysql', 
        'NAME': 'toaster_data',                     
        'USER': 'toaster',
        'PASSWORD': 'yourpasswordhere',
        'HOST': 'localhost',                 
        'PORT': '3306',                      
}

Edit the SECRET_KEY:

SECRET_KEY = 'YOUR SECRET RANDOM KEY HERE'

Edit the STATIC_ROOT:

STATIC_ROOT = '/var/www/toaster/static_files/'


4.' Now add the database and user to your mysql server that we just defined

$ mysql -u root -p
mysql> CREATE DATABASE toaster_data;
mysql> CREATE USER 'toaster'@'localhost' identified by 'yourpasswordhere';
mysql> GRANT all on toaster_data.* to 'toaster'@'localhost';
mysql> quit

n.b. You may want to decide on fewer privileges to the toaster user.


5. Get toaster to create the database schema, default data, update the TOASTER_DIR which is the build work dir and collect up the statically served files

$ cd  /var/www/toaster/poky/
$ ./bitbake/lib/toaster/manage.py migrate
$ ./bitbake/lib/toaster/manage.py loadconf ./meta-yocto/conf/toasterconf.json
$ TOASTER_DIR=/var/www/toaster/poky/ ./bitbake/lib/toaster/manage.py checksettings
$ ./bitbake/lib/toaster/manage.py lsupdates
$ ./bitbake/lib/toaster/manage.py collectstatic


6. Add a config file for Toaster to your Apache web server's configurations available directory.

Ubuntu/Debian put it here: /etc/apache2/conf-available/toaster.conf Fedora/RH usually here: /etc/httpd/conf.d/toaster.conf

Alias /static /var/www/toaster/static_files
<Directory /var/www/toaster/static_files>
	Order allow,deny
	Allow from all
	Require all granted
</Directory>
WSGIDaemonProcess toaster_wsgi python-path=/var/www/toaster/poky/bitbake/lib/toaster:/var/www/toaster/venv/lib/python2.7/site-packages
WSGIScriptAlias / "/var/www/toaster/poky/bitbake/lib/toaster/toastermain/wsgi.py"
<Location />
    WSGIProcessGroup toaster_wsgi
</Location>

In Ubuntu/Debain you will need to enable the config and module in Apache webserver

  $ sudo a2enmod wsgi
  $ sudo a2enconf toaster

Restart Apache web server to make sure all new configuration is loaded

Ubuntu/Debian:

  $ sudo service apache2 restart

Fedora/RH:

  $ sudo service httpd restart


7. Install the build runner service

This service needs to be running in order to dispatch builds the command that needs to be run is:

$ source oe-init-build-env
$ source toaster start noweb


Example upstart service /etc/init/toaster-buildservice.conf :

 author "Michael W"
 description "start and stop toaster build service"
 version "1.0"

 start on started networking
 stop on runlevel [!2345]

 respawn

 script
   exec su toasterbuilder -c "cd /var/www/toaster/poky && source oe-init-build-env && source toaster start noweb"
 end script


Example systemd service usually in /lib/systemd/system/toaster.service:

 
[Unit]
Description=Toaster runbuilds
 
[Service]
Type=forking
User=toaster
ExecStart=/var/www/toaster/toaster-service.sh start
ExecStop=/var/www/toaster/toaster-service.sh stop
 
[Install]
WantedBy=multi-user.target

script /var/www/toaster/toaster-service.sh:


#!/bin/bash
 
cd /var/www/toaster/poky/
 
source oe-init-build-env
if [ "$1" == 'start' ]; then
  source ../bitbake/bin/toaster start noweb
fi
 
if [ "$1" == 'stop' ]; then
  source ../bitbake/bin/toaster stop
fi


N.b. You may wish to add a service entry to your OS's init system so that it starts up on start up as well as adding a dedicated user.


Now open up a browser and you can start using Toaster!