Contribute to Toaster

This page summarises the Toaster development process. We hope this will help you start contributing to the project. We were previously using a process that involved the toaster-next branch on poky-contrib. This workflow was especially useful in an environment where Toaster was undergoing a lot of change and needed the ability to layer patch sets upon each other before they had been taken up by bitbake. Since that is not currently the case, toaster-next just adds complication; so it has been removed from this workflow. See Contribute to Toaster (toaster-next version) for the old toaster-next based workflow.

What can I do?

The Yocto Project Bugzilla instance lists all the things that need to be done:

• If the issue says GUI design available in the Whiteboard field, there is a design specification document attached to the issue that you should follow. Send questions / comments about it to the Toaster mailing list
• If the issue says GUI design pending in the Whiteboard field, there is some design work still to be done. Feel free to take the issue and send an email to the Toaster mailing list to find out why the design work is not done yet

Set up the local repository

For development of Toaster we recommend setting up a local install of Toaster. Installation instructions are available in the main Toaster documentation

Submitting patches

Publishing your patches to Toaster is a two step process.

1. Sending patches to the Toaster mailing list for review
2. Submitting the patches that you reviewed to the upstream repository

By submitting your patches first to the Toaster mailing list, you can be sure the patches are reviewed by the people in the community who are familiar with the Toaster source code, and who have experience developing web applications.

That also means that, by the time the patches are submitted to the upstream mailing lists, they are in pretty good shape. That helps the project maintainers, and hopefully also helps you.

Toaster code lives in Bitbake repository at [1]. All contributions must be upstreamed to the Bitbake repository in order to make it to the "master" branch of the poky/ repository.

Workflow

To contribute to toaster you will also need authorization to write to the upstream yocto project repository. Contact a member of the toaster team for details.

1) Download master branch of the yocto project

  git clone git://git.yoctoproject.org/poky && cd poky 

2) Add poky-contrib to the local repository you set up above

  [old] git remote add poky-contrib ssh://git@git.yoctoproject.org/poky-contrib 
  git remote add poky-contrib https://git.yoctoproject.org/git/poky-contrib 

3) Fetch the poky-contrib branches

  git fetch --all 

4) Start your feature branch off of master, name style of branch is convention, but suggested.

  git checkout -b username/toaster/FeatureOrBug origin/master 

5) Do Work

6) Test the changes. Run the Django unit tests. People put effort into these so we should make sure we use them. This assumes you have phantomjs installed. This can usually be done from the distribution apt-get install phantomjs, for example. If you want to test against Chrome or Firefox, see the README in bitbake/lib/toaster/tests/browser.

  pip3 install selenium 
 TOASTER_TESTS_BROWSER=phantomjs bitbake/lib/toaster/manage.py test orm toastergui tests.browser 

Note: If you would like to run the tests in a container so they are repeatable and do not continually break due to host upgrades see Running Toaster Tests with Containers

7) Rebase on master. It has probably changed while you were working (unless you are really really fast!)

  git rebase origin/master 

8) Push your feature branch to poky-contrib

 git push -u poky-contrib username/toaster/FeatureOrBug:username/toaster/FeatureOrBug

9) Send to the toaster-mailing list using one of the methods outlined below.

10) NOTE: when the patch has been accepted upstream, you can clean up your poy-contrib branch with:

  git push -u poky-contrib :username/toaster/FeatureOrBug 

Sending patches to Toaster Project

NOTE: The format of the commit message should be like this

    toaster: <module> <short one line summary>

    long(er) description

    [YOCTO #0000]

    Signed-off-by: First Last <name@domain.com>

Where YOCTO #0000 is the related bug number if there is one. Signed off by with your git commit -s credentials.

We accept patches on the toaster mailing list ( toaster@yoctoproject.org ) by "git send-email".

e.g.

   $ git send-email HEAD^ 

Alternatively, you can use the utilities in the script directory to prepare your patch

1) Use the create-pull-request script (from poky) to create a pull request while on your feature branch

  ./scripts/create-pull-request -s "toaster: Fixes and clean ups" -u poky-contrib -r  origin/master

2) Review their content, especially the summary mail:

  edit ./pull-<pid>/0000-cover-letter.patch

3)When you are satisfied, you can send them with:

  ./scripts/send-pull-request -a -p ./pull-<pid> -t toaster@yoctoproject.org

A comprehensive document about commit messages is available on the openembedded wiki

More help learning git is available on github and the official documentation

Sending branches to Toaster Project

If you wish to submit whole branches please use the poky-contrib repository see Poky Contributions#Poky_Contrib_Branch for setup guide.

Once you have pushed a branch please then send an email to the toaster mailing list with the subject in the following format:

[review-request] my_branch_name

In the body of the email it's useful to describe your branch's functionality, which commits and a link to the git web.

If you need any assistance please post on the mailing list.

Submitting patch sets for integration into Bitbake

Toaster patches are normally submitted upstream to the BitBake repository by the reviewer (not the author). This tells the upstream maintainers that the patches have been reviewed by the people who are familiar with the Toaster source code, and makes their busy lives a bit easier.

Since development happens on the poky-contrib repository, but the patches need to be merged to the Bitbake repository, the following process should be executed.

1) Bring master up to date

  git fetch origin master
  git checkout master 
  git pull [master] 

2) Checkout the target branch you wish to upstream

(a) If you are working with a single branch do this:

  git checkout username/toaster/FeatureOrBug

(b) If you are including several branches into this one submission, gather the commit IDs for each branch and do this:

  git cherry-pick commit-id-a 
  git cherry-pick commit-id-b 
  ... 

3) Create a new branch for submission

   git checkout -b yourname/submit/username/toaster/FeatureOrBug 

This will look something like

   git checkout -b bavery/submit/bob/toaster/FixBug1234 

where bavery is signing off on and upstreaming bob's fix to the Yocto Bugzilla bug 1234.

---

---

• Alternatively, you can use the patchworks web site:

 2)  Create a new branch for submission 
   git checkout -b yourname/submit/username/toaster/FeatureOrBug 

This will look something like

   git checkout -b bavery/submit/bob/toaster/FixBug1234 

where bavery is signing off on and upstreaming bob's fix to the Yocto Bugzilla bug 1234.

 3)  Download and apply the mbox patch from the website: https://patchwork.openembedded.org/project/toaster/patches/

 git am -s the-downloaded-patch.mbox 

---

---

4) Make sure the branch is rebased on current master.

  git rebase origin/master

5) Test the changes. Run the Django unit tests. People put effort into these so we should make sure we use them. his assumes you have phantomjs installed. This can usually be done from the distribution apt-get install phantomjs, for example. If you want to test against Chrome or Firefox, see the README in bitbake/lib/toaster/tests/browser.

  pip3 install selenium 
 TOASTER_TESTS_BROWSER=phantomjs bitbake/lib/toaster/manage.py test orm toastergui tests.browser 

6) Add signed off by to the commit messages (Note: If you applied the mbox patch with git am -s foo.mbox You do not need to sign off again.)

  git filter-branch -f --msg-filter 'cat && echo "Signed-off-by: $(git config --get user.name) <$(git config --get user.email)>"' master..HEAD

7) Push the modified commit messages and rebased version to poky-contrib

  git push -u poky-contrib yourname/submit/username/toaster/FeatureOrBug  

8) Use the create-pull-request script (from poky) to create a pull request for the appropriate tree/mailing list

(a) For the 'bitbake' tree (e.g. bitbake/.../toasterui.py, bitbake/.../toastergui/*):

  ./scripts/create-pull-request -d bitbake -s "toaster: Fixes and clean ups" -u poky-contrib -r origin/master

(b) for the 'meta' tree (e.g. meta/classes/toaster.class):

  ./scripts/create-pull-request -d meta -s "toaster: Fixes and clean ups" -u poky-contrib -r origin/master

Note: If the patch creates any NEW files, the integration scripts that pull it into bitbake will fail. So, if new files are created as part of this patch set, you need to explicitly point that out in the body of the email for the patch set or do it as a PR rather than as a patch set.

9) Review their content, especially the summary mail:

  edit ./pull-<pid>/0000-cover-letter.patch

10) When you are satisfied, you can send them with:

  --- for the 'bitbake' tree:
  ./scripts/send-pull-request -a -p ./pull-<pid> -t bitbake-devel@lists.openembedded.org
  --- for the 'meta' tree:
  ./scripts/send-pull-request -a -p ./pull-<pid> -t openembedded-core@lists.openembedded.org

Submitting patches for prior releases

The procedure is the same, but using the prior release as the base branch instead of the "master" branch in bitbake.

Also, make sure that you add the name of the prior release for which the patchset is intended in the prefix of the patchset, as parameter to the "create-pull-request" command, e.g. -p 1.26 for the 1.26 branch.

Gotchas

Too Big

Sometimes the mailer will refuse to send patches, especially on binary or long-line files. The proper way to go around that is to reply to the patchset you've submitted to the mailing list, asking for a git pull directly from the poky-contrib branch.

One Patch of a Long Patch Set Needs Resubmission

Suppose you upstream a 10 commit patch set to the bitbake-devel list and someone finds an issue with patch #3. Regenerating the whole series is silly so how do you address this? First, follow the bitbake submission steps until you end up on the yourname/submit/the/target/branch branch. Then you can (note the reset --hard will wipe any local changes in your working dir so commit or stash first):

 git checkout -b yourname/submit/the/target/branch-newHEAD
 git reset --hard <commit of resubmission issue> 
 git commit --amend --signoff 
 git send-email --in-reply-to="longNumber.git.me@mycomp.com" --subject-prefix="bitbake-devel] [PATCHVX 03/10"  --to=bitbake-devel@lists.openembedded.org --no-chain-reply-to --suppress-cc=all -M -1 --relative=bitbake

The longNumber.git.me@mycomp.com comes from the email message id for the particular patch #3 so that the email threading works. In gmail, you can click on the dropdown button on the right side of the screen and choose "Show Original". This will have a field in the header like Message-Id: <4551b56f132497c055f39567946a5d3be347d770.1468363530.git.myemailusername@mycompany.com> The entire string except the '<>' are used. for example:

 --in-reply-to="4551b56f132497c055f39567946a5d3be347d770.1468363530.git.myemailusername@mycompany.com"

The first issue you are likely to face is that the git filter-branch command in the standard instructions may sign off on too many commits. If you know you just want to sign off on the last 7 commits on the yourname/submit/the/target/branch you can:

 git filter-branch -f --msg-filter 'cat && echo "Signed-off-by: $(git config --get user.name) <$(git config --get user.email)>"' HEAD~7..HEAD

Submitting patches for documentation

Documentation patches should be sent to Yocto mailing list with [yocto-docs] in the subject, CC Scott Rifenbark (and make sure you send it to his gmail, not his defunct Intel address). For his email address, look at this post.

Code syle guide

Templates

Django has a template language which allows us to render pages based on the data (context). We use the template language to setup the initial state of the page and to create re-usable components that can be included in other pages.

The recommend template code style is as follows

Yes please:

{{var}}

<div>
  {# Maintaining indentation #}
  {% if %}
   <p>this</p>
  {% else %}
   <p>that</p>
  {% endif %}
</div>

{% comment %}
This is a longer comment that describes all the things
that are below in quite a bit of detail because they're
a little more difficult to understand.
{% endcomment %}

{% for layer in layers_list %}
 {{layer}}
{% endfor %}

No thank you:

{{var}}
<div>
{# Maintaining indentation #}
{%if%}<p>this</p>{%else%}<p>that</p>{%endif%}
</div>


{#This is a longer comment that describes all the things that are below in quite a bit of detail because they're a little more difficult to understand. #}
{%for o in layers_list%}{{o}}{%endfor%}

Note:

• Maintain indentation as you would with other languages
• White space after '%'
• Comment blocks for longer comments

Javascript

Yes please:

"use strict";

/* These hold some numbers */
var oneVar = 1;
var twoVar = 2;

var cheesesTypes = {
  cheddar : 1,
  stilton : 2,
  emmental : 3, 
};

function doThingsHere(){
  return 1;
}

/* If one equals two do some other things and make sure that
 * if the the click handler is setup correctly.
 */
if (one === two) {
  var cheese = "cheddar";
  oneVar = doThingsHere();

  $(this).click(function (event){
    alert("Hello");
  });
}

$("#little-mouse").focusout(function(){
  alert("bye")
});

if (oneVar)
  noThingHere();
else
  doThingHere();

No thank you:

// These hold some numbers
oneVar = 1
twoVar = 2

var cheesesTypes = { cheddar : 1, stilton : 2,  emmental : 3, }

function doThingsHere ()
{
return 1;
}

//If one equals two do some other things and make sure that if the the click handler is setup correctly.
if( one === two ) {
var cheese = "cheddar";
oneVar = doThingsHere();

    $(this).click(function(event){ alert("Hello"); });
}

document.getElementById("little-mouse").addEventListener("focusout", function(){
  alert("bye")
});

if (oneVar)
{
  noThingHere();
} else {  doThingHere(); }

Note:

• Variables should be marked with "var"
• Semicolons should be used
• Keep as close to 80 cols as possible
• Use 2 space per indentation
• Open curly braces after parenthesis for functions and close on a new line
• Use camelCase for function names and variable names

Make use of running your Javascript through jshint we have a .jshint configuration file in that js directory (toastergui/static/js)

e.g. install jshint and add to your current PATH, then run:

$ npm install jshint; export PATH=$PATH:$PWD/node_modules/.bin/
$ jshint ./toastergui/static/js/base.js

HTML

Yes please:

<div id="something-area">
  <p class="important">This is some text</p>
</div>

<div>
  <p id="important-text>This is some text</p>
</div>

No thank you:

<div id="somethingarea">
    <p class="Important">This is some text</p></div>
<div id="somethingarea">
<p id="ImportantText">This is
some text
</p>

</div>

Note:

• 2 space indentation
• Lower case, ids hyphenated when multiple words
• No duplicate ids

• Run your HTML through a HTML validator available for local install. The w3c validator it's self doesn't currently validate html5, it uses as a back end Nu Html Checker which can be installed as a standalone service, full instructions in the readme.

Quick install instructions:

 $ mkdir html5-validator && cd html5-validator
 $ export JAVA_HOME=/usr/lib/jvm/java-6-openjdk
 $ git clone https://github.com/validator/validator.git
 $ python build/build.py all
 $ python build/build.py all

HTML can be indented quickly using tidy, for example:

 tidy -xml --indent auto --indent-spaces 2 --quiet yes -w -1 --show-body-only yes  ./index.html 

Python

Lenient pep8 Ignoring most of the whitespace around character issues (E124,E203,E201,E265,E303,E302,E231) see toaster/.pep8 and error code list

Fix all issues identified by running code through pep8. We have a fairly lenient config file (toaster/.pep8).

e.g.

$ pep8 ./toastergui/urls.py

Run code through pylint and fix identified issues - Some can be reasonably ignored such as doc strings for every function or star-args. No pylintrc config provided here as most issues identified are highly contextual and should be ignored on a case by case basis.

$ pylint --load-plugins pylint_django toastergui/tests.py

Working with design

Yes, the Yocto Project is one of those lucky projects with designers around to help in UI matters. We have a document explaining how to work with the design contributors: File:Working with design.pdf

Debugging Toaster

Toaster can be quite complex given that it includes a wide range of technologies and languages from bitbake, events, python, Django, Sqlite, Javescript, CCS, HTML, and more. Here are hints and tips for debugging all of these aspects of Toaster.

General debugging tips

• The logger is your best friend
• When possible, build from command line in the Toaster environment to see log messages and errors directly
• Use "quilt-native" as your quick sanity build target
• For python code you can use pudb (https://wiki.yoctoproject.org/wiki/TipsAndTricks/DebuggingBitbakeInPudb)
• For bbclass code (toaster.bbclass) find or set up a python section to use pudb (see above link)
• Brute force find is your friend for mysterious error messages and tracking all places a variable is used
  • find bitbake/lib/toaster -exec grep -l "SEARCH_TEXT" {} \; 2> /dev/null
• Look in https://wiki.yoctoproject.org/wiki/TipsAndTricks

How to debug a ...

• A stalled build:
  • Cancel the build and look at one of the below logs to find the exception. The error message should give a clue on the user resolution or indicate a bug to file
  • Failures in git and shell calls are (as of YP-2.4 Rocko) captured as visible build errors in the GUI
• Python file of a build management script (localhostbecontroller.py,...):
  • Use log statements, and watch "build/toaster_runbuilds.log" (where you started Toaster)
• Python file of a build runtime scripts:
  • Use log statements, and watch "build-toaster-x/toaster_ui.log"
• Python file of a Toaster management script (projects, tables):
  • Use log statements, and watch "build/toaster_web.log" (where you started Toaster)
• Template file
  • Usually the error will appear in the browser via Django, with a trace
  • Add random display text to insure the page your editing is the one that shows
  • Display internal values via page template to see what gets passed
  • Add internal values into the context and add to the page template
  • Use the Firefox/Chrome developer "inspector" to examine the DOM
• Javascript file
  • Use the Firefox/Chrome developer "inspector" to do inline debugging
  • Watch the Firefox/Chrome developer "console" for error messages
• The Toaster Database
  • Use an application like "sqlitebrowser" to examine the tables and records
  • Use a Python script to find and dump desired database records (see http://events.linuxfoundation.org/sites/events/files/slides/BitbakeAnalytics_ELC_Portland.pdf)

Specific error scenarios ...

• The build progress gets stuck at "Loading..."
  • You have a syntax error, either in 'localhostbecontroller.py', the helper classes in 'models.py', or in the related javascript. Look at the tail of "toaster_runbuilds.log" file for pythons errors, and in your browser's "Console" for javascript errors.
• The message "Sorry, An error has occurred loading this page"
  • The included javascript has a syntax error. The HTML page being rendered has local JS code at the top that caches these errors and displays the observed message (in order to be less ugly to the customer). That code also tries to add the error message to the browser console but that does not always work, and you are left with no clue
  • You can start bisecting the recent edits you made to the respective JS files to locate the problem change
  • You can also add a breakpoint in the mentioned catch code to see if you can see the error directly

How to update from a fix in a ...

• Template file: save your text edit and refresh page
• Javascript file: save your text edit and refresh page (or leave page and come back)
• Builder scripts: save your text edit and restart Toaster ("localhostbecontroller.py", "buildinfohelper.py", "toasterui.py", "bbcontroller.py",...)
• Views and URLs: save your text edit and refresh page
• Model class members: save your text edit and restart Toaster. You may also need a migration file.
• Model method: save your text edit and refresh page, else restart Toaster
• Fixture file (settings.xml, poky.xml, custom.xml): delete Toaster database and restart fresh

Where to debug ...

• Toaster start and stop, database init
  • bin/toaster
• Project Defaults (Default machine, distro, layers, ...)
  • bldcontrol/management/commands/checksettings.py
  • orm/fixtures/*.xml
• Layer Index content (Available machines, distros, layers, ...)
  • orm/management/commands/lsupdates.py
• Build cloning
  • bldcontrol/localhostbecontroller.py
• Build start
  • bldcontrol/localhostbecontroller.py
• URL mappings
  • toastergui/urls.py
• Views
  • toastergui/views.py
  • toastergui/templates/*
• Toaster Tables (Layers, Distros, Machines, ...)
  • toastergui/tables.py
  • toastergui/static/js/libtoaster.js
• Events
  • meta/classes/toaster.bbclass ->
  • lib/bb/ui/toasterui.py ->
  • lib/bb/ui/buildinfohelper.py ->
  • orm/models.py et. al.