Developer Workflow Improvements: Difference between revisions

From Yocto Project
Jump to navigationJump to search
m (→‎Configure: small typo)
 
(55 intermediate revisions by 2 users not shown)
Line 1: Line 1:
Back to Yocto Project [[Main Page]]
Back to Yocto Project [[Main Page]]
= Introduction =
= Introduction =
The page describes how new tools such as Toaster, Extensible SDK (eSDK) and CROPS make it easier to get the best out of Yocto. We envisage two distinct workflows, layer maintenance and application development.
The page describes how new tools such as [https://www.gitbook.com/book/dnplas/yocto-project-crops/details Cross Platform Support] (CROPS), [http://www.yoctoproject.org/docs/current/sdk-manual/sdk-manual.html#sdk-extensible-sdk-intro Extensible SDK] (eSDK), [http://www.yoctoproject.org/docs/current/dev-manual/dev-manual.html#dev-modifying-source-code devtool] and [https://www.yoctoproject.org/docs/current/toaster-manual/toaster-manual.html#toaster-manual-intro Toaster] make it easier to get the best out of Yocto. To benefit from this article the reader must be comfortable with the basics of the Yocto Project (i.e. has gone through [http://www.yoctoproject.org/docs/current/yocto-project-qs/yocto-project-qs.html Quick Start Guide] and knows how to add packages to an image).
 
We envisage two distinct workflows, layer maintenance and application development.
 
== Layer Maintenance ==
== Layer Maintenance ==
* Toaster is used to generate image that exercises layers under test
* This workflow is mainly bug fixing recipes and upgrading to never versions. Updated recipes are then submitted to layer maintainers. It also includes building images that exercise the recipes under test.
* Layers and recipes are edited as necessary. Image is rebuilt with Toaster and tested.
* Typically this is done in the distro maintainer (i.e. bitbake) environment, but the extensible SDK can be a better option as it enables an easy to configure common development environment.


== Application Development ==
== Application Development ==
* Toaster is used to generate a minimal eSDK installer.  
* Poky or Toaster is used to generate a minimal eSDK installer.  
* A developer uses resulting eSDK to create an application by using devtool (iterative process).  
* A developer uses resulting eSDK to create/modify an application by using devtool (iterative process).  
** devtool is used to create/fix/augment application recipe
** Toaster can optionally monitor builds and allow inspection of OE environment
* Once application is complete its recipe is added to a layer and Toaster is used to build an image that includes it
* Once application is complete its recipe is added to a layer and Toaster is used to build an image that includes it


There are two personas in this scenario; 'os-dev' who maintains the OS image and generates its extensible SDK and the 'app-dev' who creates new C/C++ applications to run on OS. The scenario assumes that an early version of the application exists but it needs features added and to be integrated into OS image. We recognize that in smaller organizations the roles of os-dev and app-dev may be blurry but the concept of two separate personas simplifies description of workflow.
= Workflow Vision =
This section calls out the workflows in more detail, describing specific use of tools in each step.


The set-up instructions at the end of this page represent the "what can currently be done" version of the workflows.
The [[Developer Workflow Improvements#Instructions for Workflow in Current Form | set-up instructions]] at the end of this page represent the "what can currently be done" version of the workflows.


I have a concern that the focus on C/C++ is misplaced as IoT is not proving to be just more embedded development as we knew it. Other languages are proving popular, with many pundits putting JavaScript at the forefront.
== Build OS Image ==
[https://www.linkedin.com/pulse/top-10-languages-iot-projects-toby-mcclean]
# On OS of your choice start containerized Poky or Toaster instance.
[https://www.quora.com/Which-programming-languages-will-be-most-valuable-in-the-IoT-Internet-of-Things]
# If running Poky container on Windows or Mac you must also start the samba container so that host OS has access to files in the container.
[https://blog.jscrambler.com/javascript-the-perfect-language-for-the-internet-of-things-iot]
# Add new layers and packages as required to create OS on which applications will run
# Build image and validate
 
== Creating Extensible SDK ==
# Build image as above
# Then build Extensible SDK
# Publish components required for app development
## eSDK installer
## eSDK updater files
## shared state


= Workflow Vision =
This section calls out the workflows in more detail, describing specific use of tools in each step.
== Application Development ==
== Application Development ==
# os-dev: On OS of your choice install containerized 'production' toaster instance. There will a number of containers; one with toaster and core tools, the other with a toaster database. These will have to be hooked together via bind mounts. Details TBD.# os-dev: Add new layer and application package to Toaster
=== One time set-up on OS of your choice ===
# os-dev: Build image and validate
# Create docker volume so that files fetched and created in the container are visible to host OS.
# os-dev: Add new layer and application package to Toaster
# Use extensible SDK container to install eSDK to volume.
# os-dev: Build image and validate
# Development can also be done on the command line.
 
# Optionally install Eclipse and CROPS plug-in that
# os-dev: Set up your base distro build with toaster by simply importing appropriate configuration (e.g. Ostro)
=== App development ===
# os-dev: Use toaster to create and publish components required for app development
# Start extensible SDK container so you can run devtool from command line or invoke it from Eclipse plug-in.
## OS image
# Start samba container so that files are visible to host OS
## eSDK
# Use '''devtool''' to 'add' application project from its URL. This pulls down source code, tries to work out dependent packages and generates a recipe
## sstate-cache created by building image
# Although generated recipe will likely build, it may need some tweaks to some of the tasks (e.g. configure, compile, install)
## CROPS toolchain container installers for Windows, Mac and Linux
# Once app is building, iteratvely develop using devtool (or Eclipse) to deploy and test on target
# app-dev: On OS of your choice install Eclipse and CROPS and configure
# Build new image including new layer and recipe and validate
# app-dev: Using Eclipse devtool plug-in add app-name, this pulls down all dependent packages and makes a good guess at recipe
# app-dev: Develop application code in Eclipse, deploying and testing on target
# app-dev: Check in "completed" code
# app-dev: Edit recipe using Eclipse devtool plug-in until it builds app correctly
# app-dev: Build new image using Eclipse devtool plug-in and test
# app-dev: Hand off recipe to os-dev
# os-dev: Create base image
# os-dev: Add recipe to layer or create a new one
# os-dev: Add new layer and application package to Toaster
# os-dev: Build image and validate


== Layer Maintenance ==
== Layer Maintenance ==
# os-dev: On OS of your choice install containerized 'production' toaster instance. There will a number of containers; one with toaster and core tools, the other with a toaster database. These will have to be hooked together via bind mounts. Details TBD.
# Build image as above
# os-dev: Set up your base distro build with toaster by simply importing appropriate configuration (e.g. Ostro)
# Use '''devtool modify''' to update recipe
# os-dev: Add layer(s) required by packages you add to image
# Iterative development as per application development.
# os-dev: Update and/or add layers as required with iterative image builds
# Build image and validate
# os-dev: Push layer updates to appropriate repo when done
# Publish updated recipe
# os-dev: Add new layer and application package to Toaster
# os-dev: Build image and validate


= Instructions for Workflow in Current Form =
= Instructions for Workflow in Current Form =
== Missing features ==
If you are inside a corporate firewall, install and configure the [https://github.com/crops/chameleonsocks Chameleonsocks] container based proxy solution.  
* Toaster on Windows and Mac [https://bugzilla.yoctoproject.org/show_bug.cgi?id=9793 [9793]]
 
* Base configuration (e.g. define what was Ostro in single configuration file) [https://bugzilla.yoctoproject.org/show_bug.cgi?id=9588 [9588]]
You have choice between using the Yocto Project tools on the command line with the Poky container or by using the Toaster web interface. Toaster is easier to get going but most needs more work to get the best of out interface (see section below), so these instructions will assume use of the command line.
* Simple generation of eSDK installer in Toaster. [https://bugzilla.yoctoproject.org/show_bug.cgi?id=9297 [9297]]
* CROPS application builder (but running eSDK in a container gets us close). [https://bugzilla.yoctoproject.org/show_bug.cgi?id=10119 [10119]] [https://bugzilla.yoctoproject.org/show_bug.cgi?id=4209 [4209]]
* CROPS Eclipse integration. ''In progress. No bugzilla entry.''
* Layer Maintenance workflow as Toaster reverts all layer and recipes files before starting a build. ''Patch under review. No bugzilla entry.''
* Simple way of publishing layer developed in Toaster to Layer Index [https://bugzilla.yoctoproject.org/show_bug.cgi?id=9895 [9895]]


== Toaster Setup ==
== Build OS image ==
=== Install Toaster ===
We recommend that you first [https://github.com/crops/docker-win-mac-docs/wiki install docker] on your OS of choice and then follow instructions to set up [https://github.com/crops/poky-container/blob/master/README.md Poky] or [https://github.com/crops/toaster-container/blob/master/README.md Toaster] containers. However, these instructions will work for Poky or Toaster on Linux directly but you're on your own for installation and configuration.
* Install [https://docs.docker.com/engine/installation/ docker for your Linux distro]. Install using your distro's package feed as this will also configure you to be a docker user so you don't need to use sudo when running a container
* If inside a corporate firewall, install and configure the [https://github.com/crops/chameleonsocks Chameleonsocks] container based proxy solution.  
* Make a folder for toaster output files on your workstation.
mkdir /path/to/toaster/tmp_files


=== Start Toaster Web Service ===
Now add layers and configure so you can build your image.
Start toaster container so it's only available on localhost
docker run -it --rm -p 127.0.0.1:18000:8000 -v /path/to/toaster/tmp_files:/workdir crops/toaster
Or start toaster container so anyone can access it remotely
docker run -it --rm -p 18000:8000 -v /path/to/toaster/tmp_files:/workdir crops/toaster
For more details on Toaster container see https://github.com/crops/toaster-container.


== Application Development ==
== Build and Publish eSDK ==
=== Build and Publish eSDK ===
This section is not part of the developer workflow but documents how the SDK components are generated and published.  
This section is not part of the developer workflow but documents how the SDK components are generated and published.  
This step creates a small installer that is made available to the developer. You must have access to an http server so you can publish SDK update files.
This step creates a small installer that is made available to the developer. You must have access to an http server so you can publish SDK update files. <br/>
* Set the following variables. Normally this would be done in local.conf but as we are using toaster we add them as bitbake variables.
Building and publishing the SDK takes a number of steps.
** SDK_INCLUDE_PKGDATA = "1" forces a 'world' build up to do_packagedata and is required to populate sstate-cache with all packages, as we don't know which ones developers will need. If you know the packages the application developers will require, add then the your build and avoid setting SDK_INCLUDE_PKGDATA for a much faster build.  
=== Configure ===
Set the following variables in local.conf  
:<tt>SDK_EXT_TYPE = "minimal"</tt> ensure a minimal SDK build. This produces a small installer and enable components to de downloaded as they are installed
: SDK_UPDATE_URL points eSDK to URL that contains updates
:Optionally set <tt>SDK_INCLUDE_PKGDATA = "1"</tt> to help eSDK users by pre-building all packages in all layers (i.e. not just packages included in the image). The installer remains small as build data is made available via sstate mirror. Downside is that eSDK build will take much longer.
 
Here are some example settings
  SDK_EXT_TYPE = "minimal"
  SDK_EXT_TYPE = "minimal"
SDK_UPDATE_URL = "http://my.server.com/path/to/esdk-update"
  SDK_INCLUDE_PKGDATA = "1"
  SDK_INCLUDE_PKGDATA = "1"
* Now set SSTATE_MIRRORS URL so devtool can use your sstate-cache to avoid re-building packages. This is normally done by setting SSTATE_MIRRORS in conf/sdk-extra.conf but this is not possible as we're using toaster. Instead ensure you are not already using an existing SSTATE_MIRROR and set bitbake variables as follows
 
The URL for the eSDK sstate mirror is not included in the above settings as it would conflict with the mirror used by the main build system. To handle this conflict, the value in put into a separate conf file <tt>conf/sdk-extra.conf</tt>
SSTATE_MIRRORS = "file://.* http://my.server.com/path/to/sstate-cache/PATH"
Note that if you are using Toaster, you don't have access to <tt>conf/sdk-extra.conf</tt>, so you have to set variables via the U/I. This means that Toaster cannot have an sstate mirror for the OS build and set the eSDK's as follows:
  SDK_LOCAL_CONF_WHITELIST = "SSTATE_MIRRORS"
  SDK_LOCAL_CONF_WHITELIST = "SSTATE_MIRRORS"
  SSTATE_MIRRORS = "file://.* http://path/to/build/sstate-cache/PATH"  
  SSTATE_MIRRORS = "file://.* http://my.server.com/path/to/sstate-cache/PATH"
* Build SDK by building 'your-image:populate_sdk_ext' in toaster.
* Copy installer to web server and give this URL to your app developers
cp tmp/deploy/sdk/distro-image-toolchain-ext-ver.sh /var/www/html/path/to/sdk-installer


=== Build ===
Assuming image name is my-image, build as follows from command line
bitbake my-image -c populate_sdk_ext
If using Toaster, build <tt>my-image:populate_sdk_ext</tt>.
=== Publish SDK Updater ===
The eSDK allows web updates, so you have to ensure they appeat at the URK specified by SDK_UPDATE_URL. Assuming /var/www/html/path/to is served up as http://my.server.com/path/to, publish as follows
oe-publish-sdk tmp/deploy/sdk/distro-image-toolchain-ext-ver.sh /var/www/html/path/to/esdk-update
=== Publish SSTATE ===
cp -r build/sstate-cache /var/www/html/path/to/sstate-cache
=== Publish installer ===
Copy installer to web server and give this URL to your app developers.
cp tmp/deploy/sdk/distro-image-toolchain-ext-ver.sh /var/www/html/path/to/sdk-installer.sh
== Application Development ==
=== Install and Use eSDK ===
=== Install and Use eSDK ===
* This assumes you have docker already installed as per Toaster instructions. No other dependencies are required (that the beauty of containers).
* This assumes you have docker already installed as per Toaster instructions. No other dependencies are required (that the beauty of containers).
* Create folder for esdk bind mount where output from container will appear, say /path/to/esdk/workdir
* Create folder for esdk bind mount where output from container will appear, say /path/to/esdk/workdir
  mkdir /path/to/esdk/workdir
  mkdir /path/to/esdk/workdir
* Run container, pointing to URL of eSDK minimal installer. You must do this every time you want to use eSDK. In this example I'm suing an eSDk installer I created that is available at  http://downloads.yoctoproject.org/tools/support/workflow/esdk/poky-glibc-x86_64-core-image-minimal-i586-toolchain-ext-2.0%2bsnapshot.sh
* Run container, pointing to URL of eSDK minimal installer. You must do this every time you want to use eSDK. In this example I'm suing an eSDk installer I created that is available at  https://downloads.yoctoproject.org/tools/support/workflow/poky-glibc-x86_64-core-image-minimal-i586-toolchain-ext-2.2.sh
  docker run --rm -it -v /path/to/esdk-workdir:/workdir crops/extsdk-container --url http://downloads.yoctoproject.org/tools/support/workflow/esdk/poky-glibc-x86_64-core-image-minimal-i586-toolchain-ext-2.0%2bsnapshot.sh
  docker run --rm -it -v /path/to/esdk-workdir:/workdir crops/extsdk-container --url https://downloads.yoctoproject.org/tools/support/workflow/poky-glibc-x86_64-core-image-minimal-i586-toolchain-ext-2.2.sh
If you see the error '''ERROR: The extensible sdk cannot be installed as root.''' it means that the folder /path/to/esdk-workdir does not exist
If you see the error '''ERROR: The extensible sdk cannot be installed as root.''' it means that the folder /path/to/esdk-workdir does not exist
* You now be in the eSDK container shell.  
* You now be in the eSDK container shell.  
Line 107: Line 113:
* Now follow instructions from [https://wiki.yoctoproject.org/wiki/Cookbook:Example:Creating_and_updating_recipes_with_devtool Yocto Project cookbook] on building an app.
* Now follow instructions from [https://wiki.yoctoproject.org/wiki/Cookbook:Example:Creating_and_updating_recipes_with_devtool Yocto Project cookbook] on building an app.
* Recipe will be available in /path/to/esdk/workdir/workspace/recipes/bbexample/
* Recipe will be available in /path/to/esdk/workdir/workspace/recipes/bbexample/
= Missing Features =
These are the features currently missing from the above workflow vision. Think of these as stories in an Agile development environment, rather than bugs. Where possible these are linked to bugzilla entries.
* Toaster: Base configuration (e.g. define what was Ostro in single configuration file) [https://bugzilla.yoctoproject.org/show_bug.cgi?id=9588 [9588]]
* Toaster: Simple generation of eSDK installer in Toaster. [https://bugzilla.yoctoproject.org/show_bug.cgi?id=9297 [9297]]
* Toaster: Layer Maintenance workflow. Requires "local layers" re-design. No bugzilla entry, but see [https://www.youtube.com/watch?v=z5wVjBwJDzY design proposal video]
* Toaster: Publish layer to Layer Index from Toaster [https://bugzilla.yoctoproject.org/show_bug.cgi?id=9895 [9895]]
* CROPS: Application builder (but running eSDK in a container gets us close). [https://bugzilla.yoctoproject.org/show_bug.cgi?id=10119 [10119]] [https://bugzilla.yoctoproject.org/show_bug.cgi?id=4209 [4209]]
* CROPS: Eclipse integration. ''In progress. No bugzilla entry.''

Latest revision as of 19:52, 24 January 2023

Back to Yocto Project Main Page

Introduction

The page describes how new tools such as Cross Platform Support (CROPS), Extensible SDK (eSDK), devtool and Toaster make it easier to get the best out of Yocto. To benefit from this article the reader must be comfortable with the basics of the Yocto Project (i.e. has gone through Quick Start Guide and knows how to add packages to an image).

We envisage two distinct workflows, layer maintenance and application development.

Layer Maintenance

  • This workflow is mainly bug fixing recipes and upgrading to never versions. Updated recipes are then submitted to layer maintainers. It also includes building images that exercise the recipes under test.
  • Typically this is done in the distro maintainer (i.e. bitbake) environment, but the extensible SDK can be a better option as it enables an easy to configure common development environment.

Application Development

  • Poky or Toaster is used to generate a minimal eSDK installer.
  • A developer uses resulting eSDK to create/modify an application by using devtool (iterative process).
  • Once application is complete its recipe is added to a layer and Toaster is used to build an image that includes it

Workflow Vision

This section calls out the workflows in more detail, describing specific use of tools in each step.

The set-up instructions at the end of this page represent the "what can currently be done" version of the workflows.

Build OS Image

  1. On OS of your choice start containerized Poky or Toaster instance.
  2. If running Poky container on Windows or Mac you must also start the samba container so that host OS has access to files in the container.
  3. Add new layers and packages as required to create OS on which applications will run
  4. Build image and validate

Creating Extensible SDK

  1. Build image as above
  2. Then build Extensible SDK
  3. Publish components required for app development
    1. eSDK installer
    2. eSDK updater files
    3. shared state

Application Development

One time set-up on OS of your choice

  1. Create docker volume so that files fetched and created in the container are visible to host OS.
  2. Use extensible SDK container to install eSDK to volume.
  3. Development can also be done on the command line.
  4. Optionally install Eclipse and CROPS plug-in that

App development

  1. Start extensible SDK container so you can run devtool from command line or invoke it from Eclipse plug-in.
  2. Start samba container so that files are visible to host OS
  3. Use devtool to 'add' application project from its URL. This pulls down source code, tries to work out dependent packages and generates a recipe
  4. Although generated recipe will likely build, it may need some tweaks to some of the tasks (e.g. configure, compile, install)
  5. Once app is building, iteratvely develop using devtool (or Eclipse) to deploy and test on target
  6. Build new image including new layer and recipe and validate

Layer Maintenance

  1. Build image as above
  2. Use devtool modify to update recipe
  3. Iterative development as per application development.
  4. Build image and validate
  5. Publish updated recipe

Instructions for Workflow in Current Form

If you are inside a corporate firewall, install and configure the Chameleonsocks container based proxy solution.

You have choice between using the Yocto Project tools on the command line with the Poky container or by using the Toaster web interface. Toaster is easier to get going but most needs more work to get the best of out interface (see section below), so these instructions will assume use of the command line.

Build OS image

We recommend that you first install docker on your OS of choice and then follow instructions to set up Poky or Toaster containers. However, these instructions will work for Poky or Toaster on Linux directly but you're on your own for installation and configuration.

Now add layers and configure so you can build your image.

Build and Publish eSDK

This section is not part of the developer workflow but documents how the SDK components are generated and published. This step creates a small installer that is made available to the developer. You must have access to an http server so you can publish SDK update files.
Building and publishing the SDK takes a number of steps.

Configure

Set the following variables in local.conf

SDK_EXT_TYPE = "minimal" ensure a minimal SDK build. This produces a small installer and enable components to de downloaded as they are installed
SDK_UPDATE_URL points eSDK to URL that contains updates
Optionally set SDK_INCLUDE_PKGDATA = "1" to help eSDK users by pre-building all packages in all layers (i.e. not just packages included in the image). The installer remains small as build data is made available via sstate mirror. Downside is that eSDK build will take much longer.

Here are some example settings

SDK_EXT_TYPE = "minimal"
SDK_UPDATE_URL = "http://my.server.com/path/to/esdk-update"
SDK_INCLUDE_PKGDATA = "1"

The URL for the eSDK sstate mirror is not included in the above settings as it would conflict with the mirror used by the main build system. To handle this conflict, the value in put into a separate conf file conf/sdk-extra.conf

SSTATE_MIRRORS = "file://.* http://my.server.com/path/to/sstate-cache/PATH" 

Note that if you are using Toaster, you don't have access to conf/sdk-extra.conf, so you have to set variables via the U/I. This means that Toaster cannot have an sstate mirror for the OS build and set the eSDK's as follows:

SDK_LOCAL_CONF_WHITELIST = "SSTATE_MIRRORS"
SSTATE_MIRRORS = "file://.* http://my.server.com/path/to/sstate-cache/PATH"

Build

Assuming image name is my-image, build as follows from command line

bitbake my-image -c populate_sdk_ext

If using Toaster, build my-image:populate_sdk_ext.

Publish SDK Updater

The eSDK allows web updates, so you have to ensure they appeat at the URK specified by SDK_UPDATE_URL. Assuming /var/www/html/path/to is served up as http://my.server.com/path/to, publish as follows

oe-publish-sdk tmp/deploy/sdk/distro-image-toolchain-ext-ver.sh /var/www/html/path/to/esdk-update

Publish SSTATE

cp -r build/sstate-cache /var/www/html/path/to/sstate-cache

Publish installer

Copy installer to web server and give this URL to your app developers.

cp tmp/deploy/sdk/distro-image-toolchain-ext-ver.sh /var/www/html/path/to/sdk-installer.sh

Application Development

Install and Use eSDK

  • This assumes you have docker already installed as per Toaster instructions. No other dependencies are required (that the beauty of containers).
  • Create folder for esdk bind mount where output from container will appear, say /path/to/esdk/workdir
mkdir /path/to/esdk/workdir
docker run --rm -it -v /path/to/esdk-workdir:/workdir crops/extsdk-container --url https://downloads.yoctoproject.org/tools/support/workflow/poky-glibc-x86_64-core-image-minimal-i586-toolchain-ext-2.2.sh

If you see the error ERROR: The extensible sdk cannot be installed as root. it means that the folder /path/to/esdk-workdir does not exist

Missing Features

These are the features currently missing from the above workflow vision. Think of these as stories in an Agile development environment, rather than bugs. Where possible these are linked to bugzilla entries.

  • Toaster: Base configuration (e.g. define what was Ostro in single configuration file) [9588]
  • Toaster: Simple generation of eSDK installer in Toaster. [9297]
  • Toaster: Layer Maintenance workflow. Requires "local layers" re-design. No bugzilla entry, but see design proposal video
  • Toaster: Publish layer to Layer Index from Toaster [9895]
  • CROPS: Application builder (but running eSDK in a container gets us close). [10119] [4209]
  • CROPS: Eclipse integration. In progress. No bugzilla entry.