Developer Workflow Improvements: Difference between revisions

From Yocto Project
Jump to navigationJump to search
No edit summary
No edit summary
Line 1: Line 1:
Developer workflow improvements as documented in the following bugzilla entry:
Back to Yocto Project [[Main Page]]
= Introduction =
The page describes how new tools such as Toaster, Extensible SDK (eSDK) and CROPS make it easier to get the best out of Yocto. We envisage two distinct workflows, layer maintenance and application development.
== Layer Maintenance ==
* Toaster is used to generate image that exercises layers under test
* Layers and recipes are edited as necessary. Image is rebuilt with Toaster and tested.


https://bugzilla.yoctoproject.org/show_bug.cgi?id=6662
== Application Development ==
* Toaster is used to generate a minimal eSDK installer.  
* A developer uses resulting eSDK to create an application by using devtool (iterative process).  
** devtool is used to create/fix/augment application recipe
** Toaster can optionally monitor builds and allow inspection of OE environment
* Once application is complete its recipe is added to a layer and Toaster is used to build an image that includes it


== Todo list ==
There are two personas in this scenario; 'os-dev' who maintains the OS image and generates its extensible SDK and the 'app-dev' who creates new C/C++ applications to run on OS. The scenario assumes that an early version of the application exists but it needs features added and to be integrated into OS image. We recognize that in smaller organizations the roles of os-dev and app-dev may be blurry but the concept of two separate personas simplifies description of workflow.


=== SDK ===
The set-up instructions at the end of this page represent the "what can currently be done" version of the workflows.
* Randy: getting errors from bitbake about changed signatures - behaviour change after Hongxu's patch?
* Fix runqemu''(and possibly other tools?)'' since we no longer have the nativesdk sysroot ''(Add them to buildtools-lite?)''
* nativesdk-qemu-helper_1.0.bb and qemu-helper_1.0.bb don't publish the same set of files. Investigate, since runqemu(and possibly other things) may need to come from the buildsystem's native sysroot.
* When publishing the SDK(tool not available yet), warn the user if there are items added to the image/distro via local.conf and suggest moving the items the distro or image config.
* Add a message to the start of the environment script with pointers to the things you can do with it


=== recipetool ===
I have a concern that the focus on C/C++ is misplaced as IoT is not proving to be just more embedded development as we knew it. Other languages are proving popular, with many pundits putting JavaScript at the forefront.
* Add comments in on do_configure(_append), do_install_(append) etc.
[https://www.linkedin.com/pulse/top-10-languages-iot-projects-toby-mcclean]
* Handle USE_* options in Makefiles?
[https://www.quora.com/Which-programming-languages-will-be-most-valuable-in-the-IoT-Internet-of-Things]
[http://www.iotcentral.io/application-development/will-javascript-be-the-language-of-iot-1]
[https://blog.jscrambler.com/javascript-the-perfect-language-for-the-internet-of-things-iot]
 
= Workflow Vision =
This section calls out the workflows in more detail, describing specific use of tools in each step.
== Application Development ==
# os-dev: On OS of your choice install containerized 'production' toaster instance. There will a number of containers; one with toaster and core tools, the other with a toaster database. These will have to be hooked together via bind mounts. Details TBD.# os-dev: Add new layer and application package to Toaster
# os-dev: Build image and validate
# os-dev: Add new layer and application package to Toaster
# os-dev: Build image and validate
 
# os-dev: Set up your base distro build with toaster by simply importing appropriate configuration (e.g. Ostro)
# os-dev: Use toaster to create and publish components required for app development
## OS image
## eSDK
## sstate-cache created by building image
## CROPS toolchain container installers for Windows, Mac and Linux
# app-dev: On OS of your choice install Eclipse and CROPS and configure
# app-dev: Using Eclipse devtool plug-in add app-name, this pulls down all dependent packages and makes a good guess at recipe
# app-dev: Develop application code in Eclipse, deploying and testing on target
# app-dev: Check in "completed" code
# app-dev: Edit recipe using Eclipse devtool plug-in until it builds app correctly
# app-dev: Build new image using Eclipse devtool plug-in and test
# app-dev: Hand off recipe to os-dev
# os-dev: Create base image
# os-dev: Add recipe to layer or create a new one
# os-dev: Add new layer and application package to Toaster
# os-dev: Build image and validate
 
== Layer Maintenance ==
# os-dev: On OS of your choice install containerized 'production' toaster instance. There will a number of containers; one with toaster and core tools, the other with a toaster database. These will have to be hooked together via bind mounts. Details TBD.
# os-dev: Set up your base distro build with toaster by simply importing appropriate configuration (e.g. Ostro)
# os-dev: Add layer(s) required by packages you add to image
# os-dev: Update and/or add layers as required with iterative image builds
# os-dev: Push layer updates to appropriate repo when done
# os-dev: Add new layer and application package to Toaster
# os-dev: Build image and validate
 
= Instructions for Workflow in Current Form =
== Missing features ==
* Toaster container not in production mode
* CROPS (but running eSDK in a container gets us close)
* Eclipse integration
* All of Layer Maintenance workflow as Toaster reverts all layer and recipes files before starting a build
 
== Toaster Setup ==
=== Install Toaster ===
* Install [https://docs.docker.com/engine/installation/ docker for your Linux distro]. Install using your distro's package feed as this will also configure you to be a docker user so you don't need to use sudo when running a container
* If inside a corporate firewall, install and configure the [https://github.com/crops/chameleonsocks Chameleonsocks] container based proxy solution.
* Make a folder for toaster output files on your workstation.
mkdir /path/to/toaster/tmp_files
 
=== Start Toaster Web Service ===
Start toaster container so it's only available on localhost
docker run -it --rm -p 127.0.0.1:18000:8000 -v /path/to/toaster/tmp_files:/workdir crops/toaster
Or start toaster container so anyone can access it remotely
docker run -it --rm -p 18000:8000 -v /path/to/toaster/tmp_files:/workdir crops/toaster
For more details on Toaster container see https://github.com/crops/toaster-container.
 
== Application Development ==
=== Build and Publish eSDK ===
This section is not part of the developer workflow but documents how the SDK components are generated and published.
This step creates a small installer that is made available to the developer. You must have access to an http server so you can publish SDK update files.
* Set the following variables. Normally this would be done in local.conf but as we are using toaster we add them as bitbake variables.
** SDK_INCLUDE_PKGDATA = "1" forces a 'world' build up to do_packagedata and is required to populate sstate-cache with all packages, as we don't know which ones developers will need.
SDK_EXT_TYPE = "minimal"
SDK_INCLUDE_PKGDATA = "1"
* Now set SSTATE_MIRRORS URL so devtool can use your sstate-cache to avoid re-building packages. This is normally done by setting SSTATE_MIRRORS in conf/sdk-extra.conf but this is not possible as we're using toaster. Instead ensure you are not already using an existing SSTATE_MIRROR and set bitbake variables as follows
SDK_LOCAL_CONF_WHITELIST = "SSTATE_MIRRORS"
SSTATE_MIRRORS = "file://.* http://path/to/build/sstate-cache/PATH"
* Build SDK by building 'your-image:populate_sdk_ext' in toaster.
* Copy installer to web server and give this URL to your app developers
cp tmp/deploy/sdk/distro-image-toolchain-ext-ver.sh /var/www/html/path/to/sdk-installer
 
=== Install and Use eSDK ===
* This assumes you have docker already installed as per Toaster instructions. No other dependencies are required (that the beauty of containers).
* Create folder for esdk bind mount where output from container will appear, say /path/to/esdk/workdir
mkdir /path/to/esdk/workdir
* Run container, pointing to URL of eSDK minimal installer. You must do this every time you want to use eSDK. In this example I'm suing an eSDk installer I created that is available at  http://downloads.yoctoproject.org/tools/support/workflow/esdk/poky-glibc-x86_64-core-image-minimal-i586-toolchain-ext-2.0%2bsnapshot.sh
docker run --rm -it -v /path/to/esdk-workdir:/workdir crops/extsdk-container --url http://downloads.yoctoproject.org/tools/support/workflow/esdk/poky-glibc-x86_64-core-image-minimal-i586-toolchain-ext-2.0%2bsnapshot.sh
If you see the error '''ERROR: The extensible sdk cannot be installed as root.''' it means that the folder /path/to/esdk-workdir does not exist
* You now be in the eSDK container shell.
** For more details on eSDK container see https://github.com/crops/extsdk-container.
* Now follow instructions from [https://wiki.yoctoproject.org/wiki/Cookbook:Example:Creating_and_updating_recipes_with_devtool Yocto Project cookbook] on building an app.
* Recipe will be available in /path/to/esdk/workdir/workspace/recipes/bbexample/

Revision as of 00:10, 3 August 2016

Back to Yocto Project Main Page

Introduction

The page describes how new tools such as Toaster, Extensible SDK (eSDK) and CROPS make it easier to get the best out of Yocto. We envisage two distinct workflows, layer maintenance and application development.

Layer Maintenance

  • Toaster is used to generate image that exercises layers under test
  • Layers and recipes are edited as necessary. Image is rebuilt with Toaster and tested.

Application Development

  • Toaster is used to generate a minimal eSDK installer.
  • A developer uses resulting eSDK to create an application by using devtool (iterative process).
    • devtool is used to create/fix/augment application recipe
    • Toaster can optionally monitor builds and allow inspection of OE environment
  • Once application is complete its recipe is added to a layer and Toaster is used to build an image that includes it

There are two personas in this scenario; 'os-dev' who maintains the OS image and generates its extensible SDK and the 'app-dev' who creates new C/C++ applications to run on OS. The scenario assumes that an early version of the application exists but it needs features added and to be integrated into OS image. We recognize that in smaller organizations the roles of os-dev and app-dev may be blurry but the concept of two separate personas simplifies description of workflow.

The set-up instructions at the end of this page represent the "what can currently be done" version of the workflows.

I have a concern that the focus on C/C++ is misplaced as IoT is not proving to be just more embedded development as we knew it. Other languages are proving popular, with many pundits putting JavaScript at the forefront. [1] [2] [3] [4]

Workflow Vision

This section calls out the workflows in more detail, describing specific use of tools in each step.

Application Development

  1. os-dev: On OS of your choice install containerized 'production' toaster instance. There will a number of containers; one with toaster and core tools, the other with a toaster database. These will have to be hooked together via bind mounts. Details TBD.# os-dev: Add new layer and application package to Toaster
  2. os-dev: Build image and validate
  3. os-dev: Add new layer and application package to Toaster
  4. os-dev: Build image and validate
  1. os-dev: Set up your base distro build with toaster by simply importing appropriate configuration (e.g. Ostro)
  2. os-dev: Use toaster to create and publish components required for app development
    1. OS image
    2. eSDK
    3. sstate-cache created by building image
    4. CROPS toolchain container installers for Windows, Mac and Linux
  3. app-dev: On OS of your choice install Eclipse and CROPS and configure
  4. app-dev: Using Eclipse devtool plug-in add app-name, this pulls down all dependent packages and makes a good guess at recipe
  5. app-dev: Develop application code in Eclipse, deploying and testing on target
  6. app-dev: Check in "completed" code
  7. app-dev: Edit recipe using Eclipse devtool plug-in until it builds app correctly
  8. app-dev: Build new image using Eclipse devtool plug-in and test
  9. app-dev: Hand off recipe to os-dev
  10. os-dev: Create base image
  11. os-dev: Add recipe to layer or create a new one
  12. os-dev: Add new layer and application package to Toaster
  13. os-dev: Build image and validate

Layer Maintenance

  1. os-dev: On OS of your choice install containerized 'production' toaster instance. There will a number of containers; one with toaster and core tools, the other with a toaster database. These will have to be hooked together via bind mounts. Details TBD.
  2. os-dev: Set up your base distro build with toaster by simply importing appropriate configuration (e.g. Ostro)
  3. os-dev: Add layer(s) required by packages you add to image
  4. os-dev: Update and/or add layers as required with iterative image builds
  5. os-dev: Push layer updates to appropriate repo when done
  6. os-dev: Add new layer and application package to Toaster
  7. os-dev: Build image and validate

Instructions for Workflow in Current Form

Missing features

  • Toaster container not in production mode
  • CROPS (but running eSDK in a container gets us close)
  • Eclipse integration
  • All of Layer Maintenance workflow as Toaster reverts all layer and recipes files before starting a build

Toaster Setup

Install Toaster

  • Install docker for your Linux distro. Install using your distro's package feed as this will also configure you to be a docker user so you don't need to use sudo when running a container
  • If inside a corporate firewall, install and configure the Chameleonsocks container based proxy solution.
  • Make a folder for toaster output files on your workstation.
mkdir /path/to/toaster/tmp_files

Start Toaster Web Service

Start toaster container so it's only available on localhost

docker run -it --rm -p 127.0.0.1:18000:8000 -v /path/to/toaster/tmp_files:/workdir crops/toaster

Or start toaster container so anyone can access it remotely

docker run -it --rm -p 18000:8000 -v /path/to/toaster/tmp_files:/workdir crops/toaster

For more details on Toaster container see https://github.com/crops/toaster-container.

Application Development

Build and Publish eSDK

This section is not part of the developer workflow but documents how the SDK components are generated and published. This step creates a small installer that is made available to the developer. You must have access to an http server so you can publish SDK update files.

  • Set the following variables. Normally this would be done in local.conf but as we are using toaster we add them as bitbake variables.
    • SDK_INCLUDE_PKGDATA = "1" forces a 'world' build up to do_packagedata and is required to populate sstate-cache with all packages, as we don't know which ones developers will need.
SDK_EXT_TYPE = "minimal"
SDK_INCLUDE_PKGDATA = "1"
  • Now set SSTATE_MIRRORS URL so devtool can use your sstate-cache to avoid re-building packages. This is normally done by setting SSTATE_MIRRORS in conf/sdk-extra.conf but this is not possible as we're using toaster. Instead ensure you are not already using an existing SSTATE_MIRROR and set bitbake variables as follows
SDK_LOCAL_CONF_WHITELIST = "SSTATE_MIRRORS"
SSTATE_MIRRORS = "file://.* http://path/to/build/sstate-cache/PATH" 
  • Build SDK by building 'your-image:populate_sdk_ext' in toaster.
  • Copy installer to web server and give this URL to your app developers
cp tmp/deploy/sdk/distro-image-toolchain-ext-ver.sh /var/www/html/path/to/sdk-installer

Install and Use eSDK

  • This assumes you have docker already installed as per Toaster instructions. No other dependencies are required (that the beauty of containers).
  • Create folder for esdk bind mount where output from container will appear, say /path/to/esdk/workdir
mkdir /path/to/esdk/workdir
docker run --rm -it -v /path/to/esdk-workdir:/workdir crops/extsdk-container --url http://downloads.yoctoproject.org/tools/support/workflow/esdk/poky-glibc-x86_64-core-image-minimal-i586-toolchain-ext-2.0%2bsnapshot.sh

If you see the error ERROR: The extensible sdk cannot be installed as root. it means that the folder /path/to/esdk-workdir does not exist