Development Manual Rework

Intro

This page will propose an outline for Development Manual refactoring work. See also bug #11630

Henry's Input

I believe the manual lost its way as it became the dumping ground for useful (and often advanced) info that had nowhere else to land. As a result the size of the manual and top level topics became overwhelming to the new developer who had just graduated from the quick state guide. Currently this user is not served by the Yocto documentation, thus the creation of the wiki cookbook. Simple tasks such as adding an existing recipe to image and creating a recipe source should be the bread and butter of the development manual. If this approach makes the development master server too many master, we should dicuss re-structing all the docs the addition of a new one.

Basic "How To" Topics

I plan to structure these, but just a plain list for the time being. Many of these topics are already documented, but this feels like a better flow that we have just now.

Basic Skills for Systems Developers

• Deal with corporate proxies.
• Add a package to an image
• Understand the difference between a recipe and package
• Build a package by itself and why that's useful
• Find out what packages are created by a recipe
• Find out what files are in a package
• Find out what files are in an image
• Add an ssh server to an image (enable transferring of files to target)
• Anatomy of a recipe
• How to find recipes (layers.openembedded,org)
• Understand difference between machine and distro settings
• How to find and use the right BSP (machine) for your hardware
• Examples of distro features and where to set them.
• Understanding the task pipeline and executing individual tasks
• Understand devtool and how it simplifies your workflow
• Improve build speeds with shared downloads and shared state cache
• Generate and understand dependency graph
• Generate and understand bitbake environment
• Building an Extensible SDK

Basic Skills for Application Developers

• Understand intent and content of extensible SDK (eSDK) and why traditional SDK is no longer necessary
• Understand CROPS eSDK container solution
  • Source code is edited outside of container using tools of your choice on host
• Install and run CROPS/eSDK
• Intro to devtool
• Make sure you're up to date
• Build the image (may not be the same as on your target) and typical options for flashing it
• Modify an existing recipe
  • Understand how git is managing source code
  • Build and deploy modified recipe
  • Using devtool finish to create updated recipe and patches.
  • Understand how this is done without updating upstream source code
• Add a new recipe
  • Create recipe from source code
  • Review recipe to see what's missing
  • Repeat steps as for 'modify'

Tracey's Input

The way Henry described this yesterday, customers use the "getting started" guide to learn how to bring up a vanilla image, but then don't know how to approach their own custom situation. This should be a TRANSITION DOCUMENT - how to use YP once you have gotten your feet wet in a controlled situation. Feel free to delete this, I don't know how you folks work in the WIKI and I'm assuming this may not be it.

I would suggest organizing this in a couple of ways - and this might be better in a google doc, frankly. 1) by what people need to do to get started

  a) where to start to support a platform
     1. find a BSP and test it
  b) how to add support for platform components
  c) how to slim down an image size
     1. Remove code automatically included
     2. design your own BSP fine tuned to exclude extra code, unneeded support code
  d) How to approach setting up a development environment
  e) How use layers to create a flexible long term design, and recipes
  d) How do you start a new project vs build off an existing one   
  e) How do you set up a team sharable development environment (sstate)
  f) Here are some of the most common commands 
  g) How to deal with CVEs and code branches

2) By helping them collect the information they need in order to make set up decisions Brent had a great idea to make this a SW Wizard that would step a new user through a series of questions and give them at least one solution on how to proceed.

  a) what platform are you using
  b) will you include an application as part of the image or will it run on the image
  c) what is the configuration of the team (system developers, app developers, etc)
  c) How to help them choose the right kernel
  d) etc.

Scott's Input

Scott Rifenbark - This is a good idea. I was thinking of a similar way to deal with this with sort of a decision tree at the end of the YP Quick Start (flowchart for old-schoolers) that would lead the reader down a path based on information they have about what they want to get accomplished. Then, based on how they traverse the tree, we point them to areas they need to use in the manual.

Notes for Chapter 1 of the dev-manual:

I re-wrote this introductory section to work more tightly with the introductory material of the ref-manual. Also, created a better "Other Information" section to lead the reader to various places of interest. In particular, I am leveraging off the Yocto Project Backgrounders that exist on the YP Website. These backgrounders are probably the best place for an overall introduction to what the YP is supposed to be and do for you.

Notes for Chapter 2 of the dev-manual:

This chapter has been renamed "Getting Started With the Yocto Project". It is going to be a collection of beginning type procedures. It now has "Setting Up the Development Host to Use the Yocto Project", "Working With Yocto Project Source Files", "Performing a Simple Build", and "Using Pre-Built Binaries and QEMU". This last section likely will be moved. I do not know what the importance is we are placing on using pre-built binaries. We pulled the section from the Quick Start a long time ago and this section was the boiled up version that mirrored it in the dev-manual.

There are two sections that need flushed out that are impacted by the use of CROPS. One, is to set up the system for CROPS, which seems to be the preferred method now. The other is for setting up a native Linux machine, which will cover the distribution compatibilities and the host package stuff.

Notes for Chapter 3 of the dev-manual:

This chapter was mostly reference material. The entire chapter has been converted now to procedures by pulling out the reference stuff and placing it into the ref-manual. What is left in Chapter 3 are procedures related to setting up a shared YP development environment, submitting a defect, and submitting a change.

Section 3.1 - "Open Source Philosophy"

I moved this section to the ref-manual. It now is in a chapter called "The Yocto Project Development Environment". That new chapter in the ref-manual is really the old "Closer Look" chapter now with some YP development environment concepts placed on top in their own subsections.

Section 3.2 - "Using the Yocto Project in a Team Environment"

I have converted this section to a procedure. I need more information on this section from the team. Emails are out soliciting input.

Section 3.3 - "Source Repositories"

This section has been moved to the ref-manual under "The Yocto Project Development Environment" section. No procedures here except perhaps "Locating the Yocto Project Source Repositories".

Section 3.4 - "Terms"

This section has been moved to the ref-manual into the "Introduction" chapter. All links throughout the YP Documentation Manual set have been adjusted to work for the new location.

Section 3.5 - "Licensing"

I moved this section to the ref-manual under "The Yocto Project Development Environment". There are no possibilities of procedures in this section. It is pure reference material.

Section 3.6 - "Git"

The concepts and overview of how Git works is best suited for the ref-manual. I moved the concepts of Git to the ref-manual. In doing so, I decided to keep the two small examples in that section there just for completeness. However, I created a new section in the dev-manual called "Working With Git Repositories" that has pointed procedures that show the reader how to clone poky, check out a branch based on a poky branch name, and check out a branch based on a poky tag. Right now in the "Getting Set Up" section but will likely be moved as the new dev-manual gains more procedures.

Section 3.7 - "Workflows"

I have moved this section to "The Yocto Project Development Environment" section in the ref-manual. The section is concepts only.

Section 3.8 - "Submitting a Defect Against the Yocto Project"

I have reworked this section and it is done. It will live in the dev-manual.

Section 3.9 - "How to Submit a Change"

I have re-worked this section into a couple of procedures.

Notes for Chapter 4 of the dev-manual:

This whole section is about various workflows. The information is a mix of concepts, high-level workflow steps, and various details associated with the flows. The chapter is now gone from the dev-manual. I took all conceptual information out and it has landed in the ref-manual. I took out all stand-alone sections and set them up as individual sections in the "Common Tasks" chapter. I took the workflows and dropped them into the appropriate other development manuals. Devtool, in particular, now lives in the sdk-manual. The sections on devtool between the dev-manual and the sdk-manual were 99.9% identical. This will fix that Henry bug.

Tim's input

First Steps for a Real Project (Intermediate)

The two biggest ideas I find missing in teams new to Yocto Project are:

• Step 1: create a distro layer
  • Add your own branding to it
  • Add your own image recipe and starting adding packages your team needs/wants
    • When in doubt, require another base image recipe and then add to it.
    • Stop adding IMAGE_INSTALL_append to your local.conf, put it in your new image recipe.
• Step 2: create one or more application/functional layers
  • devtool finish all your custom recipes to these layers (unless they are upstream intent)

I honestly recommend this (usually as a course correction) to every single team with which I engage. It is clearly not intuitive.

Layers are like LEGOs(TM)(R)

Keep your "git" working directories clean

• Do not create your new layers as children of poky.
  • Create them as siblings, this way they can work with oe-core only and also anyone else's layers.
  • Ship your layers as git repos, not a zipped poky fork.
• Create your build directories as siblings of poky, not children.
• Do not force users of your init script (e.g. oe-init-build-env) to conform to your concept of the location for their build directory! HUGE PET PEEVE. This makes we want to abandon your layer immediately.
• Do not force users of your distro layer to conform to your concept of what their bblayers.conf should be. Let them put their layers where they want them to be!
• Any layer should be able to be picked up and dropped in line with any other layer and not interfere.
• Do not assume everyone is using poky.

(TM) and (R) indicate trademarks of the LEGO Group