Development Manual Rework

Intro

This page will propose an outline for Development Manual refactoring work. See also bug #11630

Henry's Input

I believe the manual lost its way as it became the dumping ground for useful (and often advanced) info that had nowhere else to land. As a result the size of the manual and top level topics became overwhelming to the new developer who had just graduated from the quick state guide. Currently this user is not served by the Yocto documentation, thus the creation of the wiki cookbook. Simple tasks such as adding an existing recipe to image and creating a recipe source should be the bread and butter of the development manual. If this approach makes the development master server too many master, we should dicuss re-structing all the docs the addition of a new one.

Basic "How To" Topics

I plan to structure these, but just a plain list for the time being. Many of these topics are already documented, but this feels like a better flow that we have just now.

Basic Skills for Systems Developers

• Deal with corporate proxies.
• Add a package to an image
• Understand the difference between a recipe and package
• Build a package by itself and why that's useful
• Find out what packages are created by a recipe
• Find out what files are in a package
• Find out what files are in an image
• Add an ssh server to an image (enable transferring of files to target)
• Anatomy of a recipe
• How to find recipes (layers.openembedded,org)
• Understand difference between machine and distro settings
• How to find and use the right BSP (machine) for your hardware
• Examples of distro features and where to set them.
• Understanding the task pipeline and executing individual tasks
• Understand devtool and how it simplifies your workflow
• Improve build speeds with shared downloads and shared state cache
• Generate and understand dependency graph
• Generate and understand bitbake environment
• Building an Extensible SDK

Basic Skills for Application Developers

• Understand intent and content of extensible SDK (eSDK) and why traditional SDK is no longer necessary
• Understand CROPS eSDK container solution
  • Source code is edited outside of container using tools of your choice on host
• Install and run CROPS/eSDK
• Intro to devtool
• Make sure you're up to date
• Build the image (may not be the same as on your target) and typical options for flashing it
• Modify an existing recipe
  • Understand how git is managing source code
  • Build and deploy modified recipe
  • Using devtool finish to create updated recipe and patches.
  • Understand how this is done without updating upstream source code
• Add a new recipe
  • Create recipe from source code
  • Review recipe to see what's missing
  • Repeat steps as for 'modify'

Tracey's Input

The way Henry described this yesterday, customers use the "getting started" guide to learn how to bring up a vanilla image, but then don't know how to approach their own custom situation. This should be a TRANSITION DOCUMENT - how to use YP once you have gotten your feet wet in a controlled situation. Feel free to delete this, I don't know how you folks work in the WIKI and I'm assuming this may not be it.

I would suggest organizing this in a couple of ways - and this might be better in a google doc, frankly. 1) by what people need to do to get started

  a) where to start to support a platform
     1. find a BSP and test it
  b) how to add support for platform components
  c) how to slim down an image size
     1. Remove code automatically included
     2. design your own BSP fine tuned to exclude extra code, unneeded support code
  d) How to approach setting up a development environment
  e) How use layers to create a flexible long term design, and recipes
  d) How do you start a new project vs build off an existing one   
  e) How do you set up a team sharable development environment (sstate)
  f) Here are some of the most common commands 
  g) How to deal with CVEs and code branches

2) By helping them collect the information they need in order to make set up decisions Brent had a great idea to make this a SW Wizard that would step a new user through a series of questions and give them at least one solution on how to proceed.

  a) what platform are you using
  b) will you include an application as part of the image or will it run on the image
  c) what is the configuration of the team (system developers, app developers, etc)
  c) How to help them choose the right kernel
  d) etc.

Scott's Input

Scott Rifenbark - This is a good idea. I was thinking of a similar way to deal with this with sort of a decision tree at the end of the YP Quick Start (flowchart for old-schoolers) that would lead the reader down a path based on information they have about what they want to get accomplished. Then, based on how they traverse the tree, we point them to areas they need to use in the manual.

Notes for Chapter 1 of the dev-manual:

I re-wrote this introductory section to work more tightly with the introductory material of the ref-manual. Also, created a better "Other Information" section to lead the reader to various places of interest. In particular, I am leveraging off the Yocto Project Backgrounders that exist on the YP Website. These backgrounders are probably the best place for an overall introduction to what the YP is supposed to be and do for you.

Notes for Chapter 2 of the dev-manual:

This chapter has been renamed "Getting Started With the Yocto Project". It is going to be a collection of beginning type procedures. It now has "Setting Up the Development Host to Use the Yocto Project", "Working With Yocto Project Source Files", "Performing a Simple Build", and "Using Pre-Built Binaries and QEMU". This last section likely will be moved. I do not know what the importance is we are placing on using pre-built binaries. We pulled the section from the Quick Start a long time ago and this section was the boiled up version that mirrored it in the dev-manual.

There are two sections that need flushed out that are impacted by the use of CROPS. One, is to set up the system for CROPS, which seems to be the preferred method now. The other is for setting up a native Linux machine, which will cover the distribution compatibilities and the host package stuff.

Notes for Chapter 3 of the dev-manual:

This chapter was mostly reference material. The entire chapter has been converted now to procedures by pulling out the reference stuff and placing it into the ref-manual. What is left in Chapter 3 are procedures related to setting up a shared YP development environment, submitting a defect, and submitting a change.

Section 3.1 - "Open Source Philosophy"

I moved this section to the ref-manual. It now is in a chapter called "The Yocto Project Development Environment". That new chapter in the ref-manual is really the old "Closer Look" chapter now with some YP development environment concepts placed on top in their own subsections.

Section 3.2 - "Using the Yocto Project in a Team Environment"

I have converted this section to a procedure. I need more information on this section from the team. Emails are out soliciting input.

Section 3.3 - "Source Repositories"

This section has been moved to the ref-manual under "The Yocto Project Development Environment" section. No procedures here except perhaps "Locating the Yocto Project Source Repositories".

Section 3.4 - "Terms"

This section has been moved to the ref-manual into the "Introduction" chapter. All links throughout the YP Documentation Manual set have been adjusted to work for the new location.

Section 3.5 - "Licensing"

I moved this section to the ref-manual under "The Yocto Project Development Environment". There are no possibilities of procedures in this section. It is pure reference material.

Section 3.6 - "Git"

The concepts and overview of how Git works is best suited for the ref-manual. I moved the concepts of Git to the ref-manual. In doing so, I decided to keep the two small examples in that section there just for completeness. However, I created a new section in the dev-manual called "Working With Git Repositories" that has pointed procedures that show the reader how to clone poky, check out a branch based on a poky branch name, and check out a branch based on a poky tag. Right now in the "Getting Set Up" section but will likely be moved as the new dev-manual gains more procedures.

Section 3.7 - "Workflows"

I have moved this section to "The Yocto Project Development Environment" section in the ref-manual. The section is concepts only.

Section 3.8 - "Submitting a Defect Against the Yocto Project"

I have reworked this section and it is done. It will live in the dev-manual.

Section 3.9 - "How to Submit a Change"

I have re-worked this section into a couple of procedures.

Notes for Chapter 4 of the dev-manual:

This whole section is about various workflows. The information is a mix of concepts, high-level workflow steps, and various details associated with the flows. I think the entire chapter can pretty much be recast in some conceptual area of the ref-manual. Then, for each type of workflow, we can create a specific task that has some meaningful steps and real examples for use in the dev-manual. These sections in the dev-manual can be referenced from the associated concepts in the ref-manual.

Section 4.1 - "System Development Workflow"

Introductory material that will live with the subsections for concepts (in the ref-manual) or introductory area for grouped tasks in the dev-manual.

Section 4.1.1 - "Developing a Board Support Package (BSP)"

Although the steps in this section are technically procedural, the level is more conceptual. We should move this discussion to the ref-manual and group it under some sort of new "concepts" heading where conceptual information is discussed. That is one idea. Another, is to create a conceptual section in the BSP Guide to hold this information. One thing we have not discussed is the fate of the BSP Guide in regards to the restructuring of the dev-manual to a task-based document. In any case, the current "Developing a Board Support Package (BSP)" section in the dev-manual is out as it stands now. If we want to replace it in the dev-manual with something more concrete, then I suggest a real example that would be in the form of what is on this dated wiki page - https://wiki.yoctoproject.org/wiki/Transcript:_creating_one_generic_Atom_BSP_from_another.

Section 4.1.2 - "Modifying the Kernel"

The information in this section is very conceptual and does not belong in the dev-manual. Solution for this section is similar to what I suggested for the "Developing a Board Support Package (BSP)" section above. However, this is kernel related so perhaps moving this kernel information to the kernel-dev manual. Again, if we want to have some concrete example in the dev-manual that is based on the conceptual information here, then we create a real example and use that and tie it to specific tasks.

Section 4.2 - "Application Development Workflow using and SDK"

This section can go away I think. We have the SDK manual and all it is here in the sdk-manual is a pointer to that manual.

Section 4.3 - "Modifying Source Code"

This section is conceptual by nature. Again, high-level procedures to support the concepts but I don't think they should be in the dev-manual. Both devtool and Quilt are covered in this section, and both regarding workflows designed to help the user modify source code. The actual need for this introductory stuff in 4.3 depends on really where the devtool and Quilt sections ultimately reside.

Section 4.3.1 - "Using devtool in Your Workflow"

These concepts belong in the ref-manual. Right now, this material is virtually duplicated in the sdk-manual. It should be one area and then any SDK-specific stuff can be captured in the sdk-manual. The section to put this in in the ref-manual would be under a "Concepts" heading and then maybe under something like "Workflows".

Section 4.3.2 - "Using Quilt in Your Workflow"

These concepts belong in the ref-manual in a new "Workflows" section bundled under some concepts heading. I can see creating an actual example that works and using it as a section in the dev-manual. The conceptual version from the ref-manual can reference into that example.

Section 4.3.3 - "Finding Temporary Source Code"

The reason this section is where it is was to support the Quilt workflow. If "Finding your Source Code" is a separate enough task to be in the dev-manual, then we can create a set of steps to walk a user through that give creation of some piece of software and maybe a kernel build. They seem to be different based on what you are building unless I am mis-understanding it.

Section 4.4 - "Image Development Using Toaster"

The reason this section exists is it was a development workflow and bundled in this overarching section. It is pretty much a placeholder and could be removed. There are ample areas where we can mention Toaster as a way of developing an image and then linking to the Toaster manual. If we have a workflow concepts section in the ref-manual, it can be mentioned there as a type of workflow.

Section 4.5 - "Using a Development Shell"

This is another development workflow. We could have a conceptual description in the appropriate area of the ref-manual that would point back to a specific example that is task-based in the dev-manual.

Section 4.6 - "Using a Development Python Shell"

This is the final development workflow section. Same stuff applies here as explained in the "Using a Development Shell" section.