Yocto Project Design and Development Process

Yocto Project is an open source project. It involves not only community developers all over the world but also professional development teams from industrial companies such as Intel, Wind River and many others. It’s different from software projects that are purely organized by public communities, as well as from the ones being developed by professional teams in commercial companies. This has significant impact to the marketing strategy, organizational structure, stakeholder involvement, communications, development process and many other aspects of the project. This also brings unique challenges to us regarding how to balance the open source community culture and the corporate culture. The development methodology we are adopting is largely influenced by the above thoughts and the need to welcome and manage changes in the process. Our goal for the program management is to deliver to the rapidly changing market with professional manner, align with different priorities and schedules of different organizations to benefit all stakeholders.

This document describes our current methodology and processes, which will keep evolving with our own learning, community and management feedback, and the ever changing software industry. We welcome all feedback and comments from our community. Please feel free to send emails with your thoughts to song.liu@intel.com or the Yocto Project mailing list (yocto@yoctoproject.org).

Yocto Project Release Process

Overview

Yocto Project releases on a 6-month cadence. Every 6 months, we have a major release such as 1.x. During each major release, we have milestone releases every 4-8 weeks depending on the specific schedule of each release. The last milestone of a major release is usually a stabilization milestone where we stabilize the code base the major release. Going forward, as we integrate QA process tighter into our development and have more frequent continuous integration, we might eliminate the stabilization milestone. The following figure shows this process cycle:

Release Planning

Release planning starts with the vision and theme of the next release, then selects the feature set to be delivered for the release. Because business and marketing conditions will change and the team will likely learn more about what users really want and what they really need to do as we move into the release development, changes will occur throughout the process. We recognize both the risk of over planning and the risk of chaotic or unmanaged changes. So balancing between planning and flexibility is one of the most important principles we use in all of our processes.

The Goal

The goal of release planning includes:

• Set the theme, plan the release feature set and prioritize each feature
• Discuss and establish process improvement based on retrospectives and feedback from teams and community
• Map out the schedule of the release
  • Commit to priority 1 (P1) features and schedule.
  • Setup goals for P2-P3 features for each milestone.

The priorities of each feature are defined below:

• Priority 1 (P1): Highly important, must have for the release.
• Priority 2 (P2): Important, we will strive to have it delivered for the release.
• Priority 3 (P3): Nice to have, we will deliver if we have time.
• Priority 4 (P4): it’s on our mind, but no specific plan yet.

The Process

The process of release planning involves:

1. Release Planning meetings ( F2F is possible)
   1. Attendees: engineering managers , architects, engineering program manager, business program manager, other major stakeholders
   2. Goal:
      1. Set the theme and vision, go over the feature set and prioritize
      2. Discuss and decide on necessary process improvement
2. Team lead meeting
   1. Attendees: maintainers, QA leads, release engineer, engineering program manager, engineering manager, architects
   2. Goal:
      1. Communicate and clarify release feature set and development processes.
      2. Setup milestone schedule for each milestone.
      3. Determine team ownership of each feature.
      4. Setup goals for team planning.
3. Team planning meetings (each team organize its own meeting):
   1. Attendees: individual team members, maintainers, team leads, engineering program manager
   2. Goals:
      1. Communicate any process changes.
      2. Go over each feature, determine ownership and gather initial estimate.
      3. Setup committed schedule for P1 features.
      4. Setup goals for each milestone for P2 and P3 features (a goal is something the team will strive to achieve, but the team may change plan as necessary at milestone planning between milestones)
4. Overall schedule communication
   1. Attendees: Engineering program manager, engineering manager and all teams and community members if possible. As we have various teams and community members from different GEOs, this may have to be done in several different meetings, through emails or other formats.
   2. Goal:
      1. Communicate the process, overall planning and schedule.
      2. Discover issues and make necessary adjustment.
      3. Clarify any confusion.

Outcome/Deliverables of the Release planning:

1. Prioritized release feature set.
2. Detailed development schedule for each milestone, release engineering schedule and various checkpoints for things such as feature freeze, major kernel version lockup, etc. The schedule will be published on the Google Calendar (https://wiki.yoctoproject.org/wiki/YoctoCalendar).
3. Feature set for each development milestone, published on the wiki.
4. Development process and methodology published on the Wiki

Release Review

The release review could start once we reach feature freeze for the release and overlap with early part of release planning for the next release. The goal is to review major feature deliverables of the release and also learn from our experience in the release.

Process

There are following major activities in the release review:

1. Feature review or demo: selected features will be reviewed and demonstrated to the team, product managers, stakeholders and community. Improvements will be collected as new features for the next release.
2. Release Retrospective: learn from past experience and improve on development, communications, processes, management, etc.
3. Post-release usability review (optional): usability is the key part of a release that drives adoption of the Yocto Project. This review will summarize the current status of the release usability and identify gaps for future releases to address.

Major Release Checkpoints

• Kernel version (major kernel version) and tool chain (including libc’s) major version lock: 6 weeks before release code freeze.
• Final stable kernel update before code freeze.
• Package update ongoing, target set and checked for each milestone.
• Feature complete/freeze before stabilization weeks of the last milestone: after this point, no new features or functionality should be added into the release.
• Documentation Freeze (before last release RC)
• Beta program
  • Collect users: beginning of M(n-1) (M(n) is the last milestone)
  • Start: beginning of M(n)
  • Duration: 2-3 weeks

Release Criteria and Checklist

Release criteria must be met and should be reviewed by stakeholders and the community in the release readiness meeting before release. Some generic criteria include the following:

1. All P1 features and tasks have to be completed.
2. Majority of the P2 and P3 features need to be completed.
3. The release need to meet certain quality standards, for example
   1. Passing full QA test, no high bugs open against the release
   2. Majority of the medium bugs are fixed
   3. Weighted Defect Density reach certain range
   4. Have plans for all known issues
4. The build performance should reach or maintain certain level.
5. We need to reach certain target for package updates.

We set up release criteria for each milestone during the planning week of the milestone. And the release criteria for the last milestone in the release serve as the one for the entire release.

Release checklist:

• All documents have been checked against the latest of the release.
• Release package has been generated and posted on the public website (yoctoproject.org), and pushed to the mirror.
• Release announcement email has been sent to the public mailing list
• A public blog entry has been published on yoctoproject.org

Milestone Releases

Overview

We divide a major release into smaller milestone releases. These milestones serve as checkpoints for our progress, preview for the public community, and especially as iterations for us to manage changes in the middle of a release and continuously improve Yocto product features, development process and practices. At the end of each milestone, we have a milestone release to the public community.

Milestone Weekly Activities

Each milestone lasts about 4-8 weeks and has one planning week, several development weeks, stabilization weeks and one release week:

• Week 1: Milestone planning and development
  • Program Manager: Consider doing team retrospective within each geo team for the past milestone.
  • Product Manager: Reprioritize existing and new features if necessary, add or remove features in the milestone.
  • Program Manager and the Team:
    • Arrange resources to commit to the P1 features scheduled for this milestone
    • Reset the milestone goal for P2, P3 features.
  • Each individual should consider other tasks they have in hands and make sure
    • All P1 features can be delivered in the milestone.
    • Majority of P2/P3 features can be delivered
    • Sets their own weekly goal and try to spread out pull requests for feature check-in among development weeks and avoid stampede of patch review and pull request in the last development week.
  • Individuals: feature development and bug fixing
• Development weeks/check-in window (1 - N-3):
  • Individuals: Feature development and bug fixing, each member should focus on 1-2 features at a time. Update feature status on Bugzilla whiteboard by SOB (Start of Business) every Monday.
  • Release Engineer: nightly build and weekly build
  • QA: sanity test, regression test, weekly test
• Week N-2 – N-1: Stabilization weeks
  • Individuals: Bug fixing (focus on critical and high)
  • Release Engineer: release candidate build every week
  • QA: Full pass testing, distro testing
  • Program Manager: Organize milestone review and/or demo
• Week N: release week
  • Program Manager: Release readiness meeting.
  • Release Engineer: Release the milestone release.

Milestone Planning

During the first week of each milestone, plan for the milestone will be analyzed again and adjusted as needed. In the release planning process, we already set plan for each milestone. But now there might be changes from the project, requirement and also the market. So this is the time and opportunity to manage those changes and adjust the plan accordingly.

In milestone planning, we need to consider the following input:

1. New information or improved understanding about scheduled features n the milestone.
2. New input or feature request from the community and the market.
3. Workload changes of each individual team member such as interruption from other projects or marketing activities, etc.
4. Features we did not complete in the last milestone and features we already started or even completed for the current milestone.
5. Outcome of reviews of the last milestone.
6. Any other changes that might have impact to the current milestone schedule.

Process and roles:

1. Product managers and planners should reprioritize, add or remove milestone features if necessary.
2. Program manager and team leads should organize and help each team with overall milestone planning and individual planning through team planning meetings and 1:1 conversations.
3. Each team member is responsible for thinking through all features scheduled and make sure all P1 features will be completed and most of the P2/P3 features will be completed also.
4. Program manager should consolidate all changes into Bugzilla and update the corresponding wiki pages.

Outcome:

The outcome of the milestone planning should be an adjusted milestone plan to reflect the most current information, changes and decisions.

Milestone Review

At the end of the milestone or during the milestone, the management or the team will select or volunteer to review or demonstrate some features that are completed or partially completed. Ideally, the invite of the review should go to the entire team and even members from the community. The purpose of the review is to:

1. Help communicate and share information among team members so that we all know what and how others are doing and whether someone can help.
2. Assess progress and gather feedback from stakeholders.

These reviews should normally be done when features are completed for the milestone, and feedback and changes will be handled in the next milestone. When it’s really necessary to review features when they are partially completed, there is the risk that we may have to adjust the milestone plan in the middle of the milestone for reviewed features, which should be avoided unless it is really necessary.

Milestone Retrospective

At the end of the milestone, a short retrospective meeting can be organized in each GEO.team by the program manager, team leads or volunteers. During the meeting the team openly talks about things we have done right and things we need to improve There are different formats of the meeting. One of them is to ask each member to anonymously write down top 3 things we have done right and top 3 things we need to improve. Then the meeting organizer will go over each of them and discuss with the team. At the end of the meeting, action items should be recorded with owners to follow up.

Milestone Release Checklist

1. Release tarball generated
2. Announced to the community through email to the public Yocto mailing list and blog.

Development Process & Feature Lifecycle

Overview

We are taking a feature/task driven development approach for the Yocto Project. Feature sets will be planned for each release and also each milestone. But in each milestone each feature goes through its own lifecycle individually. There is no single checkpoint for milestone features to reach certain development stage such as design complete at the same time. With this approach, we hope to have both flexibility and discipline in our development process. Some features involving the community may take unpredictable amount of time to complete certain development stage and they may slip for the milestone. We strive to complete all the features scheduled for each milestone, at the same time we recognize the risk for features involving the community.

Feature Lifecycle

Each feature goes through the following lifecycle:

Specification

Goals: Specify feature requirement mainly from the user’s perspective.

Process: Product managers and planners gather input from users, marketing and the community, create an enhancement/feature bug in Bugzilla to capture the user requirement for a feature. This could happen any time, and specification should evolve over time during the entire release in a managed manner.

Specification Check list:

• Requirement specified clearly and communicated with the source or author of the feature.
• Use case identified if there is any.

Deliverables:

• Enhancement bug filed with requirement specification

Scheduled

Goals: Once a feature is assigned to a milestone, the feature is in the ‘scheduled’ stage. This is to indicate that the feature is ready for development.

Scheduled Check list:

• Specification is done and the feature is scheduled to a milestone.
• First estimation is done, big features are divided into smaller ones.

Deliverables:

• Release/milestone planning meeting.

Design

Goals: understand and clarify user requirement, describe the requirement from the product perspective with required functionality, design user experience, workflow and implementation details.

Process: Once the owner of a feature is ready to start to work on a feature, she or he initiates conversations with the various stakeholders, architect and source of the feature to start the design process.

Design Checklist:

• Requirement is clearly specified on desired functionality and quality from the product perspective. List acceptance criteria if necessary.
• Estimation is done.
• Identify regression test cases that can be run by the developer during development phase.
• Test plan for the feature should be done.
• Design is documented with specific implementation tasks in the Bugzilla comment area or on a wiki page (put the link the to wiki page in Bugzilla).

Deliverables:

• Design description in Bugzilla or prototype code.

Design Review

Goals: Have the design reviewed by the community and architects, address valid feedback, get consensus.

Process: After the design is complete by the owner, she or he sends out the design to the community via the public mailing list (Yocto@yoctoproject.org). If there is no feedback from the mailing list in a week, the owner can go ahead with the implementation. Internally, the owner should drive and work with the architects and maintainers to make sure the design is reviewed and signed off by them. The feature may go back to design phase if there are valid feedback and suggestions to change.

Design Review Checklist:

• Valid feedback from the community has been addressed or no feedback from the community for one week
• The architect and maintainers should sign off the design. The signoff should be recorded as comments of the feature bug in Bugzilla.

Deliverables:

• Design comments

Development

Goals: Implement the feature/task with desired quality.

Process: Code and chat, collaborate with the team and community. The more we share this process, the more likely we can have code with better quality and get quicker pass in patch review. Sometimes, it might be better to share partially completed code with others to get feedback.

Development Checklist:

• Function complete.
• Unit testing complete.
• No regression test failure (It’s the developer’s responsibility to identify regression test that can be run by the developer for the feature in the design phase).

Deliverables:

• Patches

Patch Review

Goals: Have the code reviewed by the community and architects, address valid feedback.

Process: The owner sends out patch to the mailing list after development is complete. The feature may go back to development if there are valid feedbacks or suggestions requiring code change. In some cases, the design may need to be changed and the feature may go back to the drawing board and start from design phase again.

Patch Review Checklist:

• General coding standard or quality checklist should be checked if there is any
• Specific code review check list for individual components should be checked if there is any
• Patch in master

Deliverables:

• Patch comments or code

Merged

Goals: Merge patches into the master branch without breaking the build.

''''Process: Architects pull the patch and merge them into master and make sure the build is not broken with the patches. Once that happens the owner should mark the feature bug as ‘resolved’.

Checklist:

• Commit link to the master is recorded in Bugzilla

Deliverables:

• Patches in the master

Testing

Goals: Ensure that the feature meets functional and performance requirements.

Process: Once the implementation (patches) of a feature is complete and in master or some private branch, QA should initiate execution of test cases for the feature. File bugs and update feature status in Bugzilla.

After testing is completed, if the feature meets requirement it goes to ‘Done’. Otherwise, either the feature itself goes to specification for another cycle or a new feature is created to address new or incomplete requirement not implemented by the last cycle.

Test Checklist:

• Test cases for the new feature (if testable) are fully executed
• Regression test (QA) is done
• Performance test is done
• All found issues are filed as bugs in Bugzilla.
• Meet all acceptance criteria if defined

Deliverables:

• Test Report

Done

Once a feature is tested by QA, the feature bug should be marked as ‘verified’ in Bugzilla by QA. There might be bugs filed against the feature due to problems or feedback. But the feature is considered done at this point.

In some cases, a ‘Done’ feature may not be acceptable to stakeholders and problems maybe found during review or demo. Then new features or bugs should be created to address these concerns depending on the scope of the change.

Feature Status Update

The status owner of the feature is responsible to update the feature bug whiteboard with the current phase of the feature once the feature goes into a different stage. The following lists the status owner of each stage:

1. Specification: architects, engineering manager, business program manager, developers, feature author
2. Design: developer/owner
3. Design review: developer/owner
4. Development: developer/owner
5. Patch Review: developer/owner
6. Merged: developer/owner
7. Testing: QA
8. Done: QA

Format

The status should be updated on the white board of feature bug in Bugzilla. The following format should be used:

(stage) %x completed

For example:

(Developemnt) 80%

Iterations

The development of each feature is an iterative process. In many cases, when a feature is first proposed or even when a feature is fully specified, we don’t know whether the implementation will be what we really want until we actually see and use it. We may also realize or find various issues during each stage of the feature lifecycle. So a feature may go back and forth between stages such as going back to design after patch review, or even go back to specification after the feature is done. Furthermore, we may need to create a new feature to address problems or redesign the existing one.

In the iterative process, getting stakeholder or user feedback as early as possible is very important. With this process, we can accommodate changes better and implement product features that users really want.

Estimation

Estimation for development effort is always hard for most of software projects. There are many pitfalls and uncertainties around estimation. Sometimes, they are so inaccurate that it almost amounts to a waste of time to some team members, especially for new teams or new projects. But as the team norms and the project gets mature, estimation becomes more accurate and important in many ways and it provides value to a team and projects over the long run.

1. Estimation can stimulate the thinking about product features and functionality. Especially when estimation is done in a team environment. Questions from different members of the team can generate more thinking and help the developer comprehend more about the feature in terms of requirement and design. In this sense, the process of estimation is much more important than result of the estimation.
2. Estimation can get more accurate with a stable team and long running project over time. After that it can provide valuable input for release, milestone planning and capacity planning.

In Yocto Project, each feature should have effort estimation done during the following processes:

1. Release planning: In this process, we don’t know much about the design and implementation details of a feature. But a rough estimation will make us start thinking about how much effort it could involve and help with the overall release planning, especially for priority 1 (P1) features. Otherwise, we pretty much have no idea what features can be done at what time.
2. Design & Design Review: When a feature is in design and design review stage, details of the design and implementation will be worked out. At this stage, we can update the feature with a better estimation. This will help us schedule our work and increase predictability of our development process.

There are many ways for a team to do estimation. The best way may not be the latest trend in the industry. Instead, it should always be the most effective way for the team. We will develop our own estimation method as we practice more of this. To start, we use the following practice:

1. We use the number of ‘Perfect Days’ to estimate each feature. A perfect day means a day when an individual can work on the feature without any interruption. The advantage of using perfect day is that it’s more intuitive and developers can always imagine what a perfect day is even though they usually don’t have any.
2. The estimation is mainly to find the relative size of each feature. In another words, what matters is how ‘big’ a feature is compared with other features. And we don’t use the estimation of a single feature to predict when it can be done by a developer. But we would like to use the aggregated result for capacity planning. For example, if a team completed 200 perfect days work in a month, we may reasonably predict that the team will complete comparable amount of work in the next month if nothing else changes. This kind of predication or capacity planning makes more and more sense as the team gets more effective with the process over time.

Documentation

Documentation is one way to communicate and share knowledge among teams and individuals. Documentation is not necessary if there is enough other form of communication going on. For example, in a small collocated team, if members can freely communicate and always stay on the same page, there is rarely any need for documentation. But for teams in different locations and long running projects, documentation is necessary. Yocto Project has teams in 3 different GEOs and developers from the community, some documents such as feature description, design doc, test plans are necessary to share information and knowledge among teams.

Even though documentation is necessary for us, we should always remember that documents (except documents for the community and customers) are not our goal. Our goal is always to deliver the final product. Documentation is just one way for us to share and create information and knowledge. Similar to estimation process, the process of writing the document can always help us distill and organize our thoughts, thus is more important than the document itself in many cases.

At this point, we don’t define a standard set of documents for our development process to allow more flexibility. But over time, the team will settle into a de facto set of documents.

Contributing to the Yocto Project

As an open source project, Yocto Project welcomes anyone to contribute through various ways. An individual can make contributions through testing, bug fixing and helping out on feature development, etc. at any time. If you have a team and would like to help out as a team, then it’s better to coordinate with us up front on things such as work assignment, schedule, processes, etc.

There are various ways of asking questions and finding the information you need:

1. Mailing list: Yocto@yoctoproject.org
2. IRC channel: http://www.yoctoproject.org/community/irc
3. Wiki page: https://wiki.yoctoproject.org
4. Tuesday technical team meeting. This meeting usually happens on 8:00AM PDT every Tuesday. It is open to the public. You can find the meeting information such as agenda and timing in the public mailing (item 1 above) list every week. Please feel free to join the meeting if you have questions that cannot be answered elsewhere.
5. Ask anyone you know on the Yocto Project.

Individual Contribution

If you are new to Yocto Project, there are a few things you can do to get started:

1. Go to: http://www.yoctoproject.org/documentation. There you will find documents to get you started with the Yocto Project. The quick start guide is always a good place to try first. You can also watch a few training videos and that can help you in a more intuitive way.
2. Go to http://git.yoctoproject.org/ to check out the source code and familiarize yourself with the code structure, coding style, etc.
3. Ask questions on the Yocto Project mailing list: yocto@yoctoproject.org, or the IRC channel if you have any (http://www.yoctoproject.org/community/irc)

Once you are ready, you can pretty much contribute to any activities including the following:

1. Find out the project status: The planning wiki page (https://wiki.yoctoproject.org/wiki/Planning) lists release status, feature list for all releases. You should be able to find out where we are for the project in terms of feature development, bug fixing and milestone progress from the status page of each release.
2. New features/ideas: If you have any idea or new feature in mind for the project, you can always put it in Yocto Project Bugzilla (https://bugzilla.yoctoproject.org/) as enhancement bugs. That’s where we keep all the features for every release. Before putting in any new features, you may want to search if there is any similar feature that’s already there. We usually plan features for the next release during release planning, but there is nothing prevent us from working on it during the current release if the feature is really more valuable than all others or if you can actually help get it done.
3. Feature development: If you would like to help with feature development, please check Yocto Project Bugzilla (https://bugzilla.yoctoproject.org/) for ‘Enhancement’ (severity) bugs. We track all new features in Bugzilla with these ‘Enhancement’ bugs. If you see something that is interesting, please feel free to contact the owner and see how you can help there.
4. Bug fixing: Again, go to Yocto Project Bugzilla, check out the open bugs and contact the owner if you are interested in helping on any of them. I’m sure the owner will appreciate your help very much since we are always short of time.
5. Package update: Please see this wiki page (https://wiki.yoctoproject.org/wiki/Distro_Tracking) for more information and contact the maintainer of the package if you would like to help updating one or more of the packages.
6. BSP development: if you would like to create a BSP for some target device, here is some document that may help you: http://www.yoctoproject.org/docs/current/bsp-guide/bsp-guide.html. During development if you have questions, please feel free to ask via the the Yocto Project mailing list: yocto@yoctoproject.org, or the IRC channel (http://www.yoctoproject.org/community/irc). Please also refer to https://wiki.yoctoproject.org/wiki/Third_Party_BSP_Release_Process if you would like to have your BSP released through the Yocto Project.
7. QA: Most of the QA related information can be found here: https://wiki.yoctoproject.org/wiki/QA. If you would like to help on testing the system, some features, or testing Yocto on different Linux distributions, please feel free to contact our program manager or our QA team.

Team Contribution

If you have a team from your organization and would like to contribute to the project, that’s even better. In many cases, you may have a specific set of things you would like your team to work on or you have your own release schedule that could be different from the Yocto Project release schedule for your Yocto Project related product. In this case, it’s better to have some plan time before we start working together. Please feel free to contact our program manager or come to our Tuesday technical team meeting (you should be able to see the meeting information every week on the Yocto@yoctoproject.org mailing list). Once we establish the contact, then a detailed plan can be worked out to align your goal and schedule with the Yocto Project.

Other Processes

Point Release Process

A point release is a minor release such as 1.0.x. Point releases are necessary to address the following:

1. Security updates
2. Issues building on current Linux distributions.
3. Issues that existed in previous release and have significant impact to users.

Maintainers

The role of a point release maintainer for Yocto Project will be rotated among team members and community volunteers. The maintainer is responsible for branching out and pulling patches for the point release. Decisions regarding the content and planning of the point release should be worked out with architects and the program manager. Any related questions and requests about the point release should be sent to maintainers.

Process

1. Maintainers send request for patch recommendations to the public mailing list.
2. Review release notes and QA reports of previous point releases and the major release, make sure we integrate patches to fix significant known issues in previous releases.
3. Branching out:
   1. Create a branch for the point release
   2. Review, integrate selected patches and do necessary testing
4. Build:
   1. Merge the branch with Poky release branch for the major release. For example, point release 1.0.2 branch should merge with 1.0 release branch
   2. Rebuild BSPs including core BSPs and some other necessary ones and run these BSPs through full pass QA test. BSPs should be re-built with both the recipes and meta data of the current point release and the related major release against the point release. This is to ensure BSP compatibility where BSPs that can be built with the major release can also be build with the point release without any modification.
5. QA: go through build, weekly QA or full-pass QA test if necessary. Multiple Release Candidate builds should be planned if the changes in the point release are significant.
6. Release: generate and publish the release package, release note.
7. Announcement: announcing the release in the public mailing list. Blog the release if there is any significant impact to the community.