Yocto Architecture

Yocto Project Architecture Whitepaper

Embedded Market

The embedded market is characterised by its wide range of devices which operate in many different environments e.g. home users, industrial devices, medical use or data centres. These devices are tailored to their environments and vastly differ in connectivity capabilities, input and output methods and features like processing power, memory capacity and available data storage. These systems need software to function and Linux is desirable due to its adaptability, its functionality, quality and no royalties model.

The market is highly fragmented with many different approaches to software being adopted using both open source and proprietary solutions. Each market sub-segment tends to have a preferred approach but there are no accepted standards.

Anyone working in this market faced some difficult issues trying to deal with this fragmentation. One approach would be to take each sub-segment in turn and try and create solutions targeting that segment. This would be costly in resources. An alternative is that offered by Yocto Linux - to create a common best practises infrastructure which can fulfill the software needs of most embedded market segments.

Building a common base set of tools and a reference embedded Linux stack means this non-differentiation work which everyone currently does themselves can be shared in the form of a common project. Organisations can then focus where their effort is best spent, on their differentiation and project features.

Yocto Project

The objective is to look at all the solutions currently in the embedded space and then take the best of what is available, fill in any missing functionality and create something which can lead the market.

By doing this, the project users share the cost of the base stack and there is a common standard set of tools used by the marketplace rather than everyone having to do this themselves.

Existing Build System Solutions

Some kind of build system is the cornerstone of most embedded projects and the Yocto Project is no different. There are many such solutions in the embedded space. In order to capture the attention of the industry this project requires something open that people can see and use without royalties, much like Linux itself. For this reason only open source systems are contenders for this project.

There are a variety of possible existing contenders and its worth evaluating these for their strengths and weaknesses:

Buildroot - One of the older embedded build systems, this is makefile driven and is a natural evolution of the scripts based build solutions many people start with when starting an embedded project. The general feeling of this system is that it has had its day, it was a good learning experience but lacked the fundamental structure to its processes to grow into anything bigger than what it is. It lacks fundamental features such as packaging.

E2-Factory - A more modern from scratch solution that it very structured and rigorous in its build processes. Sadly whilst the core tool is open source, no metadata is available so its ruled out. It does have some neat features like dependency checksumming which can be learnt from though.

Desktop (PC) Distribution Systems - Whilst not embedded solutions, these are widely used for PCs so should be considered. The fundamental problem is that these systems all promote a "one size fits all" approach to software and do not support customisation for multiple targets easily or well. You cannot build two distributions of different character from one set of metadata with these solutions. In the embedded space this rules the solution out but again, there are things that can be learnt from like the OBS web interfaces.

PTXdist - Makefile driven, more advanced than buildroot but is small and only caters for a small segment of the market.

LTIB - This is a system developed and pushed by Freescale which combines spec files with a kconfig menu front end. Its nice to have the menu control of the system but it has limited package support, limited platform support and is very tied to the rpm format and is not as structured, mature or comprehensive as other solutions in the marketplace.

OpenEmbedded - If there were an industry standard, this would be it. It was written to learn from all the mistakes in buildroot, has a well defined structure to its metadata and has by far the largest coverage of open source software. It dropped Makefiles in favour of a custom data file format of its own with tools to handle them (bitbake). Feature wise its also the most advanced system in the market and it was specifically designed for the customisation needs of the embedded market space. It does lack some features such as checksumming but the structure would allow them to be added.

Poky - Created by Openedhand as a commercial version of OpenEmbedded and the rights to it passed to Intel. It has a good reputation in the market as lots of technical developments in OpenEmbedded have come via Poky.

LDAT - The build system developed and used by Wind River. This is not an open solution at this time but could become open so is worth evaluating. Its strength is that it has a large feature set and is widely used by Wind River customers. Its makefile derived and shows signs of strain due to being constrained by the file format. It has been developed behind closed doors with a narrow focus and as such does not stand a good chance of wide community adoption if opened and hence is unlikely to become an industry standard.

Yocto Linux build system

Acceptance of whichever build system is used by the wider community is essential to the project. We're in the fortunate position that Poky is an available option. The advantages of this choice include:

• Its already well established and accepted in the community
• It has commercial leanings
• Intel can donate Poky to the Yocto Project
• Its based on OpenEmbedded - industry leading

On this basis it has been decided that Poky makes the best choice of build system for the Yocto Project.

Where there are gaps in functionality we have the ability to change this.

Yocto Project Components

If Poky is the build system for the Yocto Project, what other components are there? Should the Yocto Project perhaps be Poky?

There is much more to the idea of the Yocto Project than just the build system. It makes sense to have namespaces for the components which is why Poky should remain just the build system. The other features/components that make up the Yocto Project include:

• A place of collaboration - There are very effective strong communities in the opensource world such as xorg.org, kernel.org and gnome.org but there is no such place for embedded. The Yocto Projectcan provide this, primarily by providing a place where embedded focused software projects can be housed and discussions can be had.
• A reference implementation: Yocto Linux - Any reference distribution/images created by Poky are something that is provided by the Yocto Project . This aims to reinforce the idea that Poky is the build system, not the end result of any given build of the metadata which instead would be Yocto Linux.
• Board Support Packages - The Yocto Project can provide a place for people to publish its SoC BSPs.
• Testing infrastructure - The Yocto Project can provide hardware to allow regular regression testing of the Poky metadata and demonstrate this publicly.
• Partners - The Yocto Project also provides the entity which other parties can associate with to jointly improve and advanced the embedded Linux software position.
• Standards - The Yocto Project can become the entity which sets the standards for the industry, aiming to be the model of best practises.
• Documentation - Another output possible from the build infrastructure is documentation of all the components where the source contains these. As part of the reference implementation it would be desirable and extremely useful for the community to have access to a set of documentation all in one place.

More details on the individual components follows below.

Yocto Project Partners

To succeed, Yocto Project needs to ensure people will use it and one way of doing this is through partners.

Discussions between Wind River and Intel revealed that they share frustration over the structure of the Embedded Linux market as mentioned above. The fragmented nature of the market has a cost to them and also presents a barrier to entry and their build system LDAT is different to any other on the market. All their BSPs effectively have to be developed and maintained in-house at significant cost.

Wind River see the Yocto Project as very desirable as it means everyone would share the non-differentiation and become more standardised. Creating a BSP standard reduces the amount of work they have to do in integrating BSPs themselves. For these reasons Wind River are keen to be a founding partner of Yocto Project. Where the Yocto Project solutions lack the features they need for this there is agreement to enhance it and its components like Poky.

As the Yocto Project grows and becomes public, its likely more partners will emerge and this will make the project stronger.

Yocto Project Components: A place of collaboration

The Yocto Project website is intended to house embedded related software projects since there currently isn't a good home for a lot of these. We already have a number of different projects which could be used to seed this site:

• Poky (from Intel)
• Poky Anjuta SDK integration (from Intel)
• MeeGo Eclipse SDK integration (from MeeGo)
• Pseudo - A replacement for fakeroot (from Wind River)
• qemuGL - A project to allow GL passthrough in virtualsation (from Intel)
• OProfileUI - an embedded targeted client/server profiling tool (from Intel)
• Cross platform prelink tools (from Wind River)

Anyone wishing to house the source code/mailing list/bug-tracking for an open source project furthering embedded technology would be free to do so on this site.

Yocto Project Components: Reference Implementation

A reference implementation will be available from the Yocto Project website under the name of Yocto Linux. These will be packages, images and an SDK/toolchain generated by Poky supporting the range of reference hardware we're promoting. For IA, this means we'd have images capable of being run under emulation in QEMU as well as having support for a suitable selection of our reference boards, particularly the SoCs chosen from our BSP selection.

For other architectures we will provide reference images suitable for running under emulation as proof we are architecture independent but Intel will not provide builds for other specific platforms. If other partners join the project and undertake to maintain platforms on other architectures they are free to do so however.

It is worth highlighting that the reference distribution and images created by Poky are something that is provided by the Yocto Project. We should reinforce the idea that Poky is the build system, not the end result of any given build of the metadata. The images and packages on the Yocto Project website are also a reference implementation which others are then free to build off.

At any given time there is likely to be a stable release version of Yocto Linux available and also packages and images build from the master branch as part of automated testing. These are provided to demonstrate the status of the project, that everything is working and allow people to test new features. Its expected that product development be done based on stable release branches and if cutting edge features are required, these will be ported to the last stable release either by Yocto Linux maintainers if there is a widespread need, or on an individual basis if not.

The benefit of having this reference implementation available is that anyone wishing to do embedded development has a known good place to start it from. By sharing this common base, solutions to any issues can be shared, faster development and time to market is possible, a strong community is formed and there is a common future forward migration path.

Yocto Project Components: Board Support Packages (BSPs)

Currently there is no standardised way of publishing a BSP for a given platform either in the location its made public or the format. It is a massive problem in the embedded marketplace and nobody has as yet found a way to deal with this well. ARM is a particularly bad example with hundreds of different kernel trees, let alone userspaces. A given ARM platform is unlikely to work well with a standard mainline Linux kernel and whilst effort is being made to change this, it has not yet succeeded.

The ideal situation would be having one central place where up to date code for a given platform can be found, both in the form of a kernel but also the support utilities and extras that platforms have. This needs to provide complete support for that platform including firmware and other userspace components that might be needed to make best use of the hardware features.

The Yocto Project proivdes a standardised place and standardised format to do this. The exact format will be determined by the needs to the platforms but will be generic enough that an industry BSP standard could be established as a result.

Yocto Project Components: Testing Infrastructure

One area many embedded build systems fall down is that their core developers can make them work but a newcomer to the project cannot. These systems by their nature are extremely complex and its essential they are regularly and extensively tested. There are two forms of testing that are desirable:

Incremental - This provides an "acid test" of the last change made to the metadata. Its designed to run relatively quickly and test the change but not exhaustively or from scratch. If the tests succeed, its expected that there is a 95% chance the change did not cause any regressions. It also checks this change builds against an existing build. The developer can expect a warning about problems with a change quickly from these tests.

Full - This is a test of the metadata as a whole from the ground up and ensures that the system is self consistent. Its designed to prove there was no regression but obviously the tests take longer to run.

Its expected that incremental tests will happen on every commit to the metadata and that full testing will happen once every 24 hours at a minimum. Its also desirable that in future, separate branches can be tested outside of master and only merged into the main branch when a test suite completes.

By doing this publicly, we're forced to keep the metadata buildable, consistent and people can see that we do this and care about the quality of what we're doing.

Eventually it would be desirable to also test the build output of the system either under emulation or on the real hardware in the form of "board labs". There is an significant capital expense of setting these up as well as a current lack of software to support their use. It would however fit well in the Yocto Project testing infrastructure if we can reach a point where this is practical.

Yocto Project Components: Partners

Partners are an important part of the Yocto Project as we want to defragment the embedded market and we can't do this if we don't build relationships with others. Its one of the key areas many other efforts in this marketplace fail as they tend not to cover the needs of the majority of the community.

The main criteria for becoming a supporting partner will be agreeing to standardise working practices in line with the Yocto Project's objectives and most likely starting to use it as a standard. This may also involve offering support and development of its tools and infrastructure.

In return partners benefit from the defragmented embedded market and jointly improve and advanced the embedded Linux software position.

Yocto Project Components: Documentation

Whilst there is a wealth of information online about given projects, nobody has managed to effectively document a complete Linux software stack. The reasons is related to the size of the undertaking and the effort required to maintain it.

Through the Yocto Project build infrastructure its possible that documentation generation could be largely automated. We'd have to find a way to bind all the separate pieces of documentation together but I think this is solvable problem.

This coherent and up to date source of documentation would be another feature of the project provided to the community as part of our reference implementation. There would be a documentation collection corresponding to each release as well as a version corresponding to the latest master branch.

An added benefit of this would be that individual software projects would be encouraged to put their documentation into order if they could see a high profile consumer of it which is inline with the Yocto Project's objectives of standardisation and improving the embedded ecosystem.

Build System - More details

The structure of OpenEmbedded and Poky are specifically tailored to the needs of the embedded market space. One of the main ways this happens is the powerful way the metadata is split into separate configuration areas:

“Recipe” - details of a software package

The recipe represents individual software components of the system, usually corresponding to a project which then goes on to make a release tarball. The recipe contains information on where to get this source code, how to extract it, information on any tweaks it may need for known issues (patches), what configuration options this software has and how to apply them, how to compile it and how to extract and split the output into packages.

There can be one definition of a software project which is them shared by definitions for specific version releases of the software.

“Distro” - policy of the resulting system

The distribution configuration is concerned with policy decisions such as which ABI to use, which versions of software should be preferred and is a representation of choices available and the values selected.

“Machine” - details of a specific hardware target

The machine entities are information about particular hardware targets. They can be more general such as "netbook" or provide information about specific hardware like a Sharp Zaurus handheld SL-C3000 (spitz). These given information about the capabilities of the hardware such as the presence of a keyboard, touchscreens, displays, audio capabilities, I/O interfaces and so on. Also details about the best formats to provide images for the device can be contained here.

This information is used by the system to customise output to work well on specific hardware.

“Image” - groups of software packages

These are package groups which provide a working end result. "Sato" images contain all the functionality required to boot the Sato UI for example. A console image contains enough to boot a system to a console shell prompt. The contents of an image will depend on information from both the distribution and machine configurations.

Each of these functionality areas are independent of the others and one can be changed or customised separate from the others. Bitbake through the structure of the metadata is able to combine the information from each section to provide a specifically tailored end result.

The second powerful feature of the metadata structure is the overlay or layer capability. By combining several layers of metadata, a basic set of functionality can be extended or customised at multiple levels allowing a basic core to be extended and customised for different uses.

For example we could have the following metadata stack:

• Core Poky Metadata
• MeeGo specific Overlay (adding the MeeGo UI software packages)
• Vendor specific BSP (adding support for a particular hardware platform)
• Commercial Overlay (provided by say Wind River adding commercialisation to the underlying layers)
• Customer Specific Overlay (adding customisations only relevant to the customer)

At each level, features can be added and underlying layers can be customised without duplicating code. The changes anyone makes also remain clearly isolated from the underlying system meaning they're clearly identifiable and can easily be ported to newer versions of other layers such as the core metadata.

This structure encourages reuse of code and sharing and collaboration of metadata. An objective is to make the core as useful as possible so over time code initially developed as upper layers will tend to head towards the core where it makes sense. There will be functionality blocks such as "meego" which make sense as an overlay in their own right and customisations which will only ever apply to an end user's application too and this structure supports them all well.

Build System - Configurability

Due to its modular nature, OE and Poky are very much infrastructures rather than monolithic projects. Any one component is usually isolated from the rest of the system meaning that if some area is found to need improvement, this can happen and the technology can move forwards without impacting other areas of the system. Pretty much everything is configurable.

Implicit assumptions in the system have been carefully avoided. Issues such as output package format(s) or output image format(s) are abstracted such that multiple outputs are available and future developments can be added too, very easily. This makes the system very adaptable to almost any situation at a relatively low cost compared to other systems.

Build System - Cross compiling and multi architecture

Firstly, it needs to be realised that the build system used has to be a multi architecture system. If it is specific to a single platform such as IA, it will automatically be ruled out of the embedded market and the community focus which we'd like to see will not happen. It therefore makes sense to have a multi architecture system.

Regardless of architecture issues, for embedded systems its always likely there will be some differences between the build system's configuration and that of the target systems. This may be different configuration options, compiler flags, package versions and many other possible differences, it's impractical to even list them all.

As such some way of isolating the target system from the build system is essential. The ways of doing this have been a topic of discussion for many years and will continue to be. Various approaches such as chroots, emulated environments, bare cross compilation and cross compilation using sysroots are all possibilities.

In the general multi architecture case the build system is most likely different from the target architecture. This can apply on IA with a 64 bit build system building for a 32 bit target and vice versa. Since emulation and chroot aren't always possible or reliable, OE and Poky are a cross compiling build system making the best use of new cross compilation technology such as sysroots as it becomes available.

In the embedded space, cross compilation is considered standard practise and widely accepted. The difficulties of cross compiling are also continually being reduced with new technology.

Build System - Any downsides?

The biggest downside of OE/Poky by far is the shear complexity of the inner workings of the systems. They have been designed to be powerful and comprehensive which means there is a steep learning curve to a true deep level understanding of them. There are also several concepts which are different to a traditional 'desktop' Linux system.

Poky has been at the leading edge of making the systems easier to use and is designed such that anyone with some basic Linux knowledge should be able to download it and build a Linux system from scratch. It also has a comprehensive manual covering many of the questions a new user to the system has. As always there is room for improvement but there is a strong foundation to build on.

Build System - Vendor Use Cases and Development Process

The end result most vendors require are software "images" which then can install onto the end products. In some cases these will be field upgradable either through placing a new image onto the device in totality or in some cases though package upgrades in a more traditional PC Linux distribution style. In order to reach this goal there are several stages of development and the Yocto Project infrastructure needs to support them through them all:

Prototyping - The end hardware is probably not available yet but a quick

mockup needs to be possible. This may be using CRBs, possibly customised to demonstrate key hardware features of the end device or it could be done using virtualisation.

Hardware Development - The system needs to allow for testing of new hardware as its developed.

Software Development - The system needs to allow development of the software

that will be used on the final device, often before the final hardware is available. Some developers will be interested in the broad system and how all the pieces build and fit together, others could just be application developers interested in developing a single application. Both sets of developers must be supported.

Poky itself fulfils some of these needs but not all of them. There are some cases having the full build system available is overkill for a specific developers needs. The counterpart is the SDK.

Software Development Kit (SDK)

The SDK is a standalone output from Poky which consists of a sysroot representing a target system and some tools which allow the developer to build further software to work with that sysroot. Since we have little control over the system the developer will be using the SDK on, a standalone cross compiler and cross build tools are provided with the SDK to isolate the two systems in the same way Poky itself does.

Using this toolchain the developer can build applications and libraries that will work with their target platform. Whilst usable from a shell, the SDKs are also capable of being used from within IDEs such as Eclipse and Anjuta at the preference of the developer.

By having the SDK it further reduces the barriers to entry of developing applications for embedded systems since the developer does not have to use the build system itself. There is however a migration path should the developer require to become more involved with the build system itself.

Current Status and Areas for Development

The Yocto Project itself does not currently exist and needs to be established as detailed above. OE and Poky do exist and are powerful and already extensively developed but there are some gaps in functionality which there are some gains by addressing:

• By filling functionality gaps it promotes OE/Poky/Yocto Linux as the standard in the marketplace
• By providing this technology first in Poky, it promotes Poky and Yocto Linux
• Some functionality is required to enable adoption of commercial solutions vendors such as Wind River where features they currently enjoy in systems like LDAT are unavailable in Poky.

The Yocto Project itself goes a long way to solving a number of problems by pulling pieces like the various SDK plugins and other technologies into one place which can then be consistently documented. There is also a list of features specifically being worked on for Poky and Bitbake.

Yocto Linux Development Team Structure

To fulfil its objectives we anticipate the need for 3 different teams of people working on the project:

Embedded Core Team

This team provides the core guidance on the project and consists of the architect and technical direction, programme management, some developers working on the overall picture of the project along with people responsible for overall documentation and QA.

Embedded Tools and SDK Team

This team is responsible for improvements to the tools and infrastructure. They would be implementing features like the server mode of bitbake and its integration to something with similarities to Buildbot, the Eclipse and Anjuta plugins and SoC support through the Yocto Linux SDK

Distribution Engineering

This team is much more focused on the day to day maintenance of the metadata, upgrading versions when required to keep Poky at the forefront of technology, watching for security updates, integrating any relevant technology and managing metadata releases and QA.

The Yocto Project team does not directly cover development of BSPs and SoC drivers. The teams will advise on integration of code into Yocto Linux in parallel with its submission of upstream and assist with any issues that arise however.

Author: Richard Purdie, November 2009 (names have been adapted in the document subsequently for clarity)