Planning: Difference between revisions

From Yocto Project
Jump to navigationJump to search
No edit summary
(Replaced content with '* Yocto Features * Yocto Architecture')
Line 1: Line 1:
* [[Yocto Features]]
* [[Yocto Features]]
* [[Yocto Architecture]]
* [[Yocto Architecture]]
= Yocto Linux Architecture Whitepaper =
== Embedded Market ==
The embedded market is characterised by its wide range of devices which
operate in many different environments e.g. home users, industrial devices,
medical use or data centres. These devices are tailored to their environments
and vastly differ in connectivity capabilities, input and output methods and
features like processing power, memory capacity and available data storage.
These systems need software to function and Linux is desirable due to its
adaptability, its functionality, quality and no royalties model.
The market is highly fragmented with many different approaches to software
being adopted using both open source and proprietary solutions. Each market
sub-segment tends to have a preferred approach but there are no accepted
standards.
As Intel develops SoC solutions targeted at the embedded world it will enter
this fragmented market space and needs to find a way to approach this
fragmentation. One approach would be to take each sub-segment in turn and try
and create solutions targeting that segment. This would be costly in resources
and is not the approach Intel is used to in the PC world where there are only
a small number of targets. An alternative is that offered by Yocto Linux - to create
a common best practises infrastructure which can fulfil the software needs of
most embedded market segments.
== Yocto Linux ==
The objective is to look at all the solutions currently in the embedded space
and then take the best of what is available, fill in any missing functionality
and create something which can lead the market. Intel can massively benefit in
the process by ensuring this solution has excellent support for its SoCs. By
doing this better than anyone else in the market Intel positions itself as the
supplier of first choice in the embedded space. Yocto Linux will be the project
umbrella under which do this.
== Existing Build System Solutions ==
Some kind of build system is the cornerstone of most embedded projects and
Yocto Linux is no different. There are many such solutions in the embedded space. In
order to capture the attention of the industry this project requires something
open that people can see and use without royalties, much like Linux itself.
For this reason only open source systems are contenders for this project.
There are a variety of possible existing contenders and its worth evaluating
these for their strengths and weaknesses:
Buildroot - One of the older embedded build systems, this is makefile driven
and is a natural evolution of the scripts based build solutions many people
start with when starting an embedded project. The general feeling of this
system is that it has had its day, it was a good learning experience but
lacked the fundamental structure to its processes to grow into anything bigger
than what it is. It lacks fundamental features such as packaging.
E2-Factory - A more modern from scratch solution that it very structured and
rigorous in its build processes. Sadly whilst the core tool is open source, no
metadata is available so its ruled out. It does have some neat features like
dependency checksumming which can be learnt from though.
Desktop (PC) Distribution Systems - Whilst not embedded solutions, these are
widely used for PCs so should be considered. The fundamental problem is that
these systems all promote a "one size fits all" approach to software and do
not support customisation for multiple targets easily or well. You cannot
build two distributions of different character from one set of metadata with
these solutions. In the embedded space this rules the solution out but again,
there are things that can be learnt from like the OBS web interfaces.
PTXdist - Makefile driven, more advanced than buildroot but is small and only
caters for a small segment of the market.
LTIB - This is a system developed and pushed by Freescale which combines spec
files with a kconfig menu front end. Its nice to have the menu control of the
system but it has limited package support, limited platform support and is
very tied to the rpm format and is not as structured, mature or comprehensive
as other solutions in the marketplace.
OpenEmbedded - If there were an industry standard, this would be it. It was
written to learn from all the mistakes in buildroot, has a well defined
structure to its metadata and has by far the largest coverage of open source
software. It dropped Makefiles in favour of a custom data file format of its
own with tools to handle them (bitbake). Feature wise its also the most
advanced system in the market and it was specifically designed for the
customisation needs of the embedded market space.
Poky - Created by Openedhand as a commercial version of OpenEmbedded and the
rights to it are now owned by Intel. Has a good reputation in the market as
lots of technical developments in OpenEmbedded have come via Poky.
LDAT - The build system developed and used by Wind River. This is not an open
solution at this time but could become open so is worth evaluating. Its
strength is that it has a large feature set and is widely used by Wind River
customers.  Its makefile derived and shows signs of strain due to being
constrained by the file format. It has been developed behind closed doors with
a narrow focus and as such does not stand a good chance of wide community
adoption if opened and hence is unlikely to become an industry standard.
== Yocto Linux build system ==
Acceptance of whichever build system is used by the wider community is
essential to the project. We're in the fortunate position that Poky is an
available option. The advantages of this choice include:
* Its already well established and accepted in the community
* It has commercial leanings
* Intel can control its development course and contributions to it
* Its based on OpenEmbedded - industry leading
On this basis it has been decided that Poky makes the best choice of build
system for Yocto Linux.
Where there are gaps in functionality we have the ability to influence and
change this.
== Yocto Linux Components ==
If Poky is the build system for Yocto Linux, what other components are there? Should
Yocto Linux perhaps be Poky?
There is much more to Yocto Linux than just the build system. It makes sense to have
namespaces for the components which is why Poky should remain just the build
system.  The other features/components that make up Yocto Linux include:
* A place of collaboration - There are very effective strong communities in the opensource world such as xorg.org, kernel.org and gnome.org but there is no such place for embedded. Yocto Linux can provide this, primarily by providing a place where embedded focused software projects can be housed and discussions can be had.
* A reference implementation - Any reference distribution/images created by Poky are something that is provided by Yocto Linux . This aims to reinforce the idea that Poky is the build system, not the end result of any given build of the metadata.
* Board Support Packages - Yocto Linux provides a place for Intel to publish its SoC BSPs.
* Testing infrastructure - Yocto Linux can provide hardware to allow regular regression testing of the Poky metadata and demonstrate this publicly.
* Partners - Yocto Linux also provides the entity which other parties can associate with to jointly improve and advanced the embedded Linux software position.
* Standards - Yocto Linux can become the entity which sets the standards for the industry. Yocto Linux aim to be the model of best practises.
* Documentation - Another output possible from the build infrastructure is documentation of all the components where the source contains these. As part of the reference implementation it would be desirable and extremely useful for the community to have access to a set of documentation all in one place.
More details on the individual components follows below.
== Yocto Linux Partners ==
To succeed, Yocto Linux needs to ensure people will use it and one way of doing this
is through partners.
Discussions between Wind River and Intel revealed that they share frustration
over the structure of the Embedded Linux market as mentioned above. The
fragmented nature of the market has a cost to them and also presents a barrier
to entry and their build system LDAT is different to any other on the market.
All their BSPs effectively have to be developed and maintained in-house at
significant cost.
Wind River see Yocto Linux as very desirable as it streamlines the transition path
from the community to their commercial support. Creating a BSP standard also
reduces the amount of work they have to do in integrating BSPs themselves. For
these reasons Wind River are keen to be a founding partner of Yocto Linux and adopt
it as the basis of their commercial Linux product. Where Yocto Linux  solutions lack
the features they need for this there is agreement to enhance Yocto Linux and its
components like Poky.
As Yocto Linux grows and becomes public, its likely more partners will emerge and
this will make the project stronger.
== Benefits to Intel ==
Some of the benefits to Intel are:
* Groups within Intel have one enablement target to worry about, not multiple ones
* Easy migration path for Intel customers to gain commercial support via Wind River
* By following a standard process code escapes and license issues can be controlled
* Reduced costs as only one set of enablement is required and can be reused
* Groups don't have to face a multitude of upstream open source projects as Yocto Linux buffers them from this
* Standardisation within Intel on embedded development process and infrastructure reducing costs and allowing better cooperation between groups
== Wind River Adoption ==
The shift to Yocto Linux is planned to happen in parallel with the Wind River 5
release so at release point, there is a working Yocto Linux based product which can
be shown to customers.
Intel engineers will be responsible for the major feature additions that are
needed for this project as identified and agreed upon. They will also be
providing support to Wind River as needed throughout the transition period and
beyond as experts on Poky, Yocto Linux and dealing with the open source community in
general.
Wind River will contribute some missing features and functionality. Their
engineering effort being focused on the migration paths from their existing
LDAT based system, their custom tools that integrate with this and the
requisite documentation and process updates that are required.
== Yocto Linux Components: A place of collaboration ==
The Yocto Linux website is intended to house embedded related software projects since
there currently isn't a good home for a lot of these. We already have a number
of different projects which could be used to seed this site:
* Poky (from Intel)
* Poky Anjuta SDK integration (from Intel)
* MeeGo Eclipse SDK integration (from MeeGo)
* Pseudo - A replacement for fakeroot (from Wind River)
* qemuGL - A project to allow GL passthrough in virtualsation (from Intel)
* OProfileUI - an embedded targetted client/server profiling tool (from Intel)
* Cross platform prelink tools (from Wind River)
Anyone wishing to house the source code/mailing list/bug-tracking for an open
source project furthering embedded technology would be free to do so on this
site.
== Yocto Linux Components: Reference Implementation ==
A reference implementation will be available from the Yocto Linux website. These will
be packages, images and an SDK generated by Poky supporting the range of
reference hardware we're promoting. For IA, this means we'd have images
capable of being run under emulation in QEMU as well as having support for a
suitable selection of our reference boards, particularly the SoCs chosen from
our BSP selection.
For other architectures we will provide reference images suitable for running
under emulation as proof we are architecture independent but Intel will not
provide builds for other specific platforms. If other partners join the
project and undertake to maintain platforms on other architectures they are
free to do so however.
It is worth highlighting that the reference distribution and images created by
Poky are something that is provided by Yocto Linux . We should reinforce the idea that
Poky is the build system, not the end result of any given build of the
metadata. The images and packages on the Yocto Linux website are also a reference
implementation which others are then free to build off.
At any given time there is likely to be a stable release version of Yocto Linux 
available and also packages and images build from the master branch as part of
automated testing. These are provided to demonstrate the status of the
project, that everything is working and allow people to test new features. Its
expected that product development be done based on stable release branches and
if cutting edge features are required, these will be ported to the last stable
release either by Yocto Linux maintainers if there is a widespread need, or on an
individual basis if not.
The benefit of having this reference implementation available is that anyone
wishing to do embedded development has a known good place to start it from. By
sharing this common base, solutions to any issues can be shared, faster
development and time to market is possible, a strong community is formed and
there is a common future forward migration path.
== Yocto Linux Components: Board Support Packages (BSPs) ==
Currently Intel has no standardised way of publishing a BSP for a given
platform either in the location its made public or the format. Traditionally,
Intel has not had many platforms so this has not been a problem. It is a
massive problem in the embedded marketplace and nobody has as yet found a way
to deal with this well. ARM is a particularly bad example with hundreds of
different kernel trees, let alone userspaces. A given ARM platform is unlikely
to work well with a standard mainline Linux kernel.
To be successful in the market Intel needs to learn from this and have one
central place where up to date code for a given Intel platform can be found.
This needs to provide complete support for that platform including any binary
firmware or other closed source userspace components required.
Yocto Linux gives Intel a standardised place and standardised format to do provide
this support for its SoCs and other platforms. The exact format will be
determined by the needs to the platforms but will be generic enough that an
industry BSP standard could well be established.
== Yocto Linux  Components: Testing Infrastructure ==
One area many embedded build systems fall down is that their core developers
can make them work but a newcomer to the project cannot. These systems by
their nature are extremely complex and its essential they are regularly and
extensively tested. There are two forms of testing that are desirable:
Incremental - This provides an "acid test" of the last change made to the
metadata. Its designed to run relatively quickly and test the change but not
exhaustively or from scratch. If the tests succeed, its expected that there is
a 95% chance the change did not cause any regressions. It also checks this
change builds against an existing build. The developer can expect a warning
about problems with a change quickly from these tests.
Full - This is a test of the metadata as a whole from the ground up and
ensures that the system is self consistent. Its designed to prove there was no
regression but obviously the tests take longer to run.
Its expected that incremental tests will happen on every commit to the
metadata and that full testing will happen once every 24 hours at a minimum.
Its also desirable that in future, separate branches can be tested outside of master and only merged into the main branch when a test suite completes.
By doing this publicly, we're forced to keep the metadata buildable,
consistent and people can see that we do this and care about the quality of
what we're doing.
Eventually it would be desirable to also test the build output of the system
either under emulation or on the real hardware in the form of "board labs".
There is an significant capital expense of setting these up as well as a
current lack of software to support their use. It would however fit well in
the Yocto Linux testing infrastructure if we can reach a point where this is
practical.
== Yocto Linux Components: Partners ==
Partners are an important part of Yocto Linux as we want to defragment the embedded
market and we can't do this if we don't build relationships with others. Its
one of the key areas many other efforts in this marketplace fail as they tend
not to cover the needs of the majority of the community.
The main criteria for becoming a supporting partner will be agreeing to
standardise working practises in line with Yocto Linux's objectives and most likely
starting to use it as a standard. This may also involve offering support and
development of its tools and infrastructure.
In return partners benefit from the defragmented embedded market and jointly
improve and advanced the embedded Linux software position.
== Yocto Linux Components: Standards ==
Yocto Linux aims to be a model of best practice in the industry. Whilst it will not
an official objective, the hope is that by doing this Yocto Linux effectively starts
setting standards for the industry. To do this it needs to meet the needs of a
wide section of users but this is a stated goal. The benefit of becoming a
standard, even unofficially is you are then in a position to lead the market
and also focus effort in building a better ecosystem.
== Yocto Linux Components: Documentation ==
Whilst there is a wealth of information online about given projects, nobody
has managed to effectively document a complete Linux software stack. The
reasons is related to the size of the undertaking and the effort required to
maintain it.
Through the Yocto Linux build infrastructure its possible that documentation
generation could be largely automated. We'd have to find a way to bind all the
separate pieces of documentation together but I think this is solvable
problem.
This coherent and up to date source of documentation would be another feature
of the project provided to the community as part of our reference
implementation. There would be a documentation collection corresponding to
each release as well as a version corresponding to the latest master branch.
An added benefit of this would be that individual software projects would be
encouraged to put their documentation into order if they could see a high
profile consumer of it which is inline with Yocto Linux's objectives of
standardisation and improving the embedded ecosystem.
== Build System - More details ==
The structure of OpenEmbedded and Poky are specifically tailored to the needs
of the embedded market space. One of the main ways this happens is the
powerful way the metadata is split into separate configuration areas:
: “Recipe” - details of a software package
The recipe represents individual software components of the system, usually
corresponding to a project which then goes on to make a release tarball. The
recipe contains information on where to get this source code, how to extract
it, information on any tweaks it may need for known issues (patches), what
configuration options this software has and how to apply them, how to compile
it and how to extract and split the output into packages.
There can be one definition of a software project which is them shared by
definitions for specific version releases of the software.
: “Distro” - policy of the resulting system
The distribution configuration is concerned with policy decisions such as
which ABI to use, which versions of software should be preferred and is a
representation of choices available and the values selected.
: “Machine” - details of a specific hardware target
The machine entities are information about particular hardware targets. They
can be more general such as "netbook" or provide information about specific
hardware like a Sharp Zaurus handheld SL-C3000 (spitz). These given
information about the capabilities of the hardware such as the presence of a
keyboard, touchscreens, displays, audio capabilities, I/O interfaces and so
on. Also details about the best formats to provide images for the device can
be contained here.
This information is used by the system to customise output to work well on
specific hardware.
: “Image” - groups of software packages
These are package groups which provide a working end result. "Sato" images
contain all the functionality required to boot the Sato UI for example. A
console image contains enough to boot a system to a console shell prompt. The
contents of an image will depend on information from both the distribution and
machine configurations.
Each of these functionality areas are independent of the others and one can be
changed or customised separate from the others. Bitbake through the structure
of the metadata is able to combine the information from each section to
provide a specifically tailored end result.
The second powerful feature of the metadata structure is the overlay or layer
capability. By combining several layers of metadata, a basic set of
functionality can be extended or customised at multiple levels allowing a
basic core to be extended and customised for different uses.
For example we could have the following metadata stack:
* Core Poky Metadata
* MeeGo specific Overlay (adding the MeeGo UI software packages)
* Vendor specific BSP (adding support for a particular hardware platform)
* Commercial Overlay (provided by say Wind River adding commercialisation to the underlying layers)
* Customer Specific Overlay (adding customisations only relevant to the customer)
At each level, features can be added and underlying layers can be customised
without duplicating code. The changes anyone makes also remain clearly
isolated from the underlying system meaning they're clearly identifiable and
can easily be ported to newer versions of other layers such as the core
metadata.
This structure encourages reuse of code and sharing and collaboration of
metadata. An objective is to make the core as useful as possible so over time
code initially developed as upper layers will tend to head towards the core
where it makes sense. There will be functionality blocks such as "meego" which
make sense as an overlay in their own right and customisations which will only
ever apply to an end user's application too and this structure supports them
all well.
== Build System - Configurability ==
Due to its modular nature, OE and Poky are very much infrastructures rather
than monolithic projects. Any one component is usually isolated from the rest
of the system meaning that if some area is found to need improvement, this can
happen and the technology can move forwards without impacting other areas of
the system. Pretty much everything is configurable.
Implicit assumptions in the system have been carefully avoided. Issues such as
output package format(s) or output image format(s) are abstracted such that
multiple outputs are available and future developments can be added too, very
easily. This makes the system very adaptable to almost any situation at a
relatively low cost compared to other systems.
== Build System - Cross compiling and multi architecture ==
Firstly, it needs to be realised that Yocto Linux has to be a multi architecture
system. If it is specific to a single platform such as IA, it will
automatically be ruled out of the embedded market and the community focus
which we'd like to see will not happen. It therefore makes sense to have a
multi architecture system but there is much we as Intel can do to ensure IA is
the best supported architecture out there.
Regardless of architecture issues, for embedded systems its always likely
there will be some differences between the build system's configuration and
that of the target systems. This may be different configuration options,
compiler flags, package versions and many other possible differences, it's
impractical to even list them all.
As such some way of isolating the target system from the build system is
essential. The ways of doing this have been a topic of discussion for many
years and will continue to be. Various approaches such as chroots, emulated
environments, bare cross compilation and cross compilation using sysroots are
all possibilities.
In the general multi architecture case the build system is most likely
different from the target architecture. This can apply on IA with a 64 bit
build system building for a 32 bit target and vice versa. Since emulation and
chroot aren't always possible or reliable, OE and Poky are a cross compiling
build system making the best use of new cross compilation technology such as
sysroots as it becomes available.
In the embedded space, cross compilation is considered standard practise and
widely accepted. The difficulties of cross compiling are also continually
being reduced with new technology.
== Build System - Any downsides? ==
The biggest downside of OE/Poky by far is the shear complexity of the inner
workings of the systems. They have been designed to be powerful and
comprehensive which means there is a steep learning curve to a true deep level
understanding of them. There are also several concepts which are different to
a traditional 'desktop' Linux system.
Poky has been at the leading edge of making the systems easier to use and is
designed such that anyone with some basic Linux knowledge should be able to
download it and build a Linux system from scratch. It also has a comprehensive
manual covering many of the questions a new user to the system has. As always
there is room for improvement but there is a strong foundation to build on.
== Build System - Vendor Use Cases and Development Process ==
The end result most vendors require are software "images" which then can
install onto the end products. In some cases these will be field upgradable
either through placing a new image onto the device in totality or in some
cases though package upgrades in a more traditional PC Linux distribution
style. In order to reach this goal there are several stages of development and
the Yocto Linux infrastructure needs to support them through them all:
: Prototyping - The end hardware is probably not available yet but a quick
mockup needs to be possible. This may be using CRBs, possibly customised to
demonstrate key hardware features of the end device or it could be done using
virtualisation.
: Hardware Development - The system needs to allow for testing of new hardware as its developed.
: Software Development - The system needs to allow development of the software
that will be used on the final device, often before the final hardware is
available. Some developers will be interested in the broad system and how all
the pieces build and fit together, others could just be application developers
interested in developing a single application. Both sets of developers must be
supported.
Poky itself fulfils some of these needs but not all of them. There are some
cases having the full build system available is overkill for a specific
developers needs. The counterpart is the SDK.
== Software Development Kit (SDK) ==
The SDK is a standalone output from Poky which consists of a sysroot
representing a target system and some tools which allow the developer to build
further software to work with that sysroot. Since we have little control over
the system the developer will be using the SDK on, a standalone cross compiler
and cross build tools are provided with the SDK to isolate the two systems in
the same way Poky itself does.
Using this toolchain the developer can build applications and libraries that
will work with their target platform. Whilst usable from a shell, the SDKs are
also capable of being used from within IDEs such as Eclipse and Anjuta at the
preference of the developer.
By having the SDK it further reduces the barriers to entry of developing
applications for embedded systems since the developer does not have to use the
build system itself. There is however a migration path should the developer
require to become more involved with the build system itself.
== Current Status and Areas for Development ==
Yocto Linux itself does not currently exist and needs to be established as detailed
above. OE and Poky do exist and are powerful and already extensively developed
but there are some gaps in functionality which there are some gains by
addressing:
* By filling functionality gaps it promotes OE/Poky/Yocto Linux as the standard in the marketplace
* By providing this technology first in Poky, it promotes Poky and Yocto Linux
* Some functionality is required to enable adoption of commercial solutions vendors such as Wind River where features they currently enjoy in systems like LDAT are unavailable in Poky.
Yocto Linux itself goes a long way to solving a number of problems by pulling pieces
like the various SDK plugins and other technologies into one place which can
then be consistently documented. There is also a list of features specifically being worked on for Poky and Bitbake.
== Yocto Linux Development Team Structure ==
To fulfil its objectives we anticipate the need for 3 different teams of
people working on the project:
=== Embedded Core Team ===
This team provides the core guidance on the project and consists of the
architect and technical direction, programme management, some developers
working on the overall picture of the project along with people responsible
for overall documentation and QA.
=== Embedded Tools and SDK Team ===
This team is responsible for improvements to the tools and infrastructure.
They would be implementing features like the server mode of bitbake and its
integration to something with similarities to Buildbot, the Eclipse and Anjuta
plugins and SoC support through the Yocto Linux SDK
=== Distribution Engineering ===
This team is much more focused on the day to day maintenance of the metadata,
upgrading versions when required to keep Poky at the forefront of technology,
watching for security updates, integrating any relevant MeeGo technology and
managing metadata releases and QA.
The Yocto Linux team does not directly cover development of SoC drivers. The teams
will advise on integration of code into Yocto Linux in parallel with its submission
of upstream and assist with any issues that arise however.

Revision as of 22:10, 22 October 2010