Yocto Project - User contributions [en]

Security

2021-11-12T15:41:20Z

Nicolas Dechesne: /* Yocto Security Team */

Since the Yocto Project is intended to be flexible and meet the needs of many applications, we leave policy-making decisions around security to our end users. Our goal instead is to ship each release with metadata that follows best practices in that we try our best not to release recipe versions which are known to have significant security vulnerabilities. Generally this is done by upgrading recipes to newer versions that are no longer vulnerable to these issues.

Upgrading recipes to the newer versions in the maintenance branches is not always easy, this is why we provide a patch for the existing version instead if we detect a vulnerability in a package. The patches are added to the recipes, see example below:

poky/recipes-connectivity/bind/bind_9.9.5.bb

SRC_URI = "ftp://ftp.isc.org/isc/bind9/${PV}/${BPN}-${PV}.tar.gz \
file://conf.patch \
...
file://bind9_9_5-CVE-2014-8500.patch \

We provide a tool [https://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/meta/classes/cve-check.bbclass cve-check.bbclass] to help report possible security vulnerabilities in the Yocto Project against the [http://nvd.nist.gov/home.cfm National Vulnerability Database]. Unpatched CVEs can be detected using the cve-checker which compares bitbake recipes, their versions and applied CVE patches to reported CVEs for that SW component name and version in the NVD database.

Another good source to track reported CVEs is via the oss-security mailing list (Open Source Software Security) http://www.openwall.com/lists/oss-security/

== Yocto Security Team ==

Currently the Yocto Project does not have a Security team. We have two methods of communicating to the project.

* See the Yocto Project TSC Future Directions [https://wiki.yoctoproject.org/wiki/Future_Directions#Security security]

'''How to Contact the Yocto Project regarding Securely'''
---------------------------------------------
We have set up two security-related mailing lists:
* '''Public List'''
: yocto [dash] security [at] yoctoproject[dot] org
: This is a public mailing list for anyone to subscribe to. This list is an open list to discuss public security issues/patches. For more information including subscription information please see the [https://lists.yoctoproject.org/g/yocto-security yocto-security mailing list info page].

* '''Private List'''
: security [at] yoctoproject [dot] org (Forwards to the following addresses)

: : - '''mhalstead [at] linuxfoundation [dot] org'''
: For secure communications, please send your messages encrypted to both using the following GPG keys.
: Remember message headers are not encrypted so do not include sensitive information in the subject line.

: Download public keys: [https://pgp.mit.edu/pks/lookup?op=vindex&search=0x3373170601861969 Michael Halstead]

Anyone can contribute with security patches as before, but those volunteering to this security team will sync/organize security related activities and take more responsibility.

'''What you should do if you find a security vulnerability'''
---------------------------------------------
If you find a security flaw; a crash, an information leakage, or anything that can have a security impact if exploited in any Open Source packages used by the Yocto Project, please report this to the Yocto Security Team. If you prefer to contact the upstream project directly, please send a copy to the security team at Yocto as well.
If you believe this is sensitive information, please report the vulnerability in a secure way, i.e. encrypt the email and send it to the private list. This ensures that the exploit is not leaked and exploited before a response/fix has been generated.

'''What Yocto Security Team does when it receives a security vulnerability'''
---------------------------------------------
The team performs a quick analysis and reports the flaw to the upstream project. Normally the upstream projects analyzes the problem. If they deem that it is a real security problem in their software, the project will email the linux-distros mailing list and notify all the big Linux distributions/vendors about the existence of this vulnerability/flaw. These mailing lists are normally non-public. The project and people on the linux-distros can then agree on a release date when the flaw should be made public.
There is also sometimes some coordination for handling patches or backporting of patches etc, or just understanding the problem or what caused it.

When the security issue is finally to be made public, normally upstream project is responsible to contact Mitre (cve.mitre.org) to get a CVE number assigned to it and copy the information to other Opens Source Security mailing lists to inform the whole world of the vulnerability.

'''If an upstream project does not respond quickly'''
---------------------------------------------
If an upstream project does not fix the problem the Yocto's Security Team will contact linux-distros and community and together try to solve the vulnerability as quickly as possible.
Normally big Linux vendors fix the problem if the problem affects their products.
Chances are that everyone from the enterprise distros to the commercial Yocto vendors will get fixes done first, but it is nice to have safety net for issues that really are specific to OE and embedded.

== Branches maintained with security fixes ==
See [https://wiki.yoctoproject.org/wiki/Stable_branch_maintenance Stable branches maintenance]for detailed info regarding the policies and maintenance of Stable branch.

Versions in grey are no longer actively maintained with security patches, but well-tested patches may still be accepted for them.)

== Policy for updating package versions for the stable branches ==
The Yocto project purposely limits updating of packages on oe-stable releases to items that address security problems (e.g. CVEs). For packages like QEMU we avoid updating between from one "dot.dot" to another related "dot.dot" version since it has been seen in the past that even with "simple" updates, things can go wrong and a lot more testing is required to verify compatibility. Better to only add CVE patches to fix specific point problems.

== Kernel security patches ==

Kernel security patches are backported to Linux-yocto kernels regularly from https://www.kernel.org/
=== Linux-yocto ===
linux-yocto_3 (maintainer: Bruce Ashfield)

=== Vendor kernels ===
Kernel security patches are also backported to Linux-vendor kernels from https://www.kernel.org/

* meta-intel (meta-intel uses Linux-yocto)
* meta-xilinx (meta-xilinx@lists.yoctoproject.org)
* meta-ti (meta-ti@yoctoproject.org)
* etc

== How to test ==

If there is any test case for the vulnerability by the upstream project or community
- Run the test to reproduce the problem and verify the correction.
- Run the regression test

If there isn’t any test case and it is complicated and time consuming to write a testcase
- Run the regression test

'''Regression test'''

* Build the core image for at least two architectures (preferably one big-endian and one little-endian)
* Run ptest (for those branches/packages that there is ptest mechanism)

== Patch name convention and commit message ==

Security patches like any Open Source development should follow the openembedded's Guidelines:
*[http://openembedded.org/wiki/Commit_Patch_Message_Guidelines Commit Patch Message Guidelines]
*[https://www.kernel.org/doc/Documentation/SecurityBugs kernel security bugs policy]

Note that security patches should have CVE: tag and reference to the CVE identifier both in the patch file/s and the commit message.

'''Ex upstream patch:'''

Please change the upstream patch "wscanf-allocates-too-little-memory.patch" to "CVE-2015-1472.patch" (or CVE-2015-1472-wscanf-allocates-too-little-memory.patch). Keep the original commit message and add reference to the CVE and upstream patch.

<Keep the original commit message>

Upstream-Status: Accepted <or Backport>
CVE: CVE-2015-8370

Reference to upstream patch:
https://sourceware.org/git/?p=glibc.git;a=patch;h=5bd80bfe9ca0d955bfbbc002781bc7b01b6bcb06

Signed-off-by: Joe Developer <joe.developer@example.com>

'''Ex meta patch:'''

Please make sure to add the package name in the subject and the reference to the CVE. Example for the commit message:

bash: CVE-2014-6278 <if there are multiple CVEs list all>

<short description>

<[YOCTO #xxx] if there is any>

References
https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2014-6278
https://bugzilla.redhat.com/show_bug.cgi?id=CVE-2014-6278
xxxx

Signed-off-by: <your email address>

== Workflow of Yocto Project's bugzilla ==
* To Open a Security defect go to [https://bugzilla.yoctoproject.org/enter_bug.cgi?product=Security%20-%20Recipe%20Upgrade Security ‑ Recipe Upgrade]
** Access to this issue can only be viewed by the submitter and a small group of Bug triage folks:
*** Armin Kuster
*** Randy MacLeod
*** Richard Purdie
*** Ross Burton
*** Tim Orling
*** Stephen Jolley
** The normal bug triage process will be applied.

* If the issue is already public please send the patch when available to the appropriate mailing list
* If the issue is private, attach a patch if available to the defect is preferred.

== Some security related links/useful tools ==

* [https://wiki.yoctoproject.org/wiki/images/5/58/Yocto_Summit_Lyon_Day1_2019.pdf#page=36 Yocto Project and CVEs, presentation by David Reyna in 2019 Yocto Developer day]
* [https://github.com/nluedtke/linux_kernel_cves/ Linux kernel fixed and reported CVEs in all branches and point releases]
** for example [https://github.com/nluedtke/linux_kernel_cves/blob/master/data/4.14/4.14_security.txt 4.14.x kernel series CVE status]
** note that cherry-picking CVE fixes for kernel is not recommended and users should merge full stable releases instead, see [http://www.kroah.com/log/blog/2018/08/24/what-stable-kernel-should-i-use/ What Stable Kernel Should I Use? by stable kernel maintainer Greg Kroah-Hartman]
* [http://www.cvedetails.com CVE details]
* [http://www.cvedetails.com/vulnerability-list/vendor_id-33/product_id-47/year-2014/Linux-Linux-Kernel.html CVE list, Linux kernel 2014]
* [http://layers.openembedded.org/layerindex/branch/master/layer/meta-security/ Meta-security-layer]
* [http://www.yoctoproject.org/docs/current/dev-manual/dev-manual.html#making-images-more-secure Making images more secure]
* [http://cvechecker.sourceforge.net Cvechecker]

== Security Issues Addressed in Yocto Releases ==

Member Technical Contacts

2021-09-15T12:14:45Z

Nicolas Dechesne:

This page tracks the official technical contributors to the YP core from each of the Yocto Project's member organizations. Note that many organizations provide more resources than just one, particularly in terms of hardware or specific feature support, which is why the Yocto Project is a de facto standard for building Linux for embedded systems.

Platinum Members
* Amazon: Richard Elberger <elberger@amazon.com>
* Arm: Ross Burton <ross.burton@arm.com>, Jon Mason <jon.mason@arm.com>
* Cisco: Taras Kondratiuk <takondra@cisco.com>
* Comcast: Raj, Khem <Khem_Raj@comcast.com>
* Facebook: DJ (Dharmesh Jani) <janidb@fb.com>
* Intel: apoorv sangal <apoorvsangal@gmail.com>
* Microsoft: Paul Eggleton <paul.eggleton@microsoft.com>
* Wind River: Robert Yang <liezhi.yang@windriver.com>; Saul Wold <Saul.Wold@windriver.com>
* Xilinx: Mark Hatle <mhatle@xilinx.com>

Gold Members
* AGL: Jan-Simon Moeller <jsmoeller@linuxfoundation.org>
* Huawei: Davide Ricci <davide.ricci@huawei.com>, Andrei Gherzan <andrei.gherzan@huawei.com>
* Renesas: Mr. Takamitsu Honda <takamitsu.honda.pv@renesas.com>
* Siemens: Chris Larson <chris_larson@mentor.com>
* Texas Instruments: Praneeth Bajjuri <praneeth@ti.com>

Silver Members
* AMD: Wade Farnsworth <wade_farnsworth@mentor.com>
* Dell: Michael Brown <michael.e.brown@dell.com>
* Founderiesio: Ricardo Salveti <ricardo@foundries.io>
* LG: Minjae Kim <nate.kim@lge.com>
* Linaro: Nicolas Dechesne <nicolas.dechesne@linaro.org>
* Lineo Solutions: Masahiro Miyake <miya@lineo.co.jp>
* MontaVista: Armin Kuster <akuster@mvista.com>
* NXP: Lauren Post <lauren.post@nxp.com>
* Savoir-faire Linux: Sébastien Le Stum <sebastien.le-stum@savoirfairelinux.com>
* Savoir-faire Linux (technical contact backup through December 2021): Jérome Oufella <jerome.oufella@savoirfairelinux.com>
* ST: Christophe Priouzeau <christophe.priouzeau@st.com>

Technical Partners
* OpenEmbedded: Philip Balister <philip@balister.org>

'''Note''': a ''private'' mailing list exists with all individuals listed here, mainly to facilitate the communication and the organization of meetings. All technical discussions are encouraged to be made on the existing, open, project mailing lists. If you believe you need to be added to the private list, please reach out to the Yocto Project Community Manager.

SeasonOfDocs

2021-06-02T23:43:16Z

Nicolas Dechesne: /* Candidate Criteria */

= Yocto Project Season of Docs =

==About==
The [http://yoctoproject.org Yocto Project] is an open-source project that delivers a set of tools that create operating system images for embedded Linux systems. The Yocto Project tools are based on the [http://www.openembedded.org/wiki/Main_Page OpenEmbedded] (OE) project, which uses the BitBake build tool, to construct complete Linux images. BitBake and OE are combined to form a reference build host known as Poky which includes the following [[Core Components|core components]]. This [https://www.youtube.com/watch?v=utZpKM7i5Z4 video] will help explain what it's all about.

Richard Purdie, the Yocto Project's lead developer and Linux Foundation Fellow, considers documentation to be one of the cornerstones of this project. The quality and breadth of the documentation is one of the aspects of the project which is appreciated by and commented upon by our users. One of the very first members of that team to work on what became the Yocto Project 10 years ago was Scott Rifenbark, the project's tech writer. Sadly, Scott passed away early this year. The project is therefore at a cross roads with its documentation and with a need of new contributors and with that, the potential for new ideas and direction.

Creating images and GNU/Linux-based distributions takes a wide breadth of knowledge. Even a simple and small image is the combination of dozens of individual software components, each of which has its own quirks and configurations. Each software component is free to choose whatever build system suits its developers, often with little regard for issues concerning cross-compilation. Users who are new to The Yocto Project, and especially new to image creation, are often overwhelmed by its necessary size and scope. Good, comprehensive documentation has always been at the core of The Yocto Project for both new and seasoned users alike.

The Yocto Project hopes it will be successful in its bid for GSoD. But moreover is hoping to find the right person who will stay on with the project for years to come.

==Google Season of Docs==
The Yocto Project is applying to the 2020 Google Season of Docs campaign. The deadline for the application is May 4th, 2020, and the results will be announced on May 11th, 2020. See https://developers.google.com/season-of-docs for additional information about this project.

The following mailing list was created to discuss about the Season of Docs project: https://lists.yoctoproject.org/g/season-of-docs/.

==Candidate Criteria==

* excellent written command of the English language
* you have experience as a writer, preferably writing technical documentation
* able to distill "geek gobbledygook" into simple and understandable prose
* knowledge of and the ability to use various writing tools and schemes (sgml, xml, sphinx, etc)
* ability to draw simple but meaningful diagrams, and incorporate them into your writing
* strong ability to work with and use the Linux command-line
* it would be very beneficial for this role to know how to use git, at least at a basic level
* a "nice to have" skill would be the ability to read, run, and understand short snippets of python3 and shell code
* another "nice to have" skill would be the ability to run different Linux distros (e.g. Fedora, openSUSE, Ubuntu) in virtual environments
* willingness to hang out in the #yocto channel on irc

==Ideas==

===Format Overhaul===
Currently the project's documentation is written in docbook, which is a natural choice for a documentation writer, but less so for, say, a developer. To make it as easy as possible for everyone to contribute to the documentation going forward, it would be a good time to convert the raw format of the documentation into a more modern format that is easier to learn. Hopefully this will encourage people who don't usually write project documentation to submit patches and updates. An experiment was undertaken to evaluate using Sphinx, see https://bugzilla.yoctoproject.org/show_bug.cgi?id=12970. In particular, there is significant potential to improve the way different versions of the manuals for different releases are handled and displayed to users through the website, as what is there today is not working for users.

===One Voice===
Previously the project's documentation was written with one voice. Contributors would suggest updates, and Scott would make sure the flow had a certain uniformity; as though it had all been written in one go, by one person. Moving forward, it's likely the project's documentation updates will come from many contributors. It would be great to have someone review the current documentation and future updates with an eye towards maintaining one voice. Currently there are no style guidelines for the project; establishing some written guidelines for that 'voice' may be part of this too.

===Organization===
Currently the project's documentation is spread out in several places and formats. There are a number of manuals, some tutorials, a collection of wiki pages, and more. It would be great to see all this information collected together and organized in such a way as to make it easier for people new or experienced with the project to find exactly what they're looking for, quickly. Classify all existing documentation according to various attributes to identify areas that have weak or missing documentation. Fill any gaps in the documentation by creating new docs targeting specific users, tasks, or experience levels; or help motivate the right people to help generate the content. Work with the project's webmaster to present a unified location for all documentation. Perhaps run a survey to poll users on their thoughts regarding the usefulness of the current documentation, and generate an actionable report.

===Tips and Tricks===
The project started a [[TipsAndTricks]] project which generated a significant amount of ideas and the start of content which it would be great to turn into sections in the manuals and better document some of these topics. Since the manual sections were never generated, the project stalled but could be restarted and the community would be keen to help with content and ideas.

===Update/Release Process===
Currently the documentation generation and updating process is quite manual. We believe that integration into our CI and release process is desirable and would be the best way to handle this going forward.

===How-to/Tasks Documentation===
There was work started to convert some of the manual to a more task oriented/'how-to' approach. https://bugzilla.yoctoproject.org/show_bug.cgi?id=11630 has some details but there is potential to significantly expand the task information. It would be great to see the manuals reach this goal of having a much more complete "how-to" task section.

===Better Layer Integration===
The Yocto project is organized as a "core" (which is maintained by The Yocto Project) and "layers" (which are maintained by volunteers). Creating an image for a specific device will often require information for the core and information from individual layer components. There is an opportunity to investigate how to incorporate documentation from various layer projects into a unified whole.

GSoC/Ideas

2021-06-02T23:42:51Z

Nicolas Dechesne: /* remote debugging with devtool */

= Project Ideas for the Yocto Project Google Summer of Code =

=== In General ===

What follows, in the [[#Miscellaneous Ideas]] section, are suggested projects by various members of the community. This page welcomes ideas from the community. Please add your ideas! Don't worry too much about the mentoring aspect; if you have a good idea put it here even if you are unavailable to mentor, perhaps someone else in the community will be able to act as a mentor for your idea. If you don't have an account to edit this page, please send any interesting ideas to [[User:Trevor Woerner]] by email (if you're in this community, you'll know how to email me).

If you are a student, this page is meant to help get you thinking about potential ideas for GSoC. A student is welcome to propose their own project ideas that aren't necessarily on this list.

One awesome place to look for project ideas is the [https://bugzilla.yoctoproject.org Yocto Project Bugzilla]. A number of bugs have been assigned to [https://bugzilla.yoctoproject.org/buglist.cgi?emailtype2=substring&list_id=616486&resolution=---&query_format=advanced&bug_status=UNCONFIRMED&bug_status=NEW&bug_status=ACCEPTED&bug_status=IN%20PROGRESS%20DESIGN&bug_status=IN%20PROGRESS%20DESIGN%20COMPLETE&bug_status=IN%20PROGRESS%20IMPLEMENTATION&bug_status=IN%20PROGRESS%20REVIEW&bug_status=REOPENED&bug_status=NEEDINFO&email2=newcomer%40yoctoproject.org&emailassigned_to2=1 newcomer@yoctoproject.org]. Many of these "newcomer" bugs would be too simple to comprise a GSoC project, but they could act as an inspiration for a more worthy project. Another excellent way to look for GSoC-worthy projects is to query the bugzilla looking for [https://bugzilla.yoctoproject.org/buglist.cgi?list_id=616487&bug_severity=enhancement&resolution=---&query_format=advanced&bug_status=UNCONFIRMED&bug_status=NEW&bug_status=ACCEPTED&bug_status=IN%20PROGRESS%20DESIGN&bug_status=IN%20PROGRESS%20DESIGN%20COMPLETE&bug_status=IN%20PROGRESS%20IMPLEMENTATION&bug_status=IN%20PROGRESS%20REVIEW&bug_status=REOPENED&bug_status=NEEDINFO enhancement] bugs; many of these enhancements could make for a great GSoC project!

In all cases, the student is responsible for finding a suitable mentor. Just because someone says they ''could'' be a ''potential'' mentor doesn't obligate them to mentor. Mentoring is an unpaid, volunteer position, if someone does agree to mentor it is because they are taking time away from their other obligations to do so!

=== Miscellaneous Ideas ===

==== remote debugging with ''devtool'' ====

*; Difficulty
*: medium

*; Skills Required
*: python 3

*; Hardware Required
*: no hardware is required, this can all be done with qemu

*; Where to Ask Questions
*: #yocto on irc.libera.chat or the [https://lists.yoctoproject.org/g/yocto yocto mailing list]

*; Potential Mentor
*: [[User:Trevor Woerner]] (tlwoerner on irc (Libera.chat and OFTC))

*; Description
*: A build generates all the pieces required to perform remote debugging (via cross-gdb) but getting everything to work requires a lot of manual fiddling (involving an SDK). ''devtool'' should be able to make this easier, and would also include the coding (''devtool modify''), cross-building (''devtool build''), and upload (''devtool deploy-target'') pieces to complete the remote debugging loop. Ideally remote debugging with a JTAG via openocd should also be supported too.
----

==== build improvements ====

*; Difficulty
*: medium-hard

*; Skills Required
*: python3, bitbake, building software, build systems

*; Hardware Required
*: no hardware is required

*; Where to Ask Questions
*: #yocto on irc.freenode.net or the [https://lists.yoctoproject.org/g/yocto yocto mailing list]

*; Description
*: There are many areas where a build can be improved in terms of memory consumption, recipe/config parsing, etc. For this project the student would be expected to instrument and profile either builds or some key section/stage of builds then look for ways to provably improve the build speed and/or resources used in measurable ways. Specific examples a) A key recent development is multiconfig but currently this causes significant parsing speed and memory overhead b) general parsing speed, particularly when adding many layers is a concern for many users c) memory overhead of bitbake often seems high, can it be quanitifed and reduced, particularly for parallel parsing?. Additionally it is worth noting the bitbake -P profiling option already exists and may assist.
----

==== debuginfo server====

*; Difficulty
*: medium

*; Skills Required
*: python3, bitbake, gdb usage

*; Hardware Required
*: no hardware is required

*; Where to Ask Questions
*: #yocto on irc.freenode.net or the [https://lists.yoctoproject.org/g/yocto yocto mailing list]

*; Description
See https://bugzilla.yoctoproject.org/show_bug.cgi?id=13807

----

The Yocto Autobuilder

2021-06-02T23:41:47Z

Nicolas Dechesne: /* The Yocto AutoBuilder */

== The Yocto AutoBuilder ==

The autobuilder '''[https://autobuilder.yoctoproject.org/typhoon/#/console Main Console]'''.

The Yocto AutoBuilder is a buildbot (nine) based autobuilder implementation that can be used to build out and test custom distros utilizing OE-Core (either bare or through the poky repository)

The source code can be downloaded from the '''[http://git.yoctoproject.org/cgit/cgit.cgi/yocto-autobuilder2/ yocto-autobuilder2]''' and '''[http://git.yoctoproject.org/cgit/cgit.cgi/yocto-autobuilder-helper/ yocto-autobuilder-helper]''' repositories.

For details on the design and configuration of the AutoBuilder, refer to the documentation in those repositories.

The yocto-autobuilder maintainer is [[User:Rpurdie | Richard Purdie]] (RP on IRC). All patches to the yocto-autobuilder2 should be sent to yocto@yoctoproject.org with "[yocto-autobuilder2]" in the Subject line. Please CC: richard.purdie@linuxfoundation.org. The hardware is maintained by the Linux Foundation on behalf of the Yocto Project and for infrastructure issues, please contact Michael Halstead (halstead on IRC).

'''NOTE: please use autobuilder2''', autobuilder is dead [1], buildbot eight is dead.

[1] http://git.yoctoproject.org/cgit.cgi/yocto-autobuilder/commit/?id=1369545f9819537535e4ab6ebeb49e7b173a8366

=== Starting Builds ===

For autobuilder access, contact Michael Halstead <michael@yoctoproject.org> as an account is needed to start builds. These are usually available to stable branch maintainers or in special cases other layer maintainers for running builds on the project infrastructure.

To start builds, from the main console you'd usually select 'a-quick' or 'a-full' from the top list of builders. If not logged in, login using the link on the top right of the page. You should then see a button "Start a-full build" or "Start a-quick build" which you can press. This opens a fairly complex form but in most cases you can use the "Release Shortcut Selector" to pre-populate the form with a given release's defaults. For a stable branch build on a test branch, you may then want to change the poky repository to point to poky-contrib and the branch to be the one you want to test. You should enter a reason for the build in the box at the top of the form. This is added to the [[BuildLog]] and us used by SWAT to decide what to do with bugs. When the form is correct, click "Start Build" at the bottom of the form.

If making a release build, be sure to check all three check boxes, "Do we want to save build output?", "Generate a release?" and "Send QA alert emails?". The release milestone, release number and release rc number need to be filled in as appropriate too.

The autobuilder can run multiple builds in parallel so builds can be queued as needed but please be sensible. The autobuilder users are usually around in #yp-infra on IRC which can be useful to schedule builds between us.

The autobuilder maintenance window is morning for US PST on Fridays and builds should not be run over this period to allow weekly maintanance on the worker distros to be carried out.

Autobuilder output for non-release builds is available at: https://autobuilder.yocto.io/pub/non-release/ and for release builds: https://autobuilder.yocto.io/pub/release/.

=== Autobuilder Build User Guidelines/Conditions of Use ===

If you have the ability to run autobuilder builds there are some things you need to be mindful of:

* It is expected that normally users should have resolved minor issues and done some testing before using the project infrastructure.
* You are expected to triage the results of your own build
* If you see unexplained failures, it is expected that bugs are filed for these, or where there are existing bugs, the bug should be updated. Please include which host/worker the build failed on. This allows triage to know which issues are occurring, their frequency and patterns like which host(s) they occur on.
* The maintenance window is on Friday mornings US pacific time. Please do not start builds until maintenance is complete or run builds which wouldn't finish before maintenance is due to start. Michael can start a build when maintenance is completed if you let him know.
* There is a rough priority hierachy of builds where master and stable branch release builds have priority. Avoiding builds during release periods if possible is helpful.
* Being present on #yp-infra on IRC is helpful and infrastructure/autobuilder discussion may happen there
* If a partially complete build is no longer useful for some reason, please stop it to allow the resources to be used by others

=== Resources ===

* [[Accessing Autobuilders]]
* [https://autobuilder.yocto.io/pub/non-release/ non-release build output] -- https://autobuilder.yocto.io/pub/non-release/
* [https://autobuilder.yocto.io/pub/releases/ release build output] -- https://autobuilder.yocto.io/pub/releases/
* [[AutoBuilder Maintenance]]
* [[AutoBuilder Cluster Setup]]
* [[Frequently Asked Yocto Autobuilder Questions]]
* [[Entropy on Autobuilders]]

Community Guidelines

2021-06-02T23:40:41Z

Nicolas Dechesne: /* IRC Guidelines */

==Overview and General Guidlines==

We want to keep the Yocto Community a great place to participate, but we need your help to keep it that way. While we have specific guidelines for various tools, in general, you should:

* '''Be nice''': Be courteous and polite to fellow members of the list.
* '''Respect other people''': No regional, racial, gender or other abuse will be tolerated.
* '''Keep it clean''': Keep the language clean (no swearing)
* '''Be helpful''': Be patient with new people and be willing to jump in to answer questions.
* '''Stay calm''': The written word is always subject to interpretation, so give people the benefit of the doubt and try not to let emotions get out of control.
* '''Keep it legal''': Make sure that you have the legal right to post your content and are not violating copyright or other laws.
* '''Adhere to Terms of Service''': All community contributions are also subject to our [https://www.yoctoproject.org/about/terms-of-service Terms of Service].

==Read Before Contributing - Search for existing answers==
'''IMPORTANT: Before you contribute, be sure that you have done a thorough search to see if your question has already been answered.''' We've already answered many questions, and you will get a better response from people if you have already done your due diligence to find obvious or partial answers.
* Search this wiki
* Search our [https://www.yoctoproject.org/tools-resources/community/mailing-lists mailing lists]
* Search the [http://www.yoctoproject.org/ Yocto Project website]

==Wiki Guidelines==

You can [http://www.yoctoproject.org/documentation learn more about our documentation] on the Yocto website, and if you are unfamiliar with MediaWiki syntax, we have a short how-to document with instructions for [[Using the Wiki]].

Creating articles in the wiki is a collaborative process. After you have written your piece others may:
* Edit
* Alter
* Adapt
* Add

So don't worry about making your article perfect the first time through. Don't hesitate to add content you think is useful and don't hesitate to make edits where you think you can help. There's always somebody to fix anything that breaks.

'''Posting Guidelines'''

Here are a few guidelines to keep in mind when using the Yocto wiki:
* '''Search first''': Before creating a new page or making significant contributions to a page, please do a quick search to make sure that you aren't duplicating existing content on the wiki or other areas on the Yocto Project website.
* '''Must be a registered user''': If you want to edit the wiki, you need to [[Special:UserLogin|create an account]] and confirm your email address (you'll get an email after registering with a confirmation link). Anyone can create an account.
* '''Contribute''': The wiki is a resource for anyone to use. Just keep the content relevant, that is anything related to Yocto as long as it meets our other guidelines should be appropriate.
* '''Make improvements''': If you find a typo or inaccurate information, just fix it.
* '''Respect links''': Please provide [http://www.mediawiki.org/wiki/Help:Redirects redirects] when you move content. Many people use the wiki and may have created bookmarks or linked to your content.

'''The Wiki is Public'''

Everything in the wiki is public.

Every edit and every new page created goes into the [[Special:RecentChanges|recent changes]] feed, which means that people will see your edits even if you haven't yet linked to a page.

Once it's out there, it's public.
* Technically, administrators can delete things, but the wiki content may be mirrored, has feeds and is in the Google cache, so deleting something doesn't make it go away.
* When you delete content from a page, the original content will still remain in the history for that page.

==Mailing List Guidelines==

[https://www.yoctoproject.org/tools-resources/community/mailing-lists More information about our mailing lists] can be found on the Yocto Project website.

'''Keep It Short'''

Remember that thousands of copies of your message will exist in mailboxes:
* Keep your messages as short as possible.
* Avoid including log output (select only the most relevant lines, or place the log on a website or in a [http://pastebin.com/ pastebin] instead)
* Don't excessively quote previous messages in the thread (trim the quoted text down to the most recent/relevant messages only).

'''Use Proper Posting Style'''

* '''No HTML or Rich Text''': Set your mailer to send only plain text messages to avoid getting caught in our spam filters.
* '''Do not top post''': Top posting is replying to a message on "top" of the quoted text of the previous correspondence. This is highly unwanted in mailing lists because it increases the size of the daily digests to be sent out & is highly confusing and incoherent. By default, most email clients top post. Please, remove the irrelevant part of the previous communication(in case of more than a single correspondence) and use bottom, interleaved posting.
* '''Using interleaved posting''': Bottom, interleaved posting is replying to the relevant parts of the previous correspondence just below the block(s) of sentences. For a comment to another block of sentences of the same quoted text, you should move below that relevant block again. Do not reply below the whole of the quoted text. Also remove any irrelevant text.
* '''Use links''': Please provide URLs to articles wherever possible. Avoid cutting and pasting whole articles especially considering the fact that all may not be interested.
* '''Don't include attached files''': Instead of including attached files, please upload your file to the [[Special:Upload|this wiki]] or another website and post a link to the file from your email message.

'''Do Not Hijack Threads'''

Post new questions or new topics as new threads (new email message). Please do not reply to a random thread with a new question or start an unrelated topic of conversation in an existing thread. This creates confusion and makes it much less likely that you will get a response.

'''Do Not Cross Post'''

Avoid posting to multiple lists simultaneously. Pick a mailing list that is most suitable for your post and just use that. CC'ing multiple lists should be avoided.

'''Subscribers Only'''

Only subscribers can post to our mailing lists. If you would like to contribute to our mailing lists, we think it is only fair that you be a subscriber. Please note that if you want to participate only occasionally, you can subscribe to a list and set your email options to digest or no mail and read the web archives when you want to catch up.

'''Additional Resources'''
* [http://www.shakthimaan.com/downloads/glv/presentations/mailing-list-etiquette.pdf Mailing list guidelines by Shakthi Kannan (PDF)]: Great guideline summary and examples of posting styles.
* [[Contribution_Guidelines]]

==IRC Guidelines==

[https://www.yoctoproject.org/community/mailing-lists/ More information about our IRC channels] can be found on the Yocto website.

Here are a few IRC guidelines:
* '''Don't post chunks''': Avoid posting big chunks of text - use [http://pastebin.com/ pastebin] or a similar service to shorten it to a link.
* '''Register your Nickname''': To avoid confusion and make sure you always have the same name on IRC, you should [https://libera.chat/guides/registration register your nick].
* '''Pick the right channel''': if you aren't sure, you should start in our main #yocto channel.
* '''More information''': this [http://irchelp.org/irchelp/ircprimer.html IRC primer] for new users.
* '''Web access''': If you don't normally use IRC and want to try it out, you can use the [https://web.libera.chat/?channels=#yocto Libera.chat Web IRC] chat.

==Bugzilla Guidelines==

We like to have fun but we also like the communications to run smoothly. To that end here are some guidelines for participating in the [https://bugzilla.yoctoproject.org/ Yocto Bugzilla].

* '''Be patient with others''': Sometimes people post imperfect bug reports. In case of missing information, kindly tell reporters how to provide it, and/or suggest what they can do to improve the bug report.
* '''Stay on topic''': Don't start endless debates on topics not directly related to the scope of a specific bug report.
* '''Use proper quoting practices''': Avoid quoting complete previous comments by stripping unneeded lines, and avoid answering above the quoted text.
* '''Don't abuse your privileges''': Submitters have permission to edit their bugs. If you abuse this privilege - for example, by reopening a bug that the maintainers have closed - your privileges will be revoked.

What to do if you find a bug
* '''Search first''': Try to avoid filing duplicates by searching to see whether your issue has already been filed.
* '''Ask''': You can ask on our mailing lists or IRC channels to see if anyone has seen this issue before to gather more details or see if someone has a workaround.
* '''Submit a bug report''': Give us as many details as possible about what happened and how your issue can be reproduced. '''IMPORTANT''': please include the build/commit ID as reported by BitBake, as this will help the team determine whether your bug has been fixed already.
* '''Track our progress''': Get feedback from the development community and track resolution status.
* '''Provide a fix''': Use the tools, project wiki and git source repository in the Yocto Project to fix the problem yourself.

Learn more about [[Bugzilla_Configuration_and_Bug_Tracking|our process for reporting bugs]].

==Contribution Guidelines - Sending Patches==
[[Contribution_Guidelines|Contribution Guidelines]]

[http://www.openembedded.org/wiki/How_to_submit_a_patch_to_OpenEmbedded How To Sumbit A Patch To OpenEmbedded]

==Guideline Violations - 3 Strikes Method==
The point of this section is not to find opportunities to punish people, but we do need a fair way to deal with people who are making our community an unpleasant place.
* First occurrence: Public reminder that the behavior is inappropriate according to our guidelines.
* Second occurrence: Private message warning the user that any additional violations will result in removal from the community.
* Third occurrence: Depending on the violation, may include account deletion or banning.

'''Notes:'''
* Obvious spammers are banned on first occurrence. This is necessary to keep the community free of spam.
* Violations are forgiven after 6 months of good behavior.
* Minor formatting / style infractions will be dealt with through education, not the 3 strikes process.
* Extreme violations of a threatening, abusive, destructive or illegal nature will be addressed immediately and are not subject to 3 strikes.
* Contact [[User:Nicolas_Dechesne|Nicolas Dechesne]] to report abuse or appeal violations (mistakes happen & will be corrected).

==Credits==
* [http://wiki.meego.com/Community_guidelines MeeGo Community Guidelines] were the basis for the Yocto Guidelines used under the [http://creativecommons.org/licenses/by/3.0/legalcode Creative Commons Attribution 3.0 License]
* [http://fedoraproject.org/wiki/Communicate/MailingListGuidelines Fedora Mailing List Guidelines] used as a starting point under the [http://creativecommons.org/licenses/by-sa/3.0/ Creative Commons Attribution-ShareAlike 3.0 Unported] license.

Main Page

2021-06-02T23:38:03Z

Nicolas Dechesne: /* Other resources */

== Welcome to the Yocto Project Wiki! ==
The [http://yoctoproject.org Yocto Project] is an open-source project that delivers a set of tools that create operating system images for embedded Linux systems. The Yocto Project tools are based on the [http://www.openembedded.org/wiki/Main_Page OpenEmbedded] (OE) project, which uses the BitBake build tool, to construct complete Linux images. BitBake and OE are combined to form a reference build host known as Poky which includes the following [[Core Components|core components]]. This [https://www.youtube.com/watch?v=utZpKM7i5Z4 video] will help explain what it's all about.

===Where to Start?===
If you're new to Yocto take a look at the '''[[Glossary]]''' so you're familiar with the terms used in this wiki and the project documentation. Then take a look at the [http://www.yoctoproject.org/docs/current/yocto-project-qs/yocto-project-qs.html '''Quick Start Guide''']. You can follow the steps in this document to clone the poky repository, quickly configure your build environment, and then try a build. Corporate firewalls can be problematic so network proxy configurations are detailed on the '''[[Working Behind a Network Proxy]]''' page. We advise you go straight for the [[Working_Behind_a_Network_Proxy#Option_2:_Chameleonsocks| Chameleonsocks option]].

===Where to Next?===
Thanks to the quick start guide, it's pretty easy to get your first Linux image and and running. Here are some places to look for help when improving your Yocto skills.
* The [https://linuxfoundation.org Linux Foundation] have some great [https://docs.google.com/presentation/d/1LmI3mHoD_Dzl8wplIYcUBrFF8BzDb_EadTvfbnpSK7Q/edit training slides]. There is also an [https://docs.google.com/presentation/d/1HoDtyN5SzlmuTN47ab4Y7w_i6c_VEW6EBUD944ntf38/edit#slide advanced slide deck] for more more experienced users.
* The first tool you'll need to get familiar with is '''bitbake''', so reading through the [https://www.yoctoproject.org/docs/current/bitbake-user-manual/bitbake-user-manual.html user manual] is recommended. You don't need to understand it all right now, but bookmark this page for reference. Implementation of bitake is covered in the [[Bitbake Internals Primer]]
* Once you start adding packages and configuring your image to create your own distribution, things can go wrong and it can hard to track down the root cause. There is no shortage of Yocto documentation resource, but if you're not exactly sure what you're looking for this '''[[Documentation Decoder]]''' will help you out. Also take a look at the [https://wiki.yoctoproject.org/wiki/Cookbook '''Cookbook'''] and [https://wiki.yoctoproject.org/wiki/Technical_FAQ troubleshooting guide]. Also [https://www.yoctoproject.org/blogs/jefro/2016/yocto-project-books these books] are helpful.
* Some new tools such as [http://www.yoctoproject.org/docs/current/toaster-manual/toaster-manual.html Toaster], [[Extensible SDK]] and [https://github.com/crops CROPS] are making it easier to get the best out of Yocto on Windows and Mac OS X. Take a look at the new workflow in [[Developer Workflow Improvements]].
* There is also a [https://wiki.yoctoproject.org/wiki/TipsAndTricks '''Tips and Tricks'''] section where more experienced developers contribute to articles that will help those new to Yocto Project.
* You can also read and participate on the [https://lists.yoctoproject.org mailing lists] - start with the [https://lists.yoctoproject.org/g/yocto main list] first - and the IRC channel. More information about the Yocto Project mailing lists, IRC and Matrix channels can be found [https://www.yoctoproject.org/community/mailing-lists/ here].

== Project planning ==

=== Features ===
* [[Yocto Feature Summary]] (Current and Next)

=== Project Planning for current release ===

* [[Planning]]

=== Project Status and Schedule ===
* [[Weekly_Status]]
* [[Yocto Project v2.8 Status]]
* [[Yocto 2.8 Schedule]]
* Testresults - https://git.yoctoproject.org/cgit/cgit.cgi/yocto-testresults-contrib/log/?h=intel-yocto-testresults

=== Future Directions ===
* [[Future_Directions]]

== Release Engineering ==

* [[Yocto Release Engineering | Yocto Project Release Engineering]]
* [[Yocto Release Engineering | Release Activity (with status, version info, QA links, etc)]]

== QA & Automation ==

* [[The_Yocto_Autobuilder| The Yocto Project Autobuilder]]
* [[QA| Yocto Project QA Main Page]]
* [[QA/Archive| Yocto Project QA Test Report Archive ]]

== Quick guide for newcomers ==

If you are new to the project and are willing to contribute, please refer to our [[Newcomers|guide for newcomers]].

== TSC ==
* Yocto Project Technical Steering Committee [[TSC]]

== Wiki reference sitemap ==
* [[Glossary]]
* [[Documentation Decoder]]
* [[Working Behind a Network Proxy]]
* [[FAQ]] and [[Technical FAQ]]. These need to be unified.
* [[Cookbook]] and [[TipsAndTricks | Tips and Tricks]]. Need clear messaging on how these should be differentiated.
* [[Developer Workflow Improvements]], including [[Nodejs Workflow Improvements]]
* [[Planning and Governance]]
* [[Community Guidelines]]
* [[Yocto Release Engineering | Yocto Project Release Engineering]]
* [[License Infrastructure Interest Group | License Infrastructure]]
* [[Processes and Activities]]
* [[Technical Contributors]]
* [[Projects]]
* [[Security]] - find out what we do about CVEs and security
* [[Yocto Interest Groups]]
* [[Testopia]] - The Yocto Project's community-opened test case management platform
* [[Toaster]] - the web interface
* [[YoctoProjectEvents]] - links to YoctoProject/OpenEmbedded related conferences and events
* [[Archive]] - Graveyard for out of date articles.

== Other resources ==
* [http://yoctoproject.org Yocto Project Front Page]
* [http://git.yoctoproject.org/ Yocto Project Git Source Repos]
* [https://bugzilla.yoctoproject.org/ Yocto Project Bugzilla]
* [https://www.yoctoproject.org/tools-resources/community/mailing-lists Yocto Project Mailing Lists]
* [http://recipes.yoctoproject.org/rrs Yocto Project Recipe Reporting System]
* [https://autobuilder.yoctoproject.org/typhoon Yocto Project Autobuilder]
* [http://downloads.yoctoproject.org/releases/yocto/ Yocto Project Releases Downloads]
* [http://autobuilder.yoctoproject.org/pub/nightly/ Yocto Project Nightly Build Images]
* [http://downloads.yoctoproject.org/mirror/sources/ Upstream Sources Mirror]
* [http://www.openembedded.org/wiki/Main_Page OpenEmbedded Wiki]
* [http://cgit.openembedded.org/ OpenEmbedded Git Repos]
* [http://layers.openembedded.org/ OpenEmbedded Community Layers]
* [http://patchwork.openembedded.org/ OpenEmbedded Patch Tracking System]
* '''IRC''': [https://libera.chat/ irc.libera.chat]
:* #yocto - Public discussions on the Yocto Project.
:* #oe - Public discussions on OpenEmbedded Core.

Main Page

2021-06-02T23:36:54Z

Nicolas Dechesne: /* Where to Next? */

== Welcome to the Yocto Project Wiki! ==
The [http://yoctoproject.org Yocto Project] is an open-source project that delivers a set of tools that create operating system images for embedded Linux systems. The Yocto Project tools are based on the [http://www.openembedded.org/wiki/Main_Page OpenEmbedded] (OE) project, which uses the BitBake build tool, to construct complete Linux images. BitBake and OE are combined to form a reference build host known as Poky which includes the following [[Core Components|core components]]. This [https://www.youtube.com/watch?v=utZpKM7i5Z4 video] will help explain what it's all about.

===Where to Start?===
If you're new to Yocto take a look at the '''[[Glossary]]''' so you're familiar with the terms used in this wiki and the project documentation. Then take a look at the [http://www.yoctoproject.org/docs/current/yocto-project-qs/yocto-project-qs.html '''Quick Start Guide''']. You can follow the steps in this document to clone the poky repository, quickly configure your build environment, and then try a build. Corporate firewalls can be problematic so network proxy configurations are detailed on the '''[[Working Behind a Network Proxy]]''' page. We advise you go straight for the [[Working_Behind_a_Network_Proxy#Option_2:_Chameleonsocks| Chameleonsocks option]].

===Where to Next?===
Thanks to the quick start guide, it's pretty easy to get your first Linux image and and running. Here are some places to look for help when improving your Yocto skills.
* The [https://linuxfoundation.org Linux Foundation] have some great [https://docs.google.com/presentation/d/1LmI3mHoD_Dzl8wplIYcUBrFF8BzDb_EadTvfbnpSK7Q/edit training slides]. There is also an [https://docs.google.com/presentation/d/1HoDtyN5SzlmuTN47ab4Y7w_i6c_VEW6EBUD944ntf38/edit#slide advanced slide deck] for more more experienced users.
* The first tool you'll need to get familiar with is '''bitbake''', so reading through the [https://www.yoctoproject.org/docs/current/bitbake-user-manual/bitbake-user-manual.html user manual] is recommended. You don't need to understand it all right now, but bookmark this page for reference. Implementation of bitake is covered in the [[Bitbake Internals Primer]]
* Once you start adding packages and configuring your image to create your own distribution, things can go wrong and it can hard to track down the root cause. There is no shortage of Yocto documentation resource, but if you're not exactly sure what you're looking for this '''[[Documentation Decoder]]''' will help you out. Also take a look at the [https://wiki.yoctoproject.org/wiki/Cookbook '''Cookbook'''] and [https://wiki.yoctoproject.org/wiki/Technical_FAQ troubleshooting guide]. Also [https://www.yoctoproject.org/blogs/jefro/2016/yocto-project-books these books] are helpful.
* Some new tools such as [http://www.yoctoproject.org/docs/current/toaster-manual/toaster-manual.html Toaster], [[Extensible SDK]] and [https://github.com/crops CROPS] are making it easier to get the best out of Yocto on Windows and Mac OS X. Take a look at the new workflow in [[Developer Workflow Improvements]].
* There is also a [https://wiki.yoctoproject.org/wiki/TipsAndTricks '''Tips and Tricks'''] section where more experienced developers contribute to articles that will help those new to Yocto Project.
* You can also read and participate on the [https://lists.yoctoproject.org mailing lists] - start with the [https://lists.yoctoproject.org/g/yocto main list] first - and the IRC channel. More information about the Yocto Project mailing lists, IRC and Matrix channels can be found [https://www.yoctoproject.org/community/mailing-lists/ here].

== Project planning ==

=== Features ===
* [[Yocto Feature Summary]] (Current and Next)

=== Project Planning for current release ===

* [[Planning]]

=== Project Status and Schedule ===
* [[Weekly_Status]]
* [[Yocto Project v2.8 Status]]
* [[Yocto 2.8 Schedule]]
* Testresults - https://git.yoctoproject.org/cgit/cgit.cgi/yocto-testresults-contrib/log/?h=intel-yocto-testresults

=== Future Directions ===
* [[Future_Directions]]

== Release Engineering ==

* [[Yocto Release Engineering | Yocto Project Release Engineering]]
* [[Yocto Release Engineering | Release Activity (with status, version info, QA links, etc)]]

== QA & Automation ==

* [[The_Yocto_Autobuilder| The Yocto Project Autobuilder]]
* [[QA| Yocto Project QA Main Page]]
* [[QA/Archive| Yocto Project QA Test Report Archive ]]

== Quick guide for newcomers ==

If you are new to the project and are willing to contribute, please refer to our [[Newcomers|guide for newcomers]].

== TSC ==
* Yocto Project Technical Steering Committee [[TSC]]

== Wiki reference sitemap ==
* [[Glossary]]
* [[Documentation Decoder]]
* [[Working Behind a Network Proxy]]
* [[FAQ]] and [[Technical FAQ]]. These need to be unified.
* [[Cookbook]] and [[TipsAndTricks | Tips and Tricks]]. Need clear messaging on how these should be differentiated.
* [[Developer Workflow Improvements]], including [[Nodejs Workflow Improvements]]
* [[Planning and Governance]]
* [[Community Guidelines]]
* [[Yocto Release Engineering | Yocto Project Release Engineering]]
* [[License Infrastructure Interest Group | License Infrastructure]]
* [[Processes and Activities]]
* [[Technical Contributors]]
* [[Projects]]
* [[Security]] - find out what we do about CVEs and security
* [[Yocto Interest Groups]]
* [[Testopia]] - The Yocto Project's community-opened test case management platform
* [[Toaster]] - the web interface
* [[YoctoProjectEvents]] - links to YoctoProject/OpenEmbedded related conferences and events
* [[Archive]] - Graveyard for out of date articles.

== Other resources ==
* [http://yoctoproject.org Yocto Project Front Page]
* [http://git.yoctoproject.org/ Yocto Project Git Source Repos]
* [https://bugzilla.yoctoproject.org/ Yocto Project Bugzilla]
* [https://www.yoctoproject.org/tools-resources/community/mailing-lists Yocto Project Mailing Lists]
* [http://recipes.yoctoproject.org/rrs Yocto Project Recipe Reporting System]
* [https://autobuilder.yoctoproject.org/typhoon Yocto Project Autobuilder]
* [http://downloads.yoctoproject.org/releases/yocto/ Yocto Project Releases Downloads]
* [http://autobuilder.yoctoproject.org/pub/nightly/ Yocto Project Nightly Build Images]
* [http://downloads.yoctoproject.org/mirror/sources/ Upstream Sources Mirror]
* [http://www.openembedded.org/wiki/Main_Page OpenEmbedded Wiki]
* [http://cgit.openembedded.org/ OpenEmbedded Git Repos]
* [http://layers.openembedded.org/ OpenEmbedded Community Layers]
* [http://patchwork.openembedded.org/ OpenEmbedded Patch Tracking System]
* '''IRC''': irc.freenode.net
:* #yocto - Public discussions on the Yocto Project.
:* #oe - Public discussions on OpenEmbedded Core.

YoctoProjectEvents

2021-05-14T07:08:31Z

Nicolas Dechesne: /* Events */

Traditionally the Linux Foundation organizes the Embedded Linux Conference -- one in North America (ELC) and one in Europe (ELCE). The Yocto Project often has a considerable presence at these events in the form of:
* talks submitted/presented about The Yocto Project, OpenEmbedded, and related technologies including a community BoF (Birds of a Feather) session
* a 1-day pre-event OpenEmbedded Developers Day (OEDAM [OpenEmbedded Developers Americas Meeting] for ELC and OEDEM [OpenEmbedded Developers European Meeting] for ELCE)
* a 1-day post-event DevDay
* a conference booth in the ELC/ELCE event hall, usually with demos

The ELC/ELCE events are often co-located with other LF-related events such as: the Linux Security Summit, Zephyr Summit, KVM Forum, Open Source Summit, etc… Starting around 2019, in an effort to better align some of these events, the early Embedded Linux Conference started moving later in the year, closing the gap between the ELC event and the ELCE event towards the end of the year and expanding the gap between the ELCE event and the ELC event. As a result there has been efforts to add YP/OE-specific events earlier in the year in order to maintain the roughly 6-month gap between events.

=== OE Developers Meeting ===
The OpenEmbedded Developers {Americas|European} Meeting is a 1-day event, organized by the OpenEmbedded members, for an informal, free-form round-table-style discussion of core issues of concern to core OE/YP developers, BSP maintainers, etc

=== DevDay ===
DevDay is a 1-day training event consisting of multiple tracks:
* a full-day beginners tutorial to get newcomers up-to-speed on basic YP/OE usage
* hands-on classes based on narrow topics of a more specialized nature

== Events ==
future events:
* [https://wiki.yoctoproject.org/wiki/YoctoProject_Summit_May2021 Yocto Project Summit May 2021]

past events:
* [https://wiki.yoctoproject.org/wiki/YP_Summit_Europe_2020 Yocto Project Summit Europe 2020]
* [https://wiki.yoctoproject.org/wiki/YP_DevDay_Austin_2020 DevDay Austin 2020]
* [https://wiki.yoctoproject.org/wiki/YP_Summit_Lyon_2019 Yocto Project Summit Lyon 2019]
* [https://www.socallinuxexpo.org/scale/17x/open-embedded-summit-0 OE Summit @ SCaLE 17x 2019]
* [https://wiki.yoctoproject.org/wiki/DevDay_San_Diego_2019 DevDay San Diego 2019]
* [https://wiki.yoctoproject.org/wiki/DevDay_Edinburgh_2018 DevDay Edinburgh 2018]
* [https://wiki.yoctoproject.org/wiki/DevDay_Portland_2018 DevDay Portland 2018]
* [https://wiki.yoctoproject.org/wiki/DevDay_Prague_2017 DevDay Prague 2017]
* [https://wiki.yoctoproject.org/wiki/DevDay_US_2017 DevDay US 2017]

for a list of YP/OE-related talks at ELC/ELCE see:
* [https://elinux.org/Buildsystems https://elinux.org/Buildsystems]

YoctoProjectEvents

2021-05-14T07:06:44Z

Nicolas Dechesne: /* Events */

Traditionally the Linux Foundation organizes the Embedded Linux Conference -- one in North America (ELC) and one in Europe (ELCE). The Yocto Project often has a considerable presence at these events in the form of:
* talks submitted/presented about The Yocto Project, OpenEmbedded, and related technologies including a community BoF (Birds of a Feather) session
* a 1-day pre-event OpenEmbedded Developers Day (OEDAM [OpenEmbedded Developers Americas Meeting] for ELC and OEDEM [OpenEmbedded Developers European Meeting] for ELCE)
* a 1-day post-event DevDay
* a conference booth in the ELC/ELCE event hall, usually with demos

The ELC/ELCE events are often co-located with other LF-related events such as: the Linux Security Summit, Zephyr Summit, KVM Forum, Open Source Summit, etc… Starting around 2019, in an effort to better align some of these events, the early Embedded Linux Conference started moving later in the year, closing the gap between the ELC event and the ELCE event towards the end of the year and expanding the gap between the ELCE event and the ELC event. As a result there has been efforts to add YP/OE-specific events earlier in the year in order to maintain the roughly 6-month gap between events.

=== OE Developers Meeting ===
The OpenEmbedded Developers {Americas|European} Meeting is a 1-day event, organized by the OpenEmbedded members, for an informal, free-form round-table-style discussion of core issues of concern to core OE/YP developers, BSP maintainers, etc

=== DevDay ===
DevDay is a 1-day training event consisting of multiple tracks:
* a full-day beginners tutorial to get newcomers up-to-speed on basic YP/OE usage
* hands-on classes based on narrow topics of a more specialized nature

== Events ==
future events:
* [https://wiki.yoctoproject.org/wiki/YoctoProject_Summit_May2021 Yocto Project Summit May 2021]

past events:
* [https://wiki.yoctoproject.org/wiki/YP_DevDay_Austin_2020 DevDay Austin 2020]
* [https://wiki.yoctoproject.org/wiki/YP_Summit_Lyon_2019 Yocto Project Summit Lyon 2019]
* [https://www.socallinuxexpo.org/scale/17x/open-embedded-summit-0 OE Summit @ SCaLE 17x 2019]
* [https://wiki.yoctoproject.org/wiki/DevDay_San_Diego_2019 DevDay San Diego 2019]
* [https://wiki.yoctoproject.org/wiki/DevDay_Edinburgh_2018 DevDay Edinburgh 2018]
* [https://wiki.yoctoproject.org/wiki/DevDay_Portland_2018 DevDay Portland 2018]
* [https://wiki.yoctoproject.org/wiki/DevDay_Prague_2017 DevDay Prague 2017]
* [https://wiki.yoctoproject.org/wiki/DevDay_US_2017 DevDay US 2017]

for a list of YP/OE-related talks at ELC/ELCE see:
* [https://elinux.org/Buildsystems https://elinux.org/Buildsystems]

YP DevDay Austin 2020

2021-05-14T07:04:52Z

Nicolas Dechesne:

This page contains slides for the talks in the Yocto Project DevDay Virtual, ELC Austin 2020

Starting at 9:00am, 02 July 2020, Mountain Time.

YP DevDay Virtual home page is here: [https://www.yoctoproject.org/yocto-project-dev-day-virtual-north-america-2020/ https://www.yoctoproject.org/yocto-project-dev-day-virtual-north-america-2020]

== Slides ==

* Presenter Slides

[https://wiki.yoctoproject.org/wiki/File:DD0_Introduction_NA20.pdf DD0_Introduction_NA20.pdf]
[https://wiki.yoctoproject.org/wiki/File:DD0_Introduction_NA20.pptx DD0_Introduction_NA20.pptx]
[https://wiki.yoctoproject.org/wiki/File:DD1_Highly_Scalable_Build_Automation_NA20.pdf DD1_Highly_Scalable_Build_Automation_NA20.pdf]
[https://wiki.yoctoproject.org/wiki/File:DD1_Highly_Scalable_Build_Automation_NA20.pptx DD1_Highly_Scalable_Build_Automation_NA20.pptx]
[https://wiki.yoctoproject.org/wiki/File:DD2_YoctoContainer-handout_NA20.pdf DD2_YoctoContainer-handout_NA20.pdf]
[https://wiki.yoctoproject.org/wiki/File:DD3_Kernel_Lab_NA20.pdf DD3_Kernel_Lab_NA20.pdf]
[https://wiki.yoctoproject.org/wiki/File:DD3_Kernel_Lab_NA20.pptx DD3_Kernel_Lab_NA20.pptx]
[https://wiki.yoctoproject.org/wiki/File:DD5_Security_Hardening_NA20.pdf DD5_Security_Hardening_NA20.pdf]
[https://wiki.yoctoproject.org/wiki/File:DD6_Size-Optimizations_NA20.pptx DD6_Size-Optimizations_NA20.pptx]
[https://wiki.yoctoproject.org/wiki/File:DD7_CI_CD_NA20.pptx DD7_CI_CD_NA20.pptx]
[https://wiki.yoctoproject.org/wiki/File:DD7_CI_CD_NA20.pdf DD7_CI_CD_NA20.pdf]
[https://wiki.yoctoproject.org/wiki/File:DD8_Userspace_Lab_NA20.pptx DD8_Userspace_Lab_NA20.pptx]
[https://wiki.yoctoproject.org/wiki/File:DD9_Devtool_NA20.odp DD9_Devtool_NA20.odp]
[https://wiki.yoctoproject.org/wiki/File:DD9_Devtool_NA20.pdf DD9_Devtool_NA20.pdf]
[https://wiki.yoctoproject.org/wiki/File:DD9_Devtool_NA20.pptx DD9_Devtool_NA20.pptx]
[https://wiki.yoctoproject.org/wiki/File:DD10_Xen_Hypervisor_NA20.pdf DD10_Xen_Hypervisor_NA20.pdf]
[https://wiki.yoctoproject.org/wiki/File:DD10_Virtualization_KVM_Hypervisor_NA20.pdf DD10_Virtualization_KVM_Hypervisor_NA20.pdf]

The presentations were recorded and the videos are available on the [https://www.youtube.com/watch?v=RSjQjPFFWig&list=PLD4M5FoHz-TxzQnVhNXI8OLiaEmNaW2Au Yocto Project Youtube channel]

* Class Movies, example code

[https://wiki.yoctoproject.org/wiki/File:Xen-guest-on-Rpi4.mp4 Xen-guest-on-Rpi4.mp4 (Virtualization)]
[https://wiki.yoctoproject.org/wiki/File:Xen-on-Rpi4.mp4 Xen-on-Rpi4.mp4 (Virtualization)]
[https://wiki.yoctoproject.org/wiki/File:DD10_KVM_guest_on_NUC.mov DD10_KVM_guest_on_NUC.mov]
[https://wiki.yoctoproject.org/wiki/File:Ubuntu KVM guest on NUC (Virtualization)]
[https://wiki.yoctoproject.org/wiki/File:Elc-austin.zip User Space sample source files]

* ELC Austin 2020 presentations

[https://wiki.yoctoproject.org/wiki/File:Yocto_Project_LTS_ELC_NA_June_2020.pptx YP LTS Presentation]

* Slides template

[https://wiki.yoctoproject.org/wiki/File:Yocto_Project_DevDay_NA2020_Template.pptx Yocto_Project_DevDay_NA2020_Template.pptx]

Extensible SDK

2020-10-19T10:43:19Z

Nicolas Dechesne: /* ELC 2017 */

'''This page is being continually updated with more content. Please bookmark and come back soon.''' Any questions, contact [mailto:henry.bruce@intel.com Henry Bruce]

== Introduction ==
The Yocto Project Extensible SDK (eSDK) has tools that allow you to easily add new applications and libraries to an image, modify the source of an existing component and test changes on the target hardware. The main benefit over the standard SDK is improved team workflow due to tighter integration with the OpenEmbedded build system.

Want to learn more about the Extensible SDK? You've come to the right place.

== Documentation ==
To learn about using the Extensible SDK read the [http://www.yoctoproject.org/docs/current/sdk-manual/sdk-manual.html SDK manual]. Information on building and customizing an eSDK can be found in [http://www.yoctoproject.org/docs/current/sdk-manual/sdk-manual.html#sdk-appendix-customizing Appendix B]. Also see this [http://events.linuxfoundation.org/sites/events/files/slides/2017%20ELC%20Henry%20Bruce.pdf eSDK talk] and companion [https://events.linuxfoundation.org/sites/events/files/slides/2017%20ELC%20--%20Using%20devtool%20to%20Streamline%20your%20Yocto%20Project%20Workflow.pdf devtool talk].

== Pre-built Installers ==
The Yocto Project auto-builder creates core-image-minimal and core-image-0sato eSDK installers for machines defined in oe-core. Currently only full installers are built. See http://autobuilder.yoctoproject.org/pub/releases/yocto-2.2.1/toolchain/x86_64. As an example [http://autobuilder.yoctoproject.org/pub/releases/yocto-2.2.1/toolchain/x86_64/poky-glibc-x86_64-core-image-sato-core2-64-toolchain-ext-2.2.1.sh poky-glibc-x86_64-core-image-sato-core2-64-toolchain-ext-2.2.1.sh] is the core-image-sato eSDK installer for machine genericx86-64.

== Tips and Tricks ==
* [[TipsAndTricks/Cmake,Eclipse,_and_SDKS | Add cmake to your eSDK]]

== Workflow ==
Details of the improved Yocto Project workflow enabled by using containers and the eSDK can be found on the [[Developer Workflow Improvements]] page.

== Presentations ==
=== ELC 2017 ===
* [https://elinux.org/images/7/7a/2017_ELC_Henry_Bruce.pdf Henry Bruce's eSDK talk] and [https://www.youtube.com/watch?v=d3xanDJuXRA video]
* [https://elinux.org/images/e/e2/2017_ELC_--_Using_devtool_to_Streamline_your_Yocto_Project_Workflow.pdf Tim Orling's devtool talk] and [https://www.youtube.com/watch?v=CiD7rB35CRE video]
* [https://elinux.org/images/9/94/2017_ELC_-_Yocto_Project_Containers.pdf Randy Witt's container talk] and [https://www.youtube.com/watch?v=JXHLAWveh7Y video]

== Releases ==
* First appeared as part of fido (1.8).
* Jethro (2.0) is beta release.
* Krogoth (2.1) is first usable release.
* Morty (2.2) brings "devtool finish" that generates patches to go with recipe
* Pyro (2.3) focuses on bug fixes
* Rocko (2.4) brings devtool (and underlying recipetool) enhancements

== Community ==
The eSDK is part of [http://www.openembedded.org/wiki/OpenEmbedded-Core Openembedded Core].
* Mailing list: openembedded-core@lists.openembedded.org
* Bugs: https://bugzilla.yoctoproject.org/buglist.cgi?quicksearch=extensible%20sdk&list_id=592532
* Yocto Project SDK QuickStart: https://mender.io/resources/yocto-sdk-quickstart

Documentation Sphinx

2020-09-16T15:03:36Z

Nicolas Dechesne: /* Tasks list */

== Yocto Project Documentation: Sphinx migration ==

The Yocto Project developers are currently investigating whether to move from Docbook to Sphinx (or any similar tools) for the project's documentation. The initial discussion can be found [https://lists.yoctoproject.org/g/docs/message/165 here].

Sphinx is a tool that makes it easy to create intelligent and beautiful documentation, written by Georg Brandl and licensed under the BSD license. It was originally created for the Python documentation.

See more on [https://www.sphinx-doc.org/en/master/ Sphinx website]. Furthermore Sphinx is designed to be [https://www.sphinx-doc.org/en/master/extdev/index.html extensible], and we can implement our own custom extensions, as Python modules, which will be executed during the generation of the documentation. This is a rather advance use of Sphinx, but could be very useful in the future.

This wiki page can be used as a working document to assist the discussions and/or the tasks. All discussions should be made on the [https://lists.yoctoproject.org/g/docs/ Yocto Project Docs mailing list].

== Sphinx migration for Yocto Project Documentation ==

The Sphinx migration is planned for 3.2 release, and the development is done in [https://git.yoctoproject.org/cgit/cgit.cgi/yocto-docs/log/?h=sphinx this branch], which produces the following [https://docs.yoctoproject.org/ HTML documentation]. Until the branch gets merged , patches should be sent against this branch.

A lot of work was already done to convert all the docbook files, but there is still some significant of work left to be done.

To use the proof of concept, please follow these instructions:

pip3 install sphinx
# to use the non default theme
pip3 install sphinx_rtd_theme
# your system may also need
pip3 install pyyaml

And then:

git clone https://git.yoctoproject.org/git/yocto-docs -b sphinx
cd yocto-docs/documentation
make -f Makefile.sphinx html

The resulting HTML index page will be _build/html/index.html.

== Sphinx proof of concept for Bitbake Documentation ==

A similar branch exists for the Bitbake documentation as well. The branch is available on [https://git.openembedded.org/bitbake/log/?h=sphinx here], and produces the following [http://docs.yoctoproject.org/bitbake/ HTML documentation].

To build the Bitbake documentation with Sphinx:

git clone https://git.openembedded.org/bitbake -b sphinx
cd bitbake/doc
make -f Makefile.sphinx html

== Design ideas / principles ==

=== Automated conversion ===

The initial Docbook to Sphinx migration can be done with automated tools such as [https://pandoc.org/ pandoc]. While the conversion is far from being perfect, it produces some clean output markdown text file. After the initial automated conversion developers will need to fix up at least headings, images and links. In addition Sphinx has built in mechanisms (directives) that should be used to replace similar functions implemented in Docbook such as glossary, variables substitutions, notes, ...

To illustrate how Pandoc can be used, the initial conversion for bsp-guide, ref-manual and overview-manual was done with the following commands:

for i in *.xml; do pandoc -f docbook -t rst $i -o $i.rst; done

=== Headings ===

The layout of the Yocto Project manuals is organized as follows

Book
Chapter
Section
Section
Section

During the conversion with Pandoc, some of the headings are not converted properly and need to be fixed manually. Let's propose to use the following headings styles:

Book => overline ===
Chapter => overline ***
Section => ====
Section => ----
Section => ^^^^
Section => """"

At least with this proposal, we keep the same TOCs with Sphinx and Docbook.

=== Built in glossary ===

Sphinx has a [https://www.sphinx-doc.org/en/master/usage/restructuredtext/directives.html#glossary glossary directive]. From the documentation:

This directive must contain a reST definition list with terms and definitions. The definitions will then be referencable with the [https://www.sphinx-doc.org/en/master/usage/restructuredtext/roles.html#role-term 'term' role].

So anywhere in *any* manual, we can do :term:`VAR` to refer to an item from the glossary, and a link is created.

=== Global substitutions ===

The Yocto Project documentation makes heavy use of 'global' variables. In Docbook these 'variables' are stored in the file [http://git.yoctoproject.org/cgit.cgi/yocto-docs/tree/documentation/poky.ent poky.ent]. This Docbook feature is not handled automatically with Pandoc. Sphinx has builtin support for [https://www.sphinx-doc.org/en/master/usage/restructuredtext/basics.html#substitutions substitutions] however they are local to each reST file by default. They can be made global by using ''rst_prolog'':

rst_prolog
A string of reStructuredText that will be included at the beginning of every source file that is read.

As such we will keep a documentation configuration file with all global substitutions, similar to ''poky.ent''. Here is a sample code to define variables:

rst_prolog = """
.. |DISTRO| replace:: 3.1
.. |DISTRO_COMPRESSED| replace:: 31
.. |DISTRO_NAME_NO_CAP| replace:: dunfell
.. |DISTRO_NAME| replace:: Dunfell
.. |DISTRO_NAME_NO_CAP_MINUS_ONE| replace:: zeus
.. |DISTRO_NAME_MINUS_ONE| replace:: Zeus
.. |YOCTO_DOC_VERSION| replace:: 3.1
.. |YOCTO_DOC_VERSION_MINUS_ONE| replace:: 3.0.2
.. |DISTRO_REL_TAG| replace:: yocto-3.1
.. |METAINTELVERSION| replace:: 12.0
.. |REL_MONTH_YEAR| replace:: April 2020
"""

and use them:

This is the manual for version: |DISTRO|, aka |DISTRO_NAME|.

However, there are serious shortcomings with Sphinx built-in, for example they can be used/nested inside code-block sections. Another mechanism was implemented so that we can continue to use 'variables' within our documentation. A Sphinx extension was implemented to support variable substitutions while "parsing" the .rst files. The pattern for variables substitutions is: ''{VAR}''. The implementation of the extension can be found here:
https://git.yoctoproject.org/cgit/cgit.cgi/yocto-docs/tree/documentation/sphinx/yocto-vars.py?h=sphinx. All variables are set in a file call poky.yaml, which was generated from poky.ent: https://git.yoctoproject.org/cgit/cgit.cgi/yocto-docs/tree/documentation/poky.yaml?h=sphinx. For example, the following .rst content will produce the 'expected' content:

.. code-block::
$ mkdir ~/poky-{DISTRO}
or
$ git clone {YOCTO_GIT_URL}/git/poky -b {DISTRO_NAME_NO_CAP}

Variables can be nested, like it was the case for DocBook.

=== Note directive ===

Sphinx has a builtin ''note'' directive that produces clean Note section in the output file. There are various types of directives such as "attention", "caution", "danger", "error", "hint", "important", "tip", "warning", "admonition" that are supported, and additional directive can be added as Sphinx extension if needed.

=== Figures ===

Our documentation has many figures/images. Somehow Pandoc has completely ignored all images during the conversion. It might be worth investigating why, even though it might take more time than manually fixing all documents... Sphinx as a ''figure'' directive which is straight forward to use. To include a figure in the body of the documentation:

.. image:: figures/YP-flow-diagram.png

=== Links and References ===

Please read: https://sublime-and-sphinx-guide.readthedocs.io/en/latest/references.html

You can include links to other locations in the same document, to locations in other documents and to external websites.

==== references ====

We enable this extension by default: [https://www.sphinx-doc.org/en/master/usage/extensions/autosectionlabel.html. sphinx.ext.autosectionlabel].

You can link from text to a heading in any other part of the document by using the :ref: command with the heading text as the parameter. For example, this text in another part of this document would link to this section:
:ref:`Cross-References to Locations in the Same Document`

We can use Custom link text or create custom anchor instead of using the section text:
Please check this :ref:`section <Cross-References to Locations in the Same Document>`

You can also create links/references in another .rst file, using the file name:
:ref:`overview-manual/overview-manual-development-environment:yocto project source repositories`

or with a custom link text:
:ref:`here <overview-manual/overview-manual-development-environment:yocto project source repositories>`

Here is a '''good tip''': You can use the following command to get Sphinx to dump all the references that exist:
python -msphinx.ext.intersphinx <path to build folder>/html/objects.inv

In this dump, you will find all links and for each link you will get the default "Link Text" that Sphinx will use. If the default link text is not appropriate, you can provide a custom link text in the '':ref:'' link.

==== external links ====
We enable the '[https://sublime-and-sphinx-guide.readthedocs.io/en/latest/references.html#use-the-external-links-extension extlinks]' extension by default, and it's configured with the following ``extlinks``:

'yocto_home': ('https://yoctoproject.org%s', None),
'yocto_wiki': ('https://wiki.yoctoproject.org%s', None),
'yocto_dl': ('https://downloads.yoctoproject.org%s', None),
'yocto_lists': ('https://lists.yoctoproject.org%s', None),
'yocto_bugs': ('https://bugzilla.yoctoproject.org%s', None),
'yocto_ab': ('https://autobuilder.yoctoproject.org%s', None),
'yocto_docs': ('https://docs.yoctoproject.org%s', None),
'yocto_git': ('https://git.yoctoproject.org%s', None),
'oe_home': ('https://www.openembedded.org%s', None),
'oe_lists': ('https://lists.openembedded.org%s', None),

In the documentation rst files, we can then do:

Please check this :yocto_wiki:`wiki page </Weekly_Status>`

==== intersphinx links ====

We also enable the [https://www.sphinx-doc.org/en/master/usage/extensions/intersphinx.html intersphinx extension] so that we can cross reference content from the bitbake manual for example. When building the YP docs, Sphinx will first download the objects.inv file from the Bitbake manual, so that it knows all the existing references/links set in the bitbake documentation. Then references to the bitbake manual can be done like this:

See the ":ref:`-D <bitbake:bitbake-user-manual/bitbake-user-manual-intro:usage and syntax>`" option

or

:term:`bitbake:BB_NUMBER_PARSE_THREADS`

== Tasks list ==

Here is an initial tasks list for the migration:

# Proof of concept
## <s>Initial proof of concept patches</s>
## Merge initial patch set in master, along with Docbook
## <s>Update initial patches with yocto-docs recent changes </s>
# Convert all YP manuals
## <s>Automated conversion with pandoc</s>
### <s>adt-manual</s>
### <s>brief-yoctoprojectqs</s>
### <s>dev-manual</s>
### <s>kernel-dev</s>
### <s>profile-manual</s>
### <s>sdk-manual</s>
### <s>toaster-manual</s>
### <s>test-manual</s>
### <s>bitbake-manual</s>
### <s>overview-manual</s>
## fix up all manuals for: links, figures, headings, notes, code blocks, global substitutions variables
### adt-manual
### <s>brief-yoctoprojectqs</s>
### <s>overview-manual</s>
### <s>dev-manual</s>
### <s>kernel-dev</s>
### <s>profile-manual</s>
### <s>sdk-manual</s>
### toaster-manual
### <s>test-manual</s>
### <s>bitbake-manual</s>
## Use current docs CSS file to create proper theme
### <s>initial import</s>
### tuning/optimization
## How to deal with each docbook 'first' page (license and TOC) (In progress RP and ndec looking at that)
## <s>Implement glossary</s>
# <s>Decide how to manage the mega manual. Should we create a single doc (index.rst) file per documentat, or a single one which ultimately would become the mega manual</s>
# <s>Custom Sphinx theme. The whole output can be themed using CSS and custom theme. We need to implement something that 'looks like' the Yocto Project and create beautiful rendering of our documentation</s>
# How to publish HTML online (per branch, latest, current, ...)? The whole process must be automated. (In progress, Michael on it)
# How to create PDF files?
# What's the impact of Google search and referencing?
# <s>Do we want to backport to dunfell LTS? (RP: Yes!)</s>
# <s>RP: Can we create a single page manual for ease of searching (some users need this and I personally use it a lot that way, I'm unlikely alone)</s>
# RP: Glossary headings don't appear to be externally linkable?

Documentation Sphinx

2020-09-15T15:05:16Z

Nicolas Dechesne: /* Tasks list */

== Yocto Project Documentation: Sphinx migration ==

The Yocto Project developers are currently investigating whether to move from Docbook to Sphinx (or any similar tools) for the project's documentation. The initial discussion can be found [https://lists.yoctoproject.org/g/docs/message/165 here].

Sphinx is a tool that makes it easy to create intelligent and beautiful documentation, written by Georg Brandl and licensed under the BSD license. It was originally created for the Python documentation.

See more on [https://www.sphinx-doc.org/en/master/ Sphinx website]. Furthermore Sphinx is designed to be [https://www.sphinx-doc.org/en/master/extdev/index.html extensible], and we can implement our own custom extensions, as Python modules, which will be executed during the generation of the documentation. This is a rather advance use of Sphinx, but could be very useful in the future.

This wiki page can be used as a working document to assist the discussions and/or the tasks. All discussions should be made on the [https://lists.yoctoproject.org/g/docs/ Yocto Project Docs mailing list].

== Sphinx migration for Yocto Project Documentation ==

The Sphinx migration is planned for 3.2 release, and the development is done in [https://git.yoctoproject.org/cgit/cgit.cgi/yocto-docs/log/?h=sphinx this branch], which produces the following [https://docs.yoctoproject.org/ HTML documentation]. Until the branch gets merged , patches should be sent against this branch.

A lot of work was already done to convert all the docbook files, but there is still some significant of work left to be done.

To use the proof of concept, please follow these instructions:

pip3 install sphinx
# to use the non default theme
pip3 install sphinx_rtd_theme
# your system may also need
pip3 install pyyaml

And then:

git clone https://git.yoctoproject.org/git/yocto-docs -b sphinx
cd yocto-docs/documentation
make -f Makefile.sphinx html

The resulting HTML index page will be _build/html/index.html.

== Sphinx proof of concept for Bitbake Documentation ==

A similar branch exists for the Bitbake documentation as well. The branch is available on [https://git.openembedded.org/bitbake/log/?h=sphinx here], and produces the following [http://docs.yoctoproject.org/bitbake/ HTML documentation].

To build the Bitbake documentation with Sphinx:

git clone https://git.openembedded.org/bitbake -b sphinx
cd bitbake/doc
make -f Makefile.sphinx html

== Design ideas / principles ==

=== Automated conversion ===

The initial Docbook to Sphinx migration can be done with automated tools such as [https://pandoc.org/ pandoc]. While the conversion is far from being perfect, it produces some clean output markdown text file. After the initial automated conversion developers will need to fix up at least headings, images and links. In addition Sphinx has built in mechanisms (directives) that should be used to replace similar functions implemented in Docbook such as glossary, variables substitutions, notes, ...

To illustrate how Pandoc can be used, the initial conversion for bsp-guide, ref-manual and overview-manual was done with the following commands:

for i in *.xml; do pandoc -f docbook -t rst $i -o $i.rst; done

=== Headings ===

The layout of the Yocto Project manuals is organized as follows

Book
Chapter
Section
Section
Section

During the conversion with Pandoc, some of the headings are not converted properly and need to be fixed manually. Let's propose to use the following headings styles:

Book => overline ===
Chapter => overline ***
Section => ====
Section => ----
Section => ^^^^
Section => """"

At least with this proposal, we keep the same TOCs with Sphinx and Docbook.

=== Built in glossary ===

Sphinx has a [https://www.sphinx-doc.org/en/master/usage/restructuredtext/directives.html#glossary glossary directive]. From the documentation:

This directive must contain a reST definition list with terms and definitions. The definitions will then be referencable with the [https://www.sphinx-doc.org/en/master/usage/restructuredtext/roles.html#role-term 'term' role].

So anywhere in *any* manual, we can do :term:`VAR` to refer to an item from the glossary, and a link is created.

=== Global substitutions ===

The Yocto Project documentation makes heavy use of 'global' variables. In Docbook these 'variables' are stored in the file [http://git.yoctoproject.org/cgit.cgi/yocto-docs/tree/documentation/poky.ent poky.ent]. This Docbook feature is not handled automatically with Pandoc. Sphinx has builtin support for [https://www.sphinx-doc.org/en/master/usage/restructuredtext/basics.html#substitutions substitutions] however they are local to each reST file by default. They can be made global by using ''rst_prolog'':

rst_prolog
A string of reStructuredText that will be included at the beginning of every source file that is read.

As such we will keep a documentation configuration file with all global substitutions, similar to ''poky.ent''. Here is a sample code to define variables:

rst_prolog = """
.. |DISTRO| replace:: 3.1
.. |DISTRO_COMPRESSED| replace:: 31
.. |DISTRO_NAME_NO_CAP| replace:: dunfell
.. |DISTRO_NAME| replace:: Dunfell
.. |DISTRO_NAME_NO_CAP_MINUS_ONE| replace:: zeus
.. |DISTRO_NAME_MINUS_ONE| replace:: Zeus
.. |YOCTO_DOC_VERSION| replace:: 3.1
.. |YOCTO_DOC_VERSION_MINUS_ONE| replace:: 3.0.2
.. |DISTRO_REL_TAG| replace:: yocto-3.1
.. |METAINTELVERSION| replace:: 12.0
.. |REL_MONTH_YEAR| replace:: April 2020
"""

and use them:

This is the manual for version: |DISTRO|, aka |DISTRO_NAME|.

However, there are serious shortcomings with Sphinx built-in, for example they can be used/nested inside code-block sections. Another mechanism was implemented so that we can continue to use 'variables' within our documentation. A Sphinx extension was implemented to support variable substitutions while "parsing" the .rst files. The pattern for variables substitutions is: ''{VAR}''. The implementation of the extension can be found here:
https://git.yoctoproject.org/cgit/cgit.cgi/yocto-docs/tree/documentation/sphinx/yocto-vars.py?h=sphinx. All variables are set in a file call poky.yaml, which was generated from poky.ent: https://git.yoctoproject.org/cgit/cgit.cgi/yocto-docs/tree/documentation/poky.yaml?h=sphinx. For example, the following .rst content will produce the 'expected' content:

.. code-block::
$ mkdir ~/poky-{DISTRO}
or
$ git clone {YOCTO_GIT_URL}/git/poky -b {DISTRO_NAME_NO_CAP}

Variables can be nested, like it was the case for DocBook.

=== Note directive ===

Sphinx has a builtin ''note'' directive that produces clean Note section in the output file. There are various types of directives such as "attention", "caution", "danger", "error", "hint", "important", "tip", "warning", "admonition" that are supported, and additional directive can be added as Sphinx extension if needed.

=== Figures ===

Our documentation has many figures/images. Somehow Pandoc has completely ignored all images during the conversion. It might be worth investigating why, even though it might take more time than manually fixing all documents... Sphinx as a ''figure'' directive which is straight forward to use. To include a figure in the body of the documentation:

.. image:: figures/YP-flow-diagram.png

=== Links and References ===

Please read: https://sublime-and-sphinx-guide.readthedocs.io/en/latest/references.html

You can include links to other locations in the same document, to locations in other documents and to external websites.

==== references ====

We enable this extension by default: [https://www.sphinx-doc.org/en/master/usage/extensions/autosectionlabel.html. sphinx.ext.autosectionlabel].

You can link from text to a heading in any other part of the document by using the :ref: command with the heading text as the parameter. For example, this text in another part of this document would link to this section:
:ref:`Cross-References to Locations in the Same Document`

We can use Custom link text or create custom anchor instead of using the section text:
Please check this :ref:`section <Cross-References to Locations in the Same Document>`

You can also create links/references in another .rst file, using the file name:
:ref:`overview-manual/overview-manual-development-environment:yocto project source repositories`

or with a custom link text:
:ref:`here <overview-manual/overview-manual-development-environment:yocto project source repositories>`

Here is a '''good tip''': You can use the following command to get Sphinx to dump all the references that exist:
python -msphinx.ext.intersphinx <path to build folder>/html/objects.inv

In this dump, you will find all links and for each link you will get the default "Link Text" that Sphinx will use. If the default link text is not appropriate, you can provide a custom link text in the '':ref:'' link.

==== external links ====
We enable the '[https://sublime-and-sphinx-guide.readthedocs.io/en/latest/references.html#use-the-external-links-extension extlinks]' extension by default, and it's configured with the following ``extlinks``:

'yocto_home': ('https://yoctoproject.org%s', None),
'yocto_wiki': ('https://wiki.yoctoproject.org%s', None),
'yocto_dl': ('https://downloads.yoctoproject.org%s', None),
'yocto_lists': ('https://lists.yoctoproject.org%s', None),
'yocto_bugs': ('https://bugzilla.yoctoproject.org%s', None),
'yocto_ab': ('https://autobuilder.yoctoproject.org%s', None),
'yocto_docs': ('https://docs.yoctoproject.org%s', None),
'yocto_git': ('https://git.yoctoproject.org%s', None),
'oe_home': ('https://www.openembedded.org%s', None),
'oe_lists': ('https://lists.openembedded.org%s', None),

In the documentation rst files, we can then do:

Please check this :yocto_wiki:`wiki page </Weekly_Status>`

==== intersphinx links ====

We also enable the [https://www.sphinx-doc.org/en/master/usage/extensions/intersphinx.html intersphinx extension] so that we can cross reference content from the bitbake manual for example. When building the YP docs, Sphinx will first download the objects.inv file from the Bitbake manual, so that it knows all the existing references/links set in the bitbake documentation. Then references to the bitbake manual can be done like this:

See the ":ref:`-D <bitbake:bitbake-user-manual/bitbake-user-manual-intro:usage and syntax>`" option

or

:term:`bitbake:BB_NUMBER_PARSE_THREADS`

== Tasks list ==

Here is an initial tasks list for the migration:

# Proof of concept
## <s>Initial proof of concept patches</s>
## Merge initial patch set in master, along with Docbook
## <s>Update initial patches with yocto-docs recent changes </s>
# Convert all YP manuals
## <s>Automated conversion with pandoc</s>
### <s>adt-manual</s>
### <s>brief-yoctoprojectqs</s>
### <s>dev-manual</s>
### <s>kernel-dev</s>
### <s>profile-manual</s>
### <s>sdk-manual</s>
### <s>toaster-manual</s>
### <s>test-manual</s>
### <s>bitbake-manual</s>
### <s>overview-manual</s>
## fix up all manuals for: links, figures, headings, notes, code blocks, global substitutions variables
### adt-manual
### <s>brief-yoctoprojectqs</s>
### overview-manual
### <s>dev-manual</s>
### kernel-dev
### <s>profile-manual</s>
### <s>sdk-manual</s>
### toaster-manual
### test-manual
### <s>bitbake-manual</s>
## Use current docs CSS file to create proper theme
### <s>initial import</s>
### tuning/optimization
## How to deal with each docbook 'first' page (license and TOC) (In progress RP and ndec looking at that)
## <s>Implement glossary</s>
# <s>Decide how to manage the mega manual. Should we create a single doc (index.rst) file per documentat, or a single one which ultimately would become the mega manual</s>
# <s>Custom Sphinx theme. The whole output can be themed using CSS and custom theme. We need to implement something that 'looks like' the Yocto Project and create beautiful rendering of our documentation</s>
# How to publish HTML online (per branch, latest, current, ...)? The whole process must be automated. (In progress, Michael on it)
# How to create PDF files?
# What's the impact of Google search and referencing?
# <s>Do we want to backport to dunfell LTS? (RP: Yes!)</s>
# <s>RP: Can we create a single page manual for ease of searching (some users need this and I personally use it a lot that way, I'm unlikely alone)</s>
# RP: Glossary headings don't appear to be externally linkable?

Documentation Sphinx

2020-09-14T21:02:14Z

Nicolas Dechesne: /* Tasks list */

== Yocto Project Documentation: Sphinx migration ==

The Yocto Project developers are currently investigating whether to move from Docbook to Sphinx (or any similar tools) for the project's documentation. The initial discussion can be found [https://lists.yoctoproject.org/g/docs/message/165 here].

Sphinx is a tool that makes it easy to create intelligent and beautiful documentation, written by Georg Brandl and licensed under the BSD license. It was originally created for the Python documentation.

See more on [https://www.sphinx-doc.org/en/master/ Sphinx website]. Furthermore Sphinx is designed to be [https://www.sphinx-doc.org/en/master/extdev/index.html extensible], and we can implement our own custom extensions, as Python modules, which will be executed during the generation of the documentation. This is a rather advance use of Sphinx, but could be very useful in the future.

This wiki page can be used as a working document to assist the discussions and/or the tasks. All discussions should be made on the [https://lists.yoctoproject.org/g/docs/ Yocto Project Docs mailing list].

== Sphinx migration for Yocto Project Documentation ==

The Sphinx migration is planned for 3.2 release, and the development is done in [https://git.yoctoproject.org/cgit/cgit.cgi/yocto-docs/log/?h=sphinx this branch], which produces the following [https://docs.yoctoproject.org/ HTML documentation]. Until the branch gets merged , patches should be sent against this branch.

A lot of work was already done to convert all the docbook files, but there is still some significant of work left to be done.

To use the proof of concept, please follow these instructions:

pip3 install sphinx
# to use the non default theme
pip3 install sphinx_rtd_theme
# your system may also need
pip3 install pyyaml

And then:

git clone https://git.yoctoproject.org/git/yocto-docs -b sphinx
cd yocto-docs/documentation
make -f Makefile.sphinx html

The resulting HTML index page will be _build/html/index.html.

== Sphinx proof of concept for Bitbake Documentation ==

A similar branch exists for the Bitbake documentation as well. The branch is available on [https://git.openembedded.org/bitbake/log/?h=sphinx here], and produces the following [http://docs.yoctoproject.org/bitbake/ HTML documentation].

To build the Bitbake documentation with Sphinx:

git clone https://git.openembedded.org/bitbake -b sphinx
cd bitbake/doc
make -f Makefile.sphinx html

== Design ideas / principles ==

=== Automated conversion ===

The initial Docbook to Sphinx migration can be done with automated tools such as [https://pandoc.org/ pandoc]. While the conversion is far from being perfect, it produces some clean output markdown text file. After the initial automated conversion developers will need to fix up at least headings, images and links. In addition Sphinx has built in mechanisms (directives) that should be used to replace similar functions implemented in Docbook such as glossary, variables substitutions, notes, ...

To illustrate how Pandoc can be used, the initial conversion for bsp-guide, ref-manual and overview-manual was done with the following commands:

for i in *.xml; do pandoc -f docbook -t rst $i -o $i.rst; done

=== Headings ===

The layout of the Yocto Project manuals is organized as follows

Book
Chapter
Section
Section
Section

During the conversion with Pandoc, some of the headings are not converted properly and need to be fixed manually. Let's propose to use the following headings styles:

Book => overline ===
Chapter => overline ***
Section => ====
Section => ----
Section => ^^^^
Section => """"

At least with this proposal, we keep the same TOCs with Sphinx and Docbook.

=== Built in glossary ===

Sphinx has a [https://www.sphinx-doc.org/en/master/usage/restructuredtext/directives.html#glossary glossary directive]. From the documentation:

This directive must contain a reST definition list with terms and definitions. The definitions will then be referencable with the [https://www.sphinx-doc.org/en/master/usage/restructuredtext/roles.html#role-term 'term' role].

So anywhere in *any* manual, we can do :term:`VAR` to refer to an item from the glossary, and a link is created.

=== Global substitutions ===

The Yocto Project documentation makes heavy use of 'global' variables. In Docbook these 'variables' are stored in the file [http://git.yoctoproject.org/cgit.cgi/yocto-docs/tree/documentation/poky.ent poky.ent]. This Docbook feature is not handled automatically with Pandoc. Sphinx has builtin support for [https://www.sphinx-doc.org/en/master/usage/restructuredtext/basics.html#substitutions substitutions] however they are local to each reST file by default. They can be made global by using ''rst_prolog'':

rst_prolog
A string of reStructuredText that will be included at the beginning of every source file that is read.

As such we will keep a documentation configuration file with all global substitutions, similar to ''poky.ent''. Here is a sample code to define variables:

rst_prolog = """
.. |DISTRO| replace:: 3.1
.. |DISTRO_COMPRESSED| replace:: 31
.. |DISTRO_NAME_NO_CAP| replace:: dunfell
.. |DISTRO_NAME| replace:: Dunfell
.. |DISTRO_NAME_NO_CAP_MINUS_ONE| replace:: zeus
.. |DISTRO_NAME_MINUS_ONE| replace:: Zeus
.. |YOCTO_DOC_VERSION| replace:: 3.1
.. |YOCTO_DOC_VERSION_MINUS_ONE| replace:: 3.0.2
.. |DISTRO_REL_TAG| replace:: yocto-3.1
.. |METAINTELVERSION| replace:: 12.0
.. |REL_MONTH_YEAR| replace:: April 2020
"""

and use them:

This is the manual for version: |DISTRO|, aka |DISTRO_NAME|.

However, there are serious shortcomings with Sphinx built-in, for example they can be used/nested inside code-block sections. Another mechanism was implemented so that we can continue to use 'variables' within our documentation. A Sphinx extension was implemented to support variable substitutions while "parsing" the .rst files. The pattern for variables substitutions is: ''{VAR}''. The implementation of the extension can be found here:
https://git.yoctoproject.org/cgit/cgit.cgi/yocto-docs/tree/documentation/sphinx/yocto-vars.py?h=sphinx. All variables are set in a file call poky.yaml, which was generated from poky.ent: https://git.yoctoproject.org/cgit/cgit.cgi/yocto-docs/tree/documentation/poky.yaml?h=sphinx. For example, the following .rst content will produce the 'expected' content:

.. code-block::
$ mkdir ~/poky-{DISTRO}
or
$ git clone {YOCTO_GIT_URL}/git/poky -b {DISTRO_NAME_NO_CAP}

Variables can be nested, like it was the case for DocBook.

=== Note directive ===

Sphinx has a builtin ''note'' directive that produces clean Note section in the output file. There are various types of directives such as "attention", "caution", "danger", "error", "hint", "important", "tip", "warning", "admonition" that are supported, and additional directive can be added as Sphinx extension if needed.

=== Figures ===

Our documentation has many figures/images. Somehow Pandoc has completely ignored all images during the conversion. It might be worth investigating why, even though it might take more time than manually fixing all documents... Sphinx as a ''figure'' directive which is straight forward to use. To include a figure in the body of the documentation:

.. image:: figures/YP-flow-diagram.png

=== Links and References ===

Please read: https://sublime-and-sphinx-guide.readthedocs.io/en/latest/references.html

You can include links to other locations in the same document, to locations in other documents and to external websites.

==== references ====

We enable this extension by default: [https://www.sphinx-doc.org/en/master/usage/extensions/autosectionlabel.html. sphinx.ext.autosectionlabel].

You can link from text to a heading in any other part of the document by using the :ref: command with the heading text as the parameter. For example, this text in another part of this document would link to this section:
:ref:`Cross-References to Locations in the Same Document`

We can use Custom link text or create custom anchor instead of using the section text:
Please check this :ref:`section <Cross-References to Locations in the Same Document>`

You can also create links/references in another .rst file, using the file name:
:ref:`overview-manual/overview-manual-development-environment:yocto project source repositories`

or with a custom link text:
:ref:`here <overview-manual/overview-manual-development-environment:yocto project source repositories>`

Here is a '''good tip''': You can use the following command to get Sphinx to dump all the references that exist:
python -msphinx.ext.intersphinx <path to build folder>/html/objects.inv

In this dump, you will find all links and for each link you will get the default "Link Text" that Sphinx will use. If the default link text is not appropriate, you can provide a custom link text in the '':ref:'' link.

==== external links ====
We enable the '[https://sublime-and-sphinx-guide.readthedocs.io/en/latest/references.html#use-the-external-links-extension extlinks]' extension by default, and it's configured with the following ``extlinks``:

'yocto_home': ('https://yoctoproject.org%s', None),
'yocto_wiki': ('https://wiki.yoctoproject.org%s', None),
'yocto_dl': ('https://downloads.yoctoproject.org%s', None),
'yocto_lists': ('https://lists.yoctoproject.org%s', None),
'yocto_bugs': ('https://bugzilla.yoctoproject.org%s', None),
'yocto_ab': ('https://autobuilder.yoctoproject.org%s', None),
'yocto_docs': ('https://docs.yoctoproject.org%s', None),
'yocto_git': ('https://git.yoctoproject.org%s', None),
'oe_home': ('https://www.openembedded.org%s', None),
'oe_lists': ('https://lists.openembedded.org%s', None),

In the documentation rst files, we can then do:

Please check this :yocto_wiki:`wiki page </Weekly_Status>`

==== intersphinx links ====

We also enable the [https://www.sphinx-doc.org/en/master/usage/extensions/intersphinx.html intersphinx extension] so that we can cross reference content from the bitbake manual for example. When building the YP docs, Sphinx will first download the objects.inv file from the Bitbake manual, so that it knows all the existing references/links set in the bitbake documentation. Then references to the bitbake manual can be done like this:

See the ":ref:`-D <bitbake:bitbake-user-manual/bitbake-user-manual-intro:usage and syntax>`" option

or

:term:`bitbake:BB_NUMBER_PARSE_THREADS`

== Tasks list ==

Here is an initial tasks list for the migration:

# Proof of concept
## <s>Initial proof of concept patches</s>
## Merge initial patch set in master, along with Docbook
## <s>Update initial patches with yocto-docs recent changes </s>
# Convert all YP manuals
## <s>Automated conversion with pandoc</s>
### <s>adt-manual</s>
### <s>brief-yoctoprojectqs</s>
### <s>dev-manual</s>
### <s>kernel-dev</s>
### <s>profile-manual</s>
### <s>sdk-manual</s>
### <s>toaster-manual</s>
### <s>test-manual</s>
### <s>bitbake-manual</s>
### <s>overview-manual</s>
## fix up all manuals for: links, figures, headings, notes, code blocks, global substitutions variables
### adt-manual
### <s>brief-yoctoprojectqs</s>
### overview-manual
### <s>dev-manual</s>
### kernel-dev
### profile-manual
### <s>sdk-manual</s>
### toaster-manual
### test-manual
### <s>bitbake-manual</s>
## Use current docs CSS file to create proper theme
### <s>initial import</s>
### tuning/optimization
## How to deal with each docbook 'first' page (license and TOC) (In progress RP and ndec looking at that)
## <s>Implement glossary</s>
# <s>Decide how to manage the mega manual. Should we create a single doc (index.rst) file per documentat, or a single one which ultimately would become the mega manual</s>
# <s>Custom Sphinx theme. The whole output can be themed using CSS and custom theme. We need to implement something that 'looks like' the Yocto Project and create beautiful rendering of our documentation</s>
# How to publish HTML online (per branch, latest, current, ...)? The whole process must be automated. (In progress, Michael on it)
# How to create PDF files?
# What's the impact of Google search and referencing?
# <s>Do we want to backport to dunfell LTS? (RP: Yes!)</s>
# <s>RP: Can we create a single page manual for ease of searching (some users need this and I personally use it a lot that way, I'm unlikely alone)</s>
# RP: Glossary headings don't appear to be externally linkable?

Documentation Sphinx

2020-09-14T21:01:35Z

Nicolas Dechesne: /* Tasks list */

== Yocto Project Documentation: Sphinx migration ==

The Yocto Project developers are currently investigating whether to move from Docbook to Sphinx (or any similar tools) for the project's documentation. The initial discussion can be found [https://lists.yoctoproject.org/g/docs/message/165 here].

Sphinx is a tool that makes it easy to create intelligent and beautiful documentation, written by Georg Brandl and licensed under the BSD license. It was originally created for the Python documentation.

See more on [https://www.sphinx-doc.org/en/master/ Sphinx website]. Furthermore Sphinx is designed to be [https://www.sphinx-doc.org/en/master/extdev/index.html extensible], and we can implement our own custom extensions, as Python modules, which will be executed during the generation of the documentation. This is a rather advance use of Sphinx, but could be very useful in the future.

This wiki page can be used as a working document to assist the discussions and/or the tasks. All discussions should be made on the [https://lists.yoctoproject.org/g/docs/ Yocto Project Docs mailing list].

== Sphinx migration for Yocto Project Documentation ==

The Sphinx migration is planned for 3.2 release, and the development is done in [https://git.yoctoproject.org/cgit/cgit.cgi/yocto-docs/log/?h=sphinx this branch], which produces the following [https://docs.yoctoproject.org/ HTML documentation]. Until the branch gets merged , patches should be sent against this branch.

A lot of work was already done to convert all the docbook files, but there is still some significant of work left to be done.

To use the proof of concept, please follow these instructions:

pip3 install sphinx
# to use the non default theme
pip3 install sphinx_rtd_theme
# your system may also need
pip3 install pyyaml

And then:

git clone https://git.yoctoproject.org/git/yocto-docs -b sphinx
cd yocto-docs/documentation
make -f Makefile.sphinx html

The resulting HTML index page will be _build/html/index.html.

== Sphinx proof of concept for Bitbake Documentation ==

A similar branch exists for the Bitbake documentation as well. The branch is available on [https://git.openembedded.org/bitbake/log/?h=sphinx here], and produces the following [http://docs.yoctoproject.org/bitbake/ HTML documentation].

To build the Bitbake documentation with Sphinx:

git clone https://git.openembedded.org/bitbake -b sphinx
cd bitbake/doc
make -f Makefile.sphinx html

== Design ideas / principles ==

=== Automated conversion ===

The initial Docbook to Sphinx migration can be done with automated tools such as [https://pandoc.org/ pandoc]. While the conversion is far from being perfect, it produces some clean output markdown text file. After the initial automated conversion developers will need to fix up at least headings, images and links. In addition Sphinx has built in mechanisms (directives) that should be used to replace similar functions implemented in Docbook such as glossary, variables substitutions, notes, ...

To illustrate how Pandoc can be used, the initial conversion for bsp-guide, ref-manual and overview-manual was done with the following commands:

for i in *.xml; do pandoc -f docbook -t rst $i -o $i.rst; done

=== Headings ===

The layout of the Yocto Project manuals is organized as follows

Book
Chapter
Section
Section
Section

During the conversion with Pandoc, some of the headings are not converted properly and need to be fixed manually. Let's propose to use the following headings styles:

Book => overline ===
Chapter => overline ***
Section => ====
Section => ----
Section => ^^^^
Section => """"

At least with this proposal, we keep the same TOCs with Sphinx and Docbook.

=== Built in glossary ===

Sphinx has a [https://www.sphinx-doc.org/en/master/usage/restructuredtext/directives.html#glossary glossary directive]. From the documentation:

This directive must contain a reST definition list with terms and definitions. The definitions will then be referencable with the [https://www.sphinx-doc.org/en/master/usage/restructuredtext/roles.html#role-term 'term' role].

So anywhere in *any* manual, we can do :term:`VAR` to refer to an item from the glossary, and a link is created.

=== Global substitutions ===

The Yocto Project documentation makes heavy use of 'global' variables. In Docbook these 'variables' are stored in the file [http://git.yoctoproject.org/cgit.cgi/yocto-docs/tree/documentation/poky.ent poky.ent]. This Docbook feature is not handled automatically with Pandoc. Sphinx has builtin support for [https://www.sphinx-doc.org/en/master/usage/restructuredtext/basics.html#substitutions substitutions] however they are local to each reST file by default. They can be made global by using ''rst_prolog'':

rst_prolog
A string of reStructuredText that will be included at the beginning of every source file that is read.

As such we will keep a documentation configuration file with all global substitutions, similar to ''poky.ent''. Here is a sample code to define variables:

rst_prolog = """
.. |DISTRO| replace:: 3.1
.. |DISTRO_COMPRESSED| replace:: 31
.. |DISTRO_NAME_NO_CAP| replace:: dunfell
.. |DISTRO_NAME| replace:: Dunfell
.. |DISTRO_NAME_NO_CAP_MINUS_ONE| replace:: zeus
.. |DISTRO_NAME_MINUS_ONE| replace:: Zeus
.. |YOCTO_DOC_VERSION| replace:: 3.1
.. |YOCTO_DOC_VERSION_MINUS_ONE| replace:: 3.0.2
.. |DISTRO_REL_TAG| replace:: yocto-3.1
.. |METAINTELVERSION| replace:: 12.0
.. |REL_MONTH_YEAR| replace:: April 2020
"""

and use them:

This is the manual for version: |DISTRO|, aka |DISTRO_NAME|.

However, there are serious shortcomings with Sphinx built-in, for example they can be used/nested inside code-block sections. Another mechanism was implemented so that we can continue to use 'variables' within our documentation. A Sphinx extension was implemented to support variable substitutions while "parsing" the .rst files. The pattern for variables substitutions is: ''{VAR}''. The implementation of the extension can be found here:
https://git.yoctoproject.org/cgit/cgit.cgi/yocto-docs/tree/documentation/sphinx/yocto-vars.py?h=sphinx. All variables are set in a file call poky.yaml, which was generated from poky.ent: https://git.yoctoproject.org/cgit/cgit.cgi/yocto-docs/tree/documentation/poky.yaml?h=sphinx. For example, the following .rst content will produce the 'expected' content:

.. code-block::
$ mkdir ~/poky-{DISTRO}
or
$ git clone {YOCTO_GIT_URL}/git/poky -b {DISTRO_NAME_NO_CAP}

Variables can be nested, like it was the case for DocBook.

=== Note directive ===

Sphinx has a builtin ''note'' directive that produces clean Note section in the output file. There are various types of directives such as "attention", "caution", "danger", "error", "hint", "important", "tip", "warning", "admonition" that are supported, and additional directive can be added as Sphinx extension if needed.

=== Figures ===

Our documentation has many figures/images. Somehow Pandoc has completely ignored all images during the conversion. It might be worth investigating why, even though it might take more time than manually fixing all documents... Sphinx as a ''figure'' directive which is straight forward to use. To include a figure in the body of the documentation:

.. image:: figures/YP-flow-diagram.png

=== Links and References ===

Please read: https://sublime-and-sphinx-guide.readthedocs.io/en/latest/references.html

You can include links to other locations in the same document, to locations in other documents and to external websites.

==== references ====

We enable this extension by default: [https://www.sphinx-doc.org/en/master/usage/extensions/autosectionlabel.html. sphinx.ext.autosectionlabel].

You can link from text to a heading in any other part of the document by using the :ref: command with the heading text as the parameter. For example, this text in another part of this document would link to this section:
:ref:`Cross-References to Locations in the Same Document`

We can use Custom link text or create custom anchor instead of using the section text:
Please check this :ref:`section <Cross-References to Locations in the Same Document>`

You can also create links/references in another .rst file, using the file name:
:ref:`overview-manual/overview-manual-development-environment:yocto project source repositories`

or with a custom link text:
:ref:`here <overview-manual/overview-manual-development-environment:yocto project source repositories>`

Here is a '''good tip''': You can use the following command to get Sphinx to dump all the references that exist:
python -msphinx.ext.intersphinx <path to build folder>/html/objects.inv

In this dump, you will find all links and for each link you will get the default "Link Text" that Sphinx will use. If the default link text is not appropriate, you can provide a custom link text in the '':ref:'' link.

==== external links ====
We enable the '[https://sublime-and-sphinx-guide.readthedocs.io/en/latest/references.html#use-the-external-links-extension extlinks]' extension by default, and it's configured with the following ``extlinks``:

'yocto_home': ('https://yoctoproject.org%s', None),
'yocto_wiki': ('https://wiki.yoctoproject.org%s', None),
'yocto_dl': ('https://downloads.yoctoproject.org%s', None),
'yocto_lists': ('https://lists.yoctoproject.org%s', None),
'yocto_bugs': ('https://bugzilla.yoctoproject.org%s', None),
'yocto_ab': ('https://autobuilder.yoctoproject.org%s', None),
'yocto_docs': ('https://docs.yoctoproject.org%s', None),
'yocto_git': ('https://git.yoctoproject.org%s', None),
'oe_home': ('https://www.openembedded.org%s', None),
'oe_lists': ('https://lists.openembedded.org%s', None),

In the documentation rst files, we can then do:

Please check this :yocto_wiki:`wiki page </Weekly_Status>`

==== intersphinx links ====

We also enable the [https://www.sphinx-doc.org/en/master/usage/extensions/intersphinx.html intersphinx extension] so that we can cross reference content from the bitbake manual for example. When building the YP docs, Sphinx will first download the objects.inv file from the Bitbake manual, so that it knows all the existing references/links set in the bitbake documentation. Then references to the bitbake manual can be done like this:

See the ":ref:`-D <bitbake:bitbake-user-manual/bitbake-user-manual-intro:usage and syntax>`" option

or

:term:`bitbake:BB_NUMBER_PARSE_THREADS`

== Tasks list ==

Here is an initial tasks list for the migration:

# Proof of concept
## <s>Initial proof of concept patches</s>
## Merge initial patch set in master, along with Docbook
## <s>Update initial patches with yocto-docs recent changes </s>
# Convert all YP manuals
## <s>Automated conversion with pandoc</s>
### <s>adt-manual</s>
### <s>brief-yoctoprojectqs</s>
### <s>dev-manual</s>
### <s>kernel-dev</s>
### <s>profile-manual</s>
### <s>sdk-manual</s>
### <s>toaster-manual</s>
### <s>test-manual</s>
### <s>bitbake-manual</s>
## fix up all manuals for: links, figures, headings, notes, code blocks, global substitutions variables
### adt-manual
### <s>brief-yoctoprojectqs</s>
### <s>dev-manual</s>
### kernel-dev
### profile-manual
### <s>sdk-manual</s>
### toaster-manual
### test-manual
### <s>bitbake-manual</s>
## Use current docs CSS file to create proper theme
### <s>initial import</s>
### tuning/optimization
## How to deal with each docbook 'first' page (license and TOC) (In progress RP and ndec looking at that)
## <s>Implement glossary</s>
# <s>Decide how to manage the mega manual. Should we create a single doc (index.rst) file per documentat, or a single one which ultimately would become the mega manual</s>
# <s>Custom Sphinx theme. The whole output can be themed using CSS and custom theme. We need to implement something that 'looks like' the Yocto Project and create beautiful rendering of our documentation</s>
# How to publish HTML online (per branch, latest, current, ...)? The whole process must be automated. (In progress, Michael on it)
# How to create PDF files?
# What's the impact of Google search and referencing?
# <s>Do we want to backport to dunfell LTS? (RP: Yes!)</s>
# <s>RP: Can we create a single page manual for ease of searching (some users need this and I personally use it a lot that way, I'm unlikely alone)</s>
# RP: Glossary headings don't appear to be externally linkable?

Documentation Sphinx

2020-09-14T20:56:53Z

Nicolas Dechesne: /* Sphinx proof of concept for Bitbake Documentation */

== Yocto Project Documentation: Sphinx migration ==

The Yocto Project developers are currently investigating whether to move from Docbook to Sphinx (or any similar tools) for the project's documentation. The initial discussion can be found [https://lists.yoctoproject.org/g/docs/message/165 here].

Sphinx is a tool that makes it easy to create intelligent and beautiful documentation, written by Georg Brandl and licensed under the BSD license. It was originally created for the Python documentation.

See more on [https://www.sphinx-doc.org/en/master/ Sphinx website]. Furthermore Sphinx is designed to be [https://www.sphinx-doc.org/en/master/extdev/index.html extensible], and we can implement our own custom extensions, as Python modules, which will be executed during the generation of the documentation. This is a rather advance use of Sphinx, but could be very useful in the future.

This wiki page can be used as a working document to assist the discussions and/or the tasks. All discussions should be made on the [https://lists.yoctoproject.org/g/docs/ Yocto Project Docs mailing list].

== Sphinx migration for Yocto Project Documentation ==

The Sphinx migration is planned for 3.2 release, and the development is done in [https://git.yoctoproject.org/cgit/cgit.cgi/yocto-docs/log/?h=sphinx this branch], which produces the following [https://docs.yoctoproject.org/ HTML documentation]. Until the branch gets merged , patches should be sent against this branch.

A lot of work was already done to convert all the docbook files, but there is still some significant of work left to be done.

To use the proof of concept, please follow these instructions:

pip3 install sphinx
# to use the non default theme
pip3 install sphinx_rtd_theme
# your system may also need
pip3 install pyyaml

And then:

git clone https://git.yoctoproject.org/git/yocto-docs -b sphinx
cd yocto-docs/documentation
make -f Makefile.sphinx html

The resulting HTML index page will be _build/html/index.html.

== Sphinx proof of concept for Bitbake Documentation ==

A similar branch exists for the Bitbake documentation as well. The branch is available on [https://git.openembedded.org/bitbake/log/?h=sphinx here], and produces the following [http://docs.yoctoproject.org/bitbake/ HTML documentation].

To build the Bitbake documentation with Sphinx:

git clone https://git.openembedded.org/bitbake -b sphinx
cd bitbake/doc
make -f Makefile.sphinx html

== Design ideas / principles ==

=== Automated conversion ===

The initial Docbook to Sphinx migration can be done with automated tools such as [https://pandoc.org/ pandoc]. While the conversion is far from being perfect, it produces some clean output markdown text file. After the initial automated conversion developers will need to fix up at least headings, images and links. In addition Sphinx has built in mechanisms (directives) that should be used to replace similar functions implemented in Docbook such as glossary, variables substitutions, notes, ...

To illustrate how Pandoc can be used, the initial conversion for bsp-guide, ref-manual and overview-manual was done with the following commands:

for i in *.xml; do pandoc -f docbook -t rst $i -o $i.rst; done

=== Headings ===

The layout of the Yocto Project manuals is organized as follows

Book
Chapter
Section
Section
Section

During the conversion with Pandoc, some of the headings are not converted properly and need to be fixed manually. Let's propose to use the following headings styles:

Book => overline ===
Chapter => overline ***
Section => ====
Section => ----
Section => ^^^^
Section => """"

At least with this proposal, we keep the same TOCs with Sphinx and Docbook.

=== Built in glossary ===

Sphinx has a [https://www.sphinx-doc.org/en/master/usage/restructuredtext/directives.html#glossary glossary directive]. From the documentation:

This directive must contain a reST definition list with terms and definitions. The definitions will then be referencable with the [https://www.sphinx-doc.org/en/master/usage/restructuredtext/roles.html#role-term 'term' role].

So anywhere in *any* manual, we can do :term:`VAR` to refer to an item from the glossary, and a link is created.

=== Global substitutions ===

The Yocto Project documentation makes heavy use of 'global' variables. In Docbook these 'variables' are stored in the file [http://git.yoctoproject.org/cgit.cgi/yocto-docs/tree/documentation/poky.ent poky.ent]. This Docbook feature is not handled automatically with Pandoc. Sphinx has builtin support for [https://www.sphinx-doc.org/en/master/usage/restructuredtext/basics.html#substitutions substitutions] however they are local to each reST file by default. They can be made global by using ''rst_prolog'':

rst_prolog
A string of reStructuredText that will be included at the beginning of every source file that is read.

As such we will keep a documentation configuration file with all global substitutions, similar to ''poky.ent''. Here is a sample code to define variables:

rst_prolog = """
.. |DISTRO| replace:: 3.1
.. |DISTRO_COMPRESSED| replace:: 31
.. |DISTRO_NAME_NO_CAP| replace:: dunfell
.. |DISTRO_NAME| replace:: Dunfell
.. |DISTRO_NAME_NO_CAP_MINUS_ONE| replace:: zeus
.. |DISTRO_NAME_MINUS_ONE| replace:: Zeus
.. |YOCTO_DOC_VERSION| replace:: 3.1
.. |YOCTO_DOC_VERSION_MINUS_ONE| replace:: 3.0.2
.. |DISTRO_REL_TAG| replace:: yocto-3.1
.. |METAINTELVERSION| replace:: 12.0
.. |REL_MONTH_YEAR| replace:: April 2020
"""

and use them:

This is the manual for version: |DISTRO|, aka |DISTRO_NAME|.

However, there are serious shortcomings with Sphinx built-in, for example they can be used/nested inside code-block sections. Another mechanism was implemented so that we can continue to use 'variables' within our documentation. A Sphinx extension was implemented to support variable substitutions while "parsing" the .rst files. The pattern for variables substitutions is: ''{VAR}''. The implementation of the extension can be found here:
https://git.yoctoproject.org/cgit/cgit.cgi/yocto-docs/tree/documentation/sphinx/yocto-vars.py?h=sphinx. All variables are set in a file call poky.yaml, which was generated from poky.ent: https://git.yoctoproject.org/cgit/cgit.cgi/yocto-docs/tree/documentation/poky.yaml?h=sphinx. For example, the following .rst content will produce the 'expected' content:

.. code-block::
$ mkdir ~/poky-{DISTRO}
or
$ git clone {YOCTO_GIT_URL}/git/poky -b {DISTRO_NAME_NO_CAP}

Variables can be nested, like it was the case for DocBook.

=== Note directive ===

Sphinx has a builtin ''note'' directive that produces clean Note section in the output file. There are various types of directives such as "attention", "caution", "danger", "error", "hint", "important", "tip", "warning", "admonition" that are supported, and additional directive can be added as Sphinx extension if needed.

=== Figures ===

Our documentation has many figures/images. Somehow Pandoc has completely ignored all images during the conversion. It might be worth investigating why, even though it might take more time than manually fixing all documents... Sphinx as a ''figure'' directive which is straight forward to use. To include a figure in the body of the documentation:

.. image:: figures/YP-flow-diagram.png

=== Links and References ===

Please read: https://sublime-and-sphinx-guide.readthedocs.io/en/latest/references.html

You can include links to other locations in the same document, to locations in other documents and to external websites.

==== references ====

We enable this extension by default: [https://www.sphinx-doc.org/en/master/usage/extensions/autosectionlabel.html. sphinx.ext.autosectionlabel].

You can link from text to a heading in any other part of the document by using the :ref: command with the heading text as the parameter. For example, this text in another part of this document would link to this section:
:ref:`Cross-References to Locations in the Same Document`

We can use Custom link text or create custom anchor instead of using the section text:
Please check this :ref:`section <Cross-References to Locations in the Same Document>`

You can also create links/references in another .rst file, using the file name:
:ref:`overview-manual/overview-manual-development-environment:yocto project source repositories`

or with a custom link text:
:ref:`here <overview-manual/overview-manual-development-environment:yocto project source repositories>`

Here is a '''good tip''': You can use the following command to get Sphinx to dump all the references that exist:
python -msphinx.ext.intersphinx <path to build folder>/html/objects.inv

In this dump, you will find all links and for each link you will get the default "Link Text" that Sphinx will use. If the default link text is not appropriate, you can provide a custom link text in the '':ref:'' link.

==== external links ====
We enable the '[https://sublime-and-sphinx-guide.readthedocs.io/en/latest/references.html#use-the-external-links-extension extlinks]' extension by default, and it's configured with the following ``extlinks``:

'yocto_home': ('https://yoctoproject.org%s', None),
'yocto_wiki': ('https://wiki.yoctoproject.org%s', None),
'yocto_dl': ('https://downloads.yoctoproject.org%s', None),
'yocto_lists': ('https://lists.yoctoproject.org%s', None),
'yocto_bugs': ('https://bugzilla.yoctoproject.org%s', None),
'yocto_ab': ('https://autobuilder.yoctoproject.org%s', None),
'yocto_docs': ('https://docs.yoctoproject.org%s', None),
'yocto_git': ('https://git.yoctoproject.org%s', None),
'oe_home': ('https://www.openembedded.org%s', None),
'oe_lists': ('https://lists.openembedded.org%s', None),

In the documentation rst files, we can then do:

Please check this :yocto_wiki:`wiki page </Weekly_Status>`

==== intersphinx links ====

We also enable the [https://www.sphinx-doc.org/en/master/usage/extensions/intersphinx.html intersphinx extension] so that we can cross reference content from the bitbake manual for example. When building the YP docs, Sphinx will first download the objects.inv file from the Bitbake manual, so that it knows all the existing references/links set in the bitbake documentation. Then references to the bitbake manual can be done like this:

See the ":ref:`-D <bitbake:bitbake-user-manual/bitbake-user-manual-intro:usage and syntax>`" option

or

:term:`bitbake:BB_NUMBER_PARSE_THREADS`

== Tasks list ==

Here is an initial tasks list for the migration:

# Proof of concept
## <s>Initial proof of concept patches</s>
## Merge initial patch set in master, along with Docbook
## <s>Update initial patches with yocto-docs recent changes </s>
# Convert all YP manuals
## <s>Automated conversion with pandoc</s>
### <s>adt-manual</s>
### <s>brief-yoctoprojectqs</s>
### <s>dev-manual</s>
### <s>kernel-dev</s>
### <s>profile-manual</s>
### <s>sdk-manual</s>
### <s>toaster-manual</s>
### <s>test-manual</s>
### <s>bitbake-manual</s>
## fix up all manuals for:
### links
### <s>figures</s>
### <s>headings</s>
### notes
### code blocks
### global substitutions variables
## Use current docs CSS file to create proper theme
### <s>initial import</s>
### tuning/optimization
## How to deal with each docbook 'first' page (license and TOC) (In progress RP and ndec looking at that)
## <s>Implement glossary</s>
# <s>Decide how to manage the mega manual. Should we create a single doc (index.rst) file per documentat, or a single one which ultimately would become the mega manual</s>
# <s>Custom Sphinx theme. The whole output can be themed using CSS and custom theme. We need to implement something that 'looks like' the Yocto Project and create beautiful rendering of our documentation</s>
# How to publish HTML online (per branch, latest, current, ...)? The whole process must be automated. (In progress, Michael on it)
# How to create PDF files?
# What's the impact of Google search and referencing?
# <s>Do we want to backport to dunfell LTS? (RP: Yes!)</s>
# <s>RP: Can we create a single page manual for ease of searching (some users need this and I personally use it a lot that way, I'm unlikely alone)</s>
# RP: Glossary headings don't appear to be externally linkable?

Documentation Sphinx

2020-09-11T07:40:21Z

Nicolas Dechesne: /* intersphinx links */

== Yocto Project Documentation: Sphinx migration ==

The Yocto Project developers are currently investigating whether to move from Docbook to Sphinx (or any similar tools) for the project's documentation. The initial discussion can be found [https://lists.yoctoproject.org/g/docs/message/165 here].

Sphinx is a tool that makes it easy to create intelligent and beautiful documentation, written by Georg Brandl and licensed under the BSD license. It was originally created for the Python documentation.

See more on [https://www.sphinx-doc.org/en/master/ Sphinx website]. Furthermore Sphinx is designed to be [https://www.sphinx-doc.org/en/master/extdev/index.html extensible], and we can implement our own custom extensions, as Python modules, which will be executed during the generation of the documentation. This is a rather advance use of Sphinx, but could be very useful in the future.

This wiki page can be used as a working document to assist the discussions and/or the tasks. All discussions should be made on the [https://lists.yoctoproject.org/g/docs/ Yocto Project Docs mailing list].

== Sphinx migration for Yocto Project Documentation ==

The Sphinx migration is planned for 3.2 release, and the development is done in [https://git.yoctoproject.org/cgit/cgit.cgi/yocto-docs/log/?h=sphinx this branch], which produces the following [https://docs.yoctoproject.org/ HTML documentation]. Until the branch gets merged , patches should be sent against this branch.

A lot of work was already done to convert all the docbook files, but there is still some significant of work left to be done.

To use the proof of concept, please follow these instructions:

pip3 install sphinx
# to use the non default theme
pip3 install sphinx_rtd_theme

And then:

git clone https://git.yoctoproject.org/git/yocto-docs -b sphinx
cd yocto-docs/documentation
make -f Makefile.sphinx html

The resulting HTML index page will be _build/html/index.html.

== Sphinx proof of concept for Bitbake Documentation ==

A similar branch exists for the Bitbake documentation as well. The branch is available on [https://git.openembedded.org/bitbake/log/?h=sphinx here], and produces the following [https://people.linaro.org/~nicolas.dechesne/bitbake-docs/html/ HTML documentation].

To build the Bitbake documentation with Sphinx:

git clone https://git.openembedded.org/bitbake -b sphinx
cd bitbake/doc
make -f Makefile.sphinx html

== Design ideas / principles ==

=== Automated conversion ===

The initial Docbook to Sphinx migration can be done with automated tools such as [https://pandoc.org/ pandoc]. While the conversion is far from being perfect, it produces some clean output markdown text file. After the initial automated conversion developers will need to fix up at least headings, images and links. In addition Sphinx has built in mechanisms (directives) that should be used to replace similar functions implemented in Docbook such as glossary, variables substitutions, notes, ...

To illustrate how Pandoc can be used, the initial conversion for bsp-guide, ref-manual and overview-manual was done with the following commands:

for i in *.xml; do pandoc -f docbook -t rst $i -o $i.rst; done

=== Headings ===

The layout of the Yocto Project manuals is organized as follows

Book
Chapter
Section
Section
Section

During the conversion with Pandoc, some of the headings are not converted properly and need to be fixed manually. Let's propose to use the following headings styles:

Book => overline ===
Chapter => overline ***
Section => ====
Section => ----
Section => ^^^^
Section => """"

At least with this proposal, we keep the same TOCs with Sphinx and Docbook.

=== Built in glossary ===

Sphinx has a [https://www.sphinx-doc.org/en/master/usage/restructuredtext/directives.html#glossary glossary directive]. From the documentation:

This directive must contain a reST definition list with terms and definitions. The definitions will then be referencable with the [https://www.sphinx-doc.org/en/master/usage/restructuredtext/roles.html#role-term 'term' role].

So anywhere in *any* manual, we can do :term:`VAR` to refer to an item from the glossary, and a link is created.

=== Global substitutions ===

The Yocto Project documentation makes heavy use of 'global' variables. In Docbook these 'variables' are stored in the file [http://git.yoctoproject.org/cgit.cgi/yocto-docs/tree/documentation/poky.ent poky.ent]. This Docbook feature is not handled automatically with Pandoc. Sphinx has builtin support for [https://www.sphinx-doc.org/en/master/usage/restructuredtext/basics.html#substitutions substitutions] however they are local to each reST file by default. They can be made global by using ''rst_prolog'':

rst_prolog
A string of reStructuredText that will be included at the beginning of every source file that is read.

As such we will keep a documentation configuration file with all global substitutions, similar to ''poky.ent''. Here is a sample code to define variables:

rst_prolog = """
.. |DISTRO| replace:: 3.1
.. |DISTRO_COMPRESSED| replace:: 31
.. |DISTRO_NAME_NO_CAP| replace:: dunfell
.. |DISTRO_NAME| replace:: Dunfell
.. |DISTRO_NAME_NO_CAP_MINUS_ONE| replace:: zeus
.. |DISTRO_NAME_MINUS_ONE| replace:: Zeus
.. |YOCTO_DOC_VERSION| replace:: 3.1
.. |YOCTO_DOC_VERSION_MINUS_ONE| replace:: 3.0.2
.. |DISTRO_REL_TAG| replace:: yocto-3.1
.. |METAINTELVERSION| replace:: 12.0
.. |REL_MONTH_YEAR| replace:: April 2020
"""

and use them:

This is the manual for version: |DISTRO|, aka |DISTRO_NAME|.

However, there are serious shortcomings with Sphinx built-in, for example they can be used/nested inside code-block sections. Another mechanism was implemented so that we can continue to use 'variables' within our documentation. A Sphinx extension was implemented to support variable substitutions while "parsing" the .rst files. The pattern for variables substitutions is: ''{VAR}''. The implementation of the extension can be found here:
https://git.yoctoproject.org/cgit/cgit.cgi/yocto-docs/tree/documentation/sphinx/yocto-vars.py?h=sphinx. All variables are set in a file call poky.yaml, which was generated from poky.ent: https://git.yoctoproject.org/cgit/cgit.cgi/yocto-docs/tree/documentation/poky.yaml?h=sphinx. For example, the following .rst content will produce the 'expected' content:

.. code-block::
$ mkdir ~/poky-{DISTRO}
or
$ git clone {YOCTO_GIT_URL}/git/poky -b {DISTRO_NAME_NO_CAP}

Variables can be nested, like it was the case for DocBook.

=== Note directive ===

Sphinx has a builtin ''note'' directive that produces clean Note section in the output file. There are various types of directives such as "attention", "caution", "danger", "error", "hint", "important", "tip", "warning", "admonition" that are supported, and additional directive can be added as Sphinx extension if needed.

=== Figures ===

Our documentation has many figures/images. Somehow Pandoc has completely ignored all images during the conversion. It might be worth investigating why, even though it might take more time than manually fixing all documents... Sphinx as a ''figure'' directive which is straight forward to use. To include a figure in the body of the documentation:

.. image:: figures/YP-flow-diagram.png

=== Links and References ===

Please read: https://sublime-and-sphinx-guide.readthedocs.io/en/latest/references.html

You can include links to other locations in the same document, to locations in other documents and to external websites.

==== references ====

We enable this extension by default: [https://www.sphinx-doc.org/en/master/usage/extensions/autosectionlabel.html. sphinx.ext.autosectionlabel].

You can link from text to a heading in any other part of the document by using the :ref: command with the heading text as the parameter. For example, this text in another part of this document would link to this section:
:ref:`Cross-References to Locations in the Same Document`

We can use Custom link text or create custom anchor instead of using the section text:
Please check this :ref:`section <Cross-References to Locations in the Same Document>`

You can also create links/references in another .rst file, using the file name:
:ref:`overview-manual/overview-manual-development-environment:yocto project source repositories`

or with a custom link text:
:ref:`here <overview-manual/overview-manual-development-environment:yocto project source repositories>`

Here is a '''good tip''': You can use the following command to get Sphinx to dump all the references that exist:
python -msphinx.ext.intersphinx <path to build folder>/html/objects.inv

In this dump, you will find all links and for each link you will get the default "Link Text" that Sphinx will use. If the default link text is not appropriate, you can provide a custom link text in the '':ref:'' link.

==== external links ====
We enable the '[https://sublime-and-sphinx-guide.readthedocs.io/en/latest/references.html#use-the-external-links-extension extlinks]' extension by default, and it's configured with the following ``extlinks``:

'yocto_home': ('https://yoctoproject.org%s', None),
'yocto_wiki': ('https://wiki.yoctoproject.org%s', None),
'yocto_dl': ('https://downloads.yoctoproject.org%s', None),
'yocto_lists': ('https://lists.yoctoproject.org%s', None),
'yocto_bugs': ('https://bugzilla.yoctoproject.org%s', None),
'yocto_ab': ('https://autobuilder.yoctoproject.org%s', None),
'yocto_docs': ('https://docs.yoctoproject.org%s', None),
'yocto_git': ('https://git.yoctoproject.org%s', None),
'oe_home': ('https://www.openembedded.org%s', None),
'oe_lists': ('https://lists.openembedded.org%s', None),

In the documentation rst files, we can then do:

Please check this :yocto_wiki:`wiki page </Weekly_Status>`

==== intersphinx links ====

We also enable the [https://www.sphinx-doc.org/en/master/usage/extensions/intersphinx.html intersphinx extension] so that we can cross reference content from the bitbake manual for example. When building the YP docs, Sphinx will first download the objects.inv file from the Bitbake manual, so that it knows all the existing references/links set in the bitbake documentation. Then references to the bitbake manual can be done like this:

See the ":ref:`-D <bitbake:bitbake-user-manual/bitbake-user-manual-intro:usage and syntax>`" option

or

:term:`bitbake:BB_NUMBER_PARSE_THREADS`

== Tasks list ==

Here is an initial tasks list for the migration:

# Proof of concept
## <s>Initial proof of concept patches</s>
## Merge initial patch set in master, along with Docbook
## <s>Update initial patches with yocto-docs recent changes </s>
# Convert all YP manuals
## <s>Automated conversion with pandoc</s>
### <s>adt-manual</s>
### <s>brief-yoctoprojectqs</s>
### <s>dev-manual</s>
### <s>kernel-dev</s>
### <s>profile-manual</s>
### <s>sdk-manual</s>
### <s>toaster-manual</s>
### <s>test-manual</s>
### <s>bitbake-manual</s>
## fix up all manuals for:
### links
### <s>figures</s>
### <s>headings</s>
### notes
### code blocks
### global substitutions variables
## Use current docs CSS file to create proper theme
### <s>initial import</s>
### tuning/optimization
## How to deal with each docbook 'first' page (license and TOC) (In progress RP and ndec looking at that)
## <s>Implement glossary</s>
# <s>Decide how to manage the mega manual. Should we create a single doc (index.rst) file per documentat, or a single one which ultimately would become the mega manual</s>
# <s>Custom Sphinx theme. The whole output can be themed using CSS and custom theme. We need to implement something that 'looks like' the Yocto Project and create beautiful rendering of our documentation</s>
# How to publish HTML online (per branch, latest, current, ...)? The whole process must be automated. (In progress, Michael on it)
# How to create PDF files?
# What's the impact of Google search and referencing?
# <s>Do we want to backport to dunfell LTS? (RP: Yes!)</s>
# <s>RP: Can we create a single page manual for ease of searching (some users need this and I personally use it a lot that way, I'm unlikely alone)</s>
# RP: Glossary headings don't appear to be externally linkable?

Documentation Sphinx

2020-09-11T07:40:07Z

Nicolas Dechesne: /* Links and References */

== Yocto Project Documentation: Sphinx migration ==

The Yocto Project developers are currently investigating whether to move from Docbook to Sphinx (or any similar tools) for the project's documentation. The initial discussion can be found [https://lists.yoctoproject.org/g/docs/message/165 here].

Sphinx is a tool that makes it easy to create intelligent and beautiful documentation, written by Georg Brandl and licensed under the BSD license. It was originally created for the Python documentation.

See more on [https://www.sphinx-doc.org/en/master/ Sphinx website]. Furthermore Sphinx is designed to be [https://www.sphinx-doc.org/en/master/extdev/index.html extensible], and we can implement our own custom extensions, as Python modules, which will be executed during the generation of the documentation. This is a rather advance use of Sphinx, but could be very useful in the future.

This wiki page can be used as a working document to assist the discussions and/or the tasks. All discussions should be made on the [https://lists.yoctoproject.org/g/docs/ Yocto Project Docs mailing list].

== Sphinx migration for Yocto Project Documentation ==

The Sphinx migration is planned for 3.2 release, and the development is done in [https://git.yoctoproject.org/cgit/cgit.cgi/yocto-docs/log/?h=sphinx this branch], which produces the following [https://docs.yoctoproject.org/ HTML documentation]. Until the branch gets merged , patches should be sent against this branch.

A lot of work was already done to convert all the docbook files, but there is still some significant of work left to be done.

To use the proof of concept, please follow these instructions:

pip3 install sphinx
# to use the non default theme
pip3 install sphinx_rtd_theme

And then:

git clone https://git.yoctoproject.org/git/yocto-docs -b sphinx
cd yocto-docs/documentation
make -f Makefile.sphinx html

The resulting HTML index page will be _build/html/index.html.

== Sphinx proof of concept for Bitbake Documentation ==

A similar branch exists for the Bitbake documentation as well. The branch is available on [https://git.openembedded.org/bitbake/log/?h=sphinx here], and produces the following [https://people.linaro.org/~nicolas.dechesne/bitbake-docs/html/ HTML documentation].

To build the Bitbake documentation with Sphinx:

git clone https://git.openembedded.org/bitbake -b sphinx
cd bitbake/doc
make -f Makefile.sphinx html

== Design ideas / principles ==

=== Automated conversion ===

The initial Docbook to Sphinx migration can be done with automated tools such as [https://pandoc.org/ pandoc]. While the conversion is far from being perfect, it produces some clean output markdown text file. After the initial automated conversion developers will need to fix up at least headings, images and links. In addition Sphinx has built in mechanisms (directives) that should be used to replace similar functions implemented in Docbook such as glossary, variables substitutions, notes, ...

To illustrate how Pandoc can be used, the initial conversion for bsp-guide, ref-manual and overview-manual was done with the following commands:

for i in *.xml; do pandoc -f docbook -t rst $i -o $i.rst; done

=== Headings ===

The layout of the Yocto Project manuals is organized as follows

Book
Chapter
Section
Section
Section

During the conversion with Pandoc, some of the headings are not converted properly and need to be fixed manually. Let's propose to use the following headings styles:

Book => overline ===
Chapter => overline ***
Section => ====
Section => ----
Section => ^^^^
Section => """"

At least with this proposal, we keep the same TOCs with Sphinx and Docbook.

=== Built in glossary ===

Sphinx has a [https://www.sphinx-doc.org/en/master/usage/restructuredtext/directives.html#glossary glossary directive]. From the documentation:

This directive must contain a reST definition list with terms and definitions. The definitions will then be referencable with the [https://www.sphinx-doc.org/en/master/usage/restructuredtext/roles.html#role-term 'term' role].

So anywhere in *any* manual, we can do :term:`VAR` to refer to an item from the glossary, and a link is created.

=== Global substitutions ===

The Yocto Project documentation makes heavy use of 'global' variables. In Docbook these 'variables' are stored in the file [http://git.yoctoproject.org/cgit.cgi/yocto-docs/tree/documentation/poky.ent poky.ent]. This Docbook feature is not handled automatically with Pandoc. Sphinx has builtin support for [https://www.sphinx-doc.org/en/master/usage/restructuredtext/basics.html#substitutions substitutions] however they are local to each reST file by default. They can be made global by using ''rst_prolog'':

rst_prolog
A string of reStructuredText that will be included at the beginning of every source file that is read.

As such we will keep a documentation configuration file with all global substitutions, similar to ''poky.ent''. Here is a sample code to define variables:

rst_prolog = """
.. |DISTRO| replace:: 3.1
.. |DISTRO_COMPRESSED| replace:: 31
.. |DISTRO_NAME_NO_CAP| replace:: dunfell
.. |DISTRO_NAME| replace:: Dunfell
.. |DISTRO_NAME_NO_CAP_MINUS_ONE| replace:: zeus
.. |DISTRO_NAME_MINUS_ONE| replace:: Zeus
.. |YOCTO_DOC_VERSION| replace:: 3.1
.. |YOCTO_DOC_VERSION_MINUS_ONE| replace:: 3.0.2
.. |DISTRO_REL_TAG| replace:: yocto-3.1
.. |METAINTELVERSION| replace:: 12.0
.. |REL_MONTH_YEAR| replace:: April 2020
"""

and use them:

This is the manual for version: |DISTRO|, aka |DISTRO_NAME|.

However, there are serious shortcomings with Sphinx built-in, for example they can be used/nested inside code-block sections. Another mechanism was implemented so that we can continue to use 'variables' within our documentation. A Sphinx extension was implemented to support variable substitutions while "parsing" the .rst files. The pattern for variables substitutions is: ''{VAR}''. The implementation of the extension can be found here:
https://git.yoctoproject.org/cgit/cgit.cgi/yocto-docs/tree/documentation/sphinx/yocto-vars.py?h=sphinx. All variables are set in a file call poky.yaml, which was generated from poky.ent: https://git.yoctoproject.org/cgit/cgit.cgi/yocto-docs/tree/documentation/poky.yaml?h=sphinx. For example, the following .rst content will produce the 'expected' content:

.. code-block::
$ mkdir ~/poky-{DISTRO}
or
$ git clone {YOCTO_GIT_URL}/git/poky -b {DISTRO_NAME_NO_CAP}

Variables can be nested, like it was the case for DocBook.

=== Note directive ===

Sphinx has a builtin ''note'' directive that produces clean Note section in the output file. There are various types of directives such as "attention", "caution", "danger", "error", "hint", "important", "tip", "warning", "admonition" that are supported, and additional directive can be added as Sphinx extension if needed.

=== Figures ===

Our documentation has many figures/images. Somehow Pandoc has completely ignored all images during the conversion. It might be worth investigating why, even though it might take more time than manually fixing all documents... Sphinx as a ''figure'' directive which is straight forward to use. To include a figure in the body of the documentation:

.. image:: figures/YP-flow-diagram.png

=== Links and References ===

Please read: https://sublime-and-sphinx-guide.readthedocs.io/en/latest/references.html

You can include links to other locations in the same document, to locations in other documents and to external websites.

==== references ====

We enable this extension by default: [https://www.sphinx-doc.org/en/master/usage/extensions/autosectionlabel.html. sphinx.ext.autosectionlabel].

You can link from text to a heading in any other part of the document by using the :ref: command with the heading text as the parameter. For example, this text in another part of this document would link to this section:
:ref:`Cross-References to Locations in the Same Document`

We can use Custom link text or create custom anchor instead of using the section text:
Please check this :ref:`section <Cross-References to Locations in the Same Document>`

You can also create links/references in another .rst file, using the file name:
:ref:`overview-manual/overview-manual-development-environment:yocto project source repositories`

or with a custom link text:
:ref:`here <overview-manual/overview-manual-development-environment:yocto project source repositories>`

Here is a '''good tip''': You can use the following command to get Sphinx to dump all the references that exist:
python -msphinx.ext.intersphinx <path to build folder>/html/objects.inv

In this dump, you will find all links and for each link you will get the default "Link Text" that Sphinx will use. If the default link text is not appropriate, you can provide a custom link text in the '':ref:'' link.

==== external links ====
We enable the '[https://sublime-and-sphinx-guide.readthedocs.io/en/latest/references.html#use-the-external-links-extension extlinks]' extension by default, and it's configured with the following ``extlinks``:

'yocto_home': ('https://yoctoproject.org%s', None),
'yocto_wiki': ('https://wiki.yoctoproject.org%s', None),
'yocto_dl': ('https://downloads.yoctoproject.org%s', None),
'yocto_lists': ('https://lists.yoctoproject.org%s', None),
'yocto_bugs': ('https://bugzilla.yoctoproject.org%s', None),
'yocto_ab': ('https://autobuilder.yoctoproject.org%s', None),
'yocto_docs': ('https://docs.yoctoproject.org%s', None),
'yocto_git': ('https://git.yoctoproject.org%s', None),
'oe_home': ('https://www.openembedded.org%s', None),
'oe_lists': ('https://lists.openembedded.org%s', None),

In the documentation rst files, we can then do:

Please check this :yocto_wiki:`wiki page </Weekly_Status>`

==== intersphinx links ====

We also enable the [https://www.sphinx-doc.org/en/master/usage/extensions/intersphinx.html intersphinx extension] so that we can cross reference content from the bitbake manual for example. When building the YP docs, Sphinx will first download the objects.inv file from the Bitbake manual, so that it knows all the existing references/links set in the bitbake documentation. Then references to the bitbake manual can be done like this:

See the ":ref:`-D <bitbake:bitbake-user-manual/bitbake-user-manual-intro:usage and syntax>`" option

or

:term:`bitbake:BB_NUMBER_PARSE_THREADS`

== Tasks list ==

Here is an initial tasks list for the migration:

# Proof of concept
## <s>Initial proof of concept patches</s>
## Merge initial patch set in master, along with Docbook
## <s>Update initial patches with yocto-docs recent changes </s>
# Convert all YP manuals
## <s>Automated conversion with pandoc</s>
### <s>adt-manual</s>
### <s>brief-yoctoprojectqs</s>
### <s>dev-manual</s>
### <s>kernel-dev</s>
### <s>profile-manual</s>
### <s>sdk-manual</s>
### <s>toaster-manual</s>
### <s>test-manual</s>
### <s>bitbake-manual</s>
## fix up all manuals for:
### links
### <s>figures</s>
### <s>headings</s>
### notes
### code blocks
### global substitutions variables
## Use current docs CSS file to create proper theme
### <s>initial import</s>
### tuning/optimization
## How to deal with each docbook 'first' page (license and TOC) (In progress RP and ndec looking at that)
## <s>Implement glossary</s>
# <s>Decide how to manage the mega manual. Should we create a single doc (index.rst) file per documentat, or a single one which ultimately would become the mega manual</s>
# <s>Custom Sphinx theme. The whole output can be themed using CSS and custom theme. We need to implement something that 'looks like' the Yocto Project and create beautiful rendering of our documentation</s>
# How to publish HTML online (per branch, latest, current, ...)? The whole process must be automated. (In progress, Michael on it)
# How to create PDF files?
# What's the impact of Google search and referencing?
# <s>Do we want to backport to dunfell LTS? (RP: Yes!)</s>
# <s>RP: Can we create a single page manual for ease of searching (some users need this and I personally use it a lot that way, I'm unlikely alone)</s>
# RP: Glossary headings don't appear to be externally linkable?

Documentation Sphinx

2020-09-11T07:24:02Z

Nicolas Dechesne: /* Global substitutions */

== Yocto Project Documentation: Sphinx migration ==

The Yocto Project developers are currently investigating whether to move from Docbook to Sphinx (or any similar tools) for the project's documentation. The initial discussion can be found [https://lists.yoctoproject.org/g/docs/message/165 here].

Sphinx is a tool that makes it easy to create intelligent and beautiful documentation, written by Georg Brandl and licensed under the BSD license. It was originally created for the Python documentation.

See more on [https://www.sphinx-doc.org/en/master/ Sphinx website]. Furthermore Sphinx is designed to be [https://www.sphinx-doc.org/en/master/extdev/index.html extensible], and we can implement our own custom extensions, as Python modules, which will be executed during the generation of the documentation. This is a rather advance use of Sphinx, but could be very useful in the future.

This wiki page can be used as a working document to assist the discussions and/or the tasks. All discussions should be made on the [https://lists.yoctoproject.org/g/docs/ Yocto Project Docs mailing list].

== Sphinx migration for Yocto Project Documentation ==

The Sphinx migration is planned for 3.2 release, and the development is done in [https://git.yoctoproject.org/cgit/cgit.cgi/yocto-docs/log/?h=sphinx this branch], which produces the following [https://docs.yoctoproject.org/ HTML documentation]. Until the branch gets merged , patches should be sent against this branch.

A lot of work was already done to convert all the docbook files, but there is still some significant of work left to be done.

To use the proof of concept, please follow these instructions:

pip3 install sphinx
# to use the non default theme
pip3 install sphinx_rtd_theme

And then:

git clone https://git.yoctoproject.org/git/yocto-docs -b sphinx
cd yocto-docs/documentation
make -f Makefile.sphinx html

The resulting HTML index page will be _build/html/index.html.

== Sphinx proof of concept for Bitbake Documentation ==

A similar branch exists for the Bitbake documentation as well. The branch is available on [https://git.openembedded.org/bitbake/log/?h=sphinx here], and produces the following [https://people.linaro.org/~nicolas.dechesne/bitbake-docs/html/ HTML documentation].

To build the Bitbake documentation with Sphinx:

git clone https://git.openembedded.org/bitbake -b sphinx
cd bitbake/doc
make -f Makefile.sphinx html

== Design ideas / principles ==

=== Automated conversion ===

The initial Docbook to Sphinx migration can be done with automated tools such as [https://pandoc.org/ pandoc]. While the conversion is far from being perfect, it produces some clean output markdown text file. After the initial automated conversion developers will need to fix up at least headings, images and links. In addition Sphinx has built in mechanisms (directives) that should be used to replace similar functions implemented in Docbook such as glossary, variables substitutions, notes, ...

To illustrate how Pandoc can be used, the initial conversion for bsp-guide, ref-manual and overview-manual was done with the following commands:

for i in *.xml; do pandoc -f docbook -t rst $i -o $i.rst; done

=== Headings ===

The layout of the Yocto Project manuals is organized as follows

Book
Chapter
Section
Section
Section

During the conversion with Pandoc, some of the headings are not converted properly and need to be fixed manually. Let's propose to use the following headings styles:

Book => overline ===
Chapter => overline ***
Section => ====
Section => ----
Section => ^^^^
Section => """"

At least with this proposal, we keep the same TOCs with Sphinx and Docbook.

=== Built in glossary ===

Sphinx has a [https://www.sphinx-doc.org/en/master/usage/restructuredtext/directives.html#glossary glossary directive]. From the documentation:

This directive must contain a reST definition list with terms and definitions. The definitions will then be referencable with the [https://www.sphinx-doc.org/en/master/usage/restructuredtext/roles.html#role-term 'term' role].

So anywhere in *any* manual, we can do :term:`VAR` to refer to an item from the glossary, and a link is created.

=== Global substitutions ===

The Yocto Project documentation makes heavy use of 'global' variables. In Docbook these 'variables' are stored in the file [http://git.yoctoproject.org/cgit.cgi/yocto-docs/tree/documentation/poky.ent poky.ent]. This Docbook feature is not handled automatically with Pandoc. Sphinx has builtin support for [https://www.sphinx-doc.org/en/master/usage/restructuredtext/basics.html#substitutions substitutions] however they are local to each reST file by default. They can be made global by using ''rst_prolog'':

rst_prolog
A string of reStructuredText that will be included at the beginning of every source file that is read.

As such we will keep a documentation configuration file with all global substitutions, similar to ''poky.ent''. Here is a sample code to define variables:

rst_prolog = """
.. |DISTRO| replace:: 3.1
.. |DISTRO_COMPRESSED| replace:: 31
.. |DISTRO_NAME_NO_CAP| replace:: dunfell
.. |DISTRO_NAME| replace:: Dunfell
.. |DISTRO_NAME_NO_CAP_MINUS_ONE| replace:: zeus
.. |DISTRO_NAME_MINUS_ONE| replace:: Zeus
.. |YOCTO_DOC_VERSION| replace:: 3.1
.. |YOCTO_DOC_VERSION_MINUS_ONE| replace:: 3.0.2
.. |DISTRO_REL_TAG| replace:: yocto-3.1
.. |METAINTELVERSION| replace:: 12.0
.. |REL_MONTH_YEAR| replace:: April 2020
"""

and use them:

This is the manual for version: |DISTRO|, aka |DISTRO_NAME|.

However, there are serious shortcomings with Sphinx built-in, for example they can be used/nested inside code-block sections. Another mechanism was implemented so that we can continue to use 'variables' within our documentation. A Sphinx extension was implemented to support variable substitutions while "parsing" the .rst files. The pattern for variables substitutions is: ''{VAR}''. The implementation of the extension can be found here:
https://git.yoctoproject.org/cgit/cgit.cgi/yocto-docs/tree/documentation/sphinx/yocto-vars.py?h=sphinx. All variables are set in a file call poky.yaml, which was generated from poky.ent: https://git.yoctoproject.org/cgit/cgit.cgi/yocto-docs/tree/documentation/poky.yaml?h=sphinx. For example, the following .rst content will produce the 'expected' content:

.. code-block::
$ mkdir ~/poky-{DISTRO}
or
$ git clone {YOCTO_GIT_URL}/git/poky -b {DISTRO_NAME_NO_CAP}

Variables can be nested, like it was the case for DocBook.

=== Note directive ===

Sphinx has a builtin ''note'' directive that produces clean Note section in the output file. There are various types of directives such as "attention", "caution", "danger", "error", "hint", "important", "tip", "warning", "admonition" that are supported, and additional directive can be added as Sphinx extension if needed.

=== Figures ===

Our documentation has many figures/images. Somehow Pandoc has completely ignored all images during the conversion. It might be worth investigating why, even though it might take more time than manually fixing all documents... Sphinx as a ''figure'' directive which is straight forward to use. To include a figure in the body of the documentation:

.. image:: figures/YP-flow-diagram.png

=== Links and References ===
From: https://sublime-and-sphinx-guide.readthedocs.io/en/latest/references.html

You can include links to other locations in the same document, to locations in other documents and to external websites.

You can link from text to a heading in any other part of the document by using the :ref: command with the heading text as the parameter. For example, this text in another part of this document would link to this section:
:ref:`Cross-References to Locations in the Same Document`

For that to work, we need sphinx.ext.autosectionlabel to be enabled in conf.py.

We can use Custom link text or create custom anchor instead of using the section text.

TODO: study https://sublime-and-sphinx-guide.readthedocs.io/en/latest/references.html#use-the-external-links-extension, looks like something we could use..

== Tasks list ==

Here is an initial tasks list for the migration:

# Proof of concept
## <s>Initial proof of concept patches</s>
## Merge initial patch set in master, along with Docbook
## <s>Update initial patches with yocto-docs recent changes </s>
# Convert all YP manuals
## <s>Automated conversion with pandoc</s>
### <s>adt-manual</s>
### <s>brief-yoctoprojectqs</s>
### <s>dev-manual</s>
### <s>kernel-dev</s>
### <s>profile-manual</s>
### <s>sdk-manual</s>
### <s>toaster-manual</s>
### <s>test-manual</s>
### <s>bitbake-manual</s>
## fix up all manuals for:
### links
### <s>figures</s>
### <s>headings</s>
### notes
### code blocks
### global substitutions variables
## Use current docs CSS file to create proper theme
### <s>initial import</s>
### tuning/optimization
## How to deal with each docbook 'first' page (license and TOC) (In progress RP and ndec looking at that)
## <s>Implement glossary</s>
# <s>Decide how to manage the mega manual. Should we create a single doc (index.rst) file per documentat, or a single one which ultimately would become the mega manual</s>
# <s>Custom Sphinx theme. The whole output can be themed using CSS and custom theme. We need to implement something that 'looks like' the Yocto Project and create beautiful rendering of our documentation</s>
# How to publish HTML online (per branch, latest, current, ...)? The whole process must be automated. (In progress, Michael on it)
# How to create PDF files?
# What's the impact of Google search and referencing?
# <s>Do we want to backport to dunfell LTS? (RP: Yes!)</s>
# <s>RP: Can we create a single page manual for ease of searching (some users need this and I personally use it a lot that way, I'm unlikely alone)</s>
# RP: Glossary headings don't appear to be externally linkable?

Documentation Sphinx

2020-09-10T22:00:58Z

Nicolas Dechesne: /* Sphinx proof of concept for Bitbake Documentation */

== Yocto Project Documentation: Sphinx migration ==

The Yocto Project developers are currently investigating whether to move from Docbook to Sphinx (or any similar tools) for the project's documentation. The initial discussion can be found [https://lists.yoctoproject.org/g/docs/message/165 here].

Sphinx is a tool that makes it easy to create intelligent and beautiful documentation, written by Georg Brandl and licensed under the BSD license. It was originally created for the Python documentation.

See more on [https://www.sphinx-doc.org/en/master/ Sphinx website]. Furthermore Sphinx is designed to be [https://www.sphinx-doc.org/en/master/extdev/index.html extensible], and we can implement our own custom extensions, as Python modules, which will be executed during the generation of the documentation. This is a rather advance use of Sphinx, but could be very useful in the future.

This wiki page can be used as a working document to assist the discussions and/or the tasks. All discussions should be made on the [https://lists.yoctoproject.org/g/docs/ Yocto Project Docs mailing list].

== Sphinx migration for Yocto Project Documentation ==

The Sphinx migration is planned for 3.2 release, and the development is done in [https://git.yoctoproject.org/cgit/cgit.cgi/yocto-docs/log/?h=sphinx this branch], which produces the following [https://docs.yoctoproject.org/ HTML documentation]. Until the branch gets merged , patches should be sent against this branch.

A lot of work was already done to convert all the docbook files, but there is still some significant of work left to be done.

To use the proof of concept, please follow these instructions:

pip3 install sphinx
# to use the non default theme
pip3 install sphinx_rtd_theme

And then:

git clone https://git.yoctoproject.org/git/yocto-docs -b sphinx
cd yocto-docs/documentation
make -f Makefile.sphinx html

The resulting HTML index page will be _build/html/index.html.

== Sphinx proof of concept for Bitbake Documentation ==

A similar branch exists for the Bitbake documentation as well. The branch is available on [https://git.openembedded.org/bitbake/log/?h=sphinx here], and produces the following [https://people.linaro.org/~nicolas.dechesne/bitbake-docs/html/ HTML documentation].

To build the Bitbake documentation with Sphinx:

git clone https://git.openembedded.org/bitbake -b sphinx
cd bitbake/doc
make -f Makefile.sphinx html

== Design ideas / principles ==

=== Automated conversion ===

The initial Docbook to Sphinx migration can be done with automated tools such as [https://pandoc.org/ pandoc]. While the conversion is far from being perfect, it produces some clean output markdown text file. After the initial automated conversion developers will need to fix up at least headings, images and links. In addition Sphinx has built in mechanisms (directives) that should be used to replace similar functions implemented in Docbook such as glossary, variables substitutions, notes, ...

To illustrate how Pandoc can be used, the initial conversion for bsp-guide, ref-manual and overview-manual was done with the following commands:

for i in *.xml; do pandoc -f docbook -t rst $i -o $i.rst; done

=== Headings ===

The layout of the Yocto Project manuals is organized as follows

Book
Chapter
Section
Section
Section

During the conversion with Pandoc, some of the headings are not converted properly and need to be fixed manually. Let's propose to use the following headings styles:

Book => overline ===
Chapter => overline ***
Section => ====
Section => ----
Section => ^^^^
Section => """"

At least with this proposal, we keep the same TOCs with Sphinx and Docbook.

=== Built in glossary ===

Sphinx has a [https://www.sphinx-doc.org/en/master/usage/restructuredtext/directives.html#glossary glossary directive]. From the documentation:

This directive must contain a reST definition list with terms and definitions. The definitions will then be referencable with the [https://www.sphinx-doc.org/en/master/usage/restructuredtext/roles.html#role-term 'term' role].

So anywhere in *any* manual, we can do :term:`VAR` to refer to an item from the glossary, and a link is created.

=== Global substitutions ===

The Yocto Project documentation makes heavy use of 'global' variables. In Docbook these 'variables' are stored in the file [http://git.yoctoproject.org/cgit.cgi/yocto-docs/tree/documentation/poky.ent poky.ent]. This Docbook feature is not handled automatically with Pandoc. Sphinx has builtin support for [https://www.sphinx-doc.org/en/master/usage/restructuredtext/basics.html#substitutions substitutions] however they are local to each reST file by default. They can be made global by using ''rst_prolog'':

rst_prolog
A string of reStructuredText that will be included at the beginning of every source file that is read.

As such we will keep a documentation configuration file with all global substitutions, similar to ''poky.ent''. Here is a sample code to define variables:

rst_prolog = """
.. |DISTRO| replace:: 3.1
.. |DISTRO_COMPRESSED| replace:: 31
.. |DISTRO_NAME_NO_CAP| replace:: dunfell
.. |DISTRO_NAME| replace:: Dunfell
.. |DISTRO_NAME_NO_CAP_MINUS_ONE| replace:: zeus
.. |DISTRO_NAME_MINUS_ONE| replace:: Zeus
.. |YOCTO_DOC_VERSION| replace:: 3.1
.. |YOCTO_DOC_VERSION_MINUS_ONE| replace:: 3.0.2
.. |DISTRO_REL_TAG| replace:: yocto-3.1
.. |METAINTELVERSION| replace:: 12.0
.. |REL_MONTH_YEAR| replace:: April 2020
"""

and use them:

This is the manual for version: |DISTRO|, aka |DISTRO_NAME|.

=== Note directive ===

Sphinx has a builtin ''note'' directive that produces clean Note section in the output file. There are various types of directives such as "attention", "caution", "danger", "error", "hint", "important", "tip", "warning", "admonition" that are supported, and additional directive can be added as Sphinx extension if needed.

=== Figures ===

Our documentation has many figures/images. Somehow Pandoc has completely ignored all images during the conversion. It might be worth investigating why, even though it might take more time than manually fixing all documents... Sphinx as a ''figure'' directive which is straight forward to use. To include a figure in the body of the documentation:

.. image:: figures/YP-flow-diagram.png

=== Links and References ===
From: https://sublime-and-sphinx-guide.readthedocs.io/en/latest/references.html

You can include links to other locations in the same document, to locations in other documents and to external websites.

You can link from text to a heading in any other part of the document by using the :ref: command with the heading text as the parameter. For example, this text in another part of this document would link to this section:
:ref:`Cross-References to Locations in the Same Document`

For that to work, we need sphinx.ext.autosectionlabel to be enabled in conf.py.

We can use Custom link text or create custom anchor instead of using the section text.

TODO: study https://sublime-and-sphinx-guide.readthedocs.io/en/latest/references.html#use-the-external-links-extension, looks like something we could use..

== Tasks list ==

Here is an initial tasks list for the migration:

# Proof of concept
## <s>Initial proof of concept patches</s>
## Merge initial patch set in master, along with Docbook
## <s>Update initial patches with yocto-docs recent changes </s>
# Convert all YP manuals
## <s>Automated conversion with pandoc</s>
### <s>adt-manual</s>
### <s>brief-yoctoprojectqs</s>
### <s>dev-manual</s>
### <s>kernel-dev</s>
### <s>profile-manual</s>
### <s>sdk-manual</s>
### <s>toaster-manual</s>
### <s>test-manual</s>
### <s>bitbake-manual</s>
## fix up all manuals for:
### links
### <s>figures</s>
### <s>headings</s>
### notes
### code blocks
### global substitutions variables
## Use current docs CSS file to create proper theme
### <s>initial import</s>
### tuning/optimization
## How to deal with each docbook 'first' page (license and TOC) (In progress RP and ndec looking at that)
## <s>Implement glossary</s>
# <s>Decide how to manage the mega manual. Should we create a single doc (index.rst) file per documentat, or a single one which ultimately would become the mega manual</s>
# <s>Custom Sphinx theme. The whole output can be themed using CSS and custom theme. We need to implement something that 'looks like' the Yocto Project and create beautiful rendering of our documentation</s>
# How to publish HTML online (per branch, latest, current, ...)? The whole process must be automated. (In progress, Michael on it)
# How to create PDF files?
# What's the impact of Google search and referencing?
# <s>Do we want to backport to dunfell LTS? (RP: Yes!)</s>
# <s>RP: Can we create a single page manual for ease of searching (some users need this and I personally use it a lot that way, I'm unlikely alone)</s>
# RP: Glossary headings don't appear to be externally linkable?

Documentation Sphinx

2020-09-05T00:00:19Z

Nicolas Dechesne:

== Yocto Project Documentation: Sphinx migration ==

The Yocto Project developers are currently investigating whether to move from Docbook to Sphinx (or any similar tools) for the project's documentation. The initial discussion can be found [https://lists.yoctoproject.org/g/docs/message/165 here].

Sphinx is a tool that makes it easy to create intelligent and beautiful documentation, written by Georg Brandl and licensed under the BSD license. It was originally created for the Python documentation.

See more on [https://www.sphinx-doc.org/en/master/ Sphinx website]. Furthermore Sphinx is designed to be [https://www.sphinx-doc.org/en/master/extdev/index.html extensible], and we can implement our own custom extensions, as Python modules, which will be executed during the generation of the documentation. This is a rather advance use of Sphinx, but could be very useful in the future.

This wiki page can be used as a working document to assist the discussions and/or the tasks. All discussions should be made on the [https://lists.yoctoproject.org/g/docs/ Yocto Project Docs mailing list].

== Sphinx migration for Yocto Project Documentation ==

The Sphinx migration is planned for 3.2 release, and the development is done in [https://git.yoctoproject.org/cgit/cgit.cgi/yocto-docs/log/?h=sphinx this branch], which produces the following [https://docs.yoctoproject.org/ HTML documentation]. Until the branch gets merged , patches should be sent against this branch.

A lot of work was already done to convert all the docbook files, but there is still some significant of work left to be done.

To use the proof of concept, please follow these instructions:

pip3 install sphinx
# to use the non default theme
pip3 install sphinx_rtd_theme

And then:

git clone https://git.yoctoproject.org/git/yocto-docs -b sphinx
cd yocto-docs/documentation
make -f Makefile.sphinx html

The resulting HTML index page will be _build/html/index.html.

== Sphinx proof of concept for Bitbake Documentation ==

A similar branch exists for the Bitbake documentation as well. The branch is available on [https://git.openembedded.org/bitbake/log/?h=sphinx here], and produces the following [https://people.linaro.org/~nicolas.dechesne/bitbake-docs/html/ HTML documentation].

To build the Bitbake documentation with Sphinx:

git clone https://git.openembedded.org/bitbake -b sphinx
cd bitbake/documentation
make -f Makefile.sphinx html

== Design ideas / principles ==

=== Automated conversion ===

The initial Docbook to Sphinx migration can be done with automated tools such as [https://pandoc.org/ pandoc]. While the conversion is far from being perfect, it produces some clean output markdown text file. After the initial automated conversion developers will need to fix up at least headings, images and links. In addition Sphinx has built in mechanisms (directives) that should be used to replace similar functions implemented in Docbook such as glossary, variables substitutions, notes, ...

To illustrate how Pandoc can be used, the initial conversion for bsp-guide, ref-manual and overview-manual was done with the following commands:

for i in *.xml; do pandoc -f docbook -t rst $i -o $i.rst; done

=== Headings ===

The layout of the Yocto Project manuals is organized as follows

Book
Chapter
Section
Section
Section

During the conversion with Pandoc, some of the headings are not converted properly and need to be fixed manually. Let's propose to use the following headings styles:

Book => overline ===
Chapter => overline ***
Section => ====
Section => ----
Section => ^^^^
Section => """"

At least with this proposal, we keep the same TOCs with Sphinx and Docbook.

=== Built in glossary ===

Sphinx has a [https://www.sphinx-doc.org/en/master/usage/restructuredtext/directives.html#glossary glossary directive]. From the documentation:

This directive must contain a reST definition list with terms and definitions. The definitions will then be referencable with the [https://www.sphinx-doc.org/en/master/usage/restructuredtext/roles.html#role-term 'term' role].

So anywhere in *any* manual, we can do :term:`VAR` to refer to an item from the glossary, and a link is created.

=== Global substitutions ===

The Yocto Project documentation makes heavy use of 'global' variables. In Docbook these 'variables' are stored in the file [http://git.yoctoproject.org/cgit.cgi/yocto-docs/tree/documentation/poky.ent poky.ent]. This Docbook feature is not handled automatically with Pandoc. Sphinx has builtin support for [https://www.sphinx-doc.org/en/master/usage/restructuredtext/basics.html#substitutions substitutions] however they are local to each reST file by default. They can be made global by using ''rst_prolog'':

rst_prolog
A string of reStructuredText that will be included at the beginning of every source file that is read.

As such we will keep a documentation configuration file with all global substitutions, similar to ''poky.ent''. Here is a sample code to define variables:

rst_prolog = """
.. |DISTRO| replace:: 3.1
.. |DISTRO_COMPRESSED| replace:: 31
.. |DISTRO_NAME_NO_CAP| replace:: dunfell
.. |DISTRO_NAME| replace:: Dunfell
.. |DISTRO_NAME_NO_CAP_MINUS_ONE| replace:: zeus
.. |DISTRO_NAME_MINUS_ONE| replace:: Zeus
.. |YOCTO_DOC_VERSION| replace:: 3.1
.. |YOCTO_DOC_VERSION_MINUS_ONE| replace:: 3.0.2
.. |DISTRO_REL_TAG| replace:: yocto-3.1
.. |METAINTELVERSION| replace:: 12.0
.. |REL_MONTH_YEAR| replace:: April 2020
"""

and use them:

This is the manual for version: |DISTRO|, aka |DISTRO_NAME|.

=== Note directive ===

Sphinx has a builtin ''note'' directive that produces clean Note section in the output file. There are various types of directives such as "attention", "caution", "danger", "error", "hint", "important", "tip", "warning", "admonition" that are supported, and additional directive can be added as Sphinx extension if needed.

=== Figures ===

Our documentation has many figures/images. Somehow Pandoc has completely ignored all images during the conversion. It might be worth investigating why, even though it might take more time than manually fixing all documents... Sphinx as a ''figure'' directive which is straight forward to use. To include a figure in the body of the documentation:

.. image:: figures/YP-flow-diagram.png

=== Links and References ===
From: https://sublime-and-sphinx-guide.readthedocs.io/en/latest/references.html

You can include links to other locations in the same document, to locations in other documents and to external websites.

You can link from text to a heading in any other part of the document by using the :ref: command with the heading text as the parameter. For example, this text in another part of this document would link to this section:
:ref:`Cross-References to Locations in the Same Document`

For that to work, we need sphinx.ext.autosectionlabel to be enabled in conf.py.

We can use Custom link text or create custom anchor instead of using the section text.

TODO: study https://sublime-and-sphinx-guide.readthedocs.io/en/latest/references.html#use-the-external-links-extension, looks like something we could use..

== Tasks list ==

Here is an initial tasks list for the migration:

# Proof of concept
## <s>Initial proof of concept patches</s>
## Merge initial patch set in master, along with Docbook
## <s>Update initial patches with yocto-docs recent changes </s>
# Convert all YP manuals
## <s>Automated conversion with pandoc</s>
### <s>adt-manual</s>
### <s>brief-yoctoprojectqs</s>
### <s>dev-manual</s>
### <s>kernel-dev</s>
### <s>profile-manual</s>
### <s>sdk-manual</s>
### <s>toaster-manual</s>
### <s>test-manual</s>
### <s>bitbake-manual</s>
## fix up all manuals for:
### links
### <s>figures</s>
### <s>headings</s>
### notes
### code blocks
### global substitutions variables
## Use current docs CSS file to create proper theme
### <s>initial import</s>
### tuning/optimization
## How to deal with each docbook 'first' page (license and TOC) (In progress RP and ndec looking at that)
## <s>Implement glossary</s>
# <s>Decide how to manage the mega manual. Should we create a single doc (index.rst) file per documentat, or a single one which ultimately would become the mega manual</s>
# <s>Custom Sphinx theme. The whole output can be themed using CSS and custom theme. We need to implement something that 'looks like' the Yocto Project and create beautiful rendering of our documentation</s>
# How to publish HTML online (per branch, latest, current, ...)? The whole process must be automated. (In progress, Michael on it)
# How to create PDF files?
# What's the impact of Google search and referencing?
# <s>Do we want to backport to dunfell LTS? (RP: Yes!)</s>
# <s>RP: Can we create a single page manual for ease of searching (some users need this and I personally use it a lot that way, I'm unlikely alone)</s>
# RP: Glossary headings don't appear to be externally linkable?

Documentation Sphinx

2020-09-04T23:59:40Z

Nicolas Dechesne:

== Yocto Project Documentation: Sphinx migration ==

The Yocto Project developers are currently investigating whether to move from Docbook to Sphinx (or any similar tools) for the project's documentation. The initial discussion can be found [https://lists.yoctoproject.org/g/docs/message/165 here].

Sphinx is a tool that makes it easy to create intelligent and beautiful documentation, written by Georg Brandl and licensed under the BSD license. It was originally created for the Python documentation.

See more on [https://www.sphinx-doc.org/en/master/ Sphinx website]. Furthermore Sphinx is designed to be [https://www.sphinx-doc.org/en/master/extdev/index.html extensible], and we can implement our own custom extensions, as Python modules, which will be executed during the generation of the documentation. This is a rather advance use of Sphinx, but could be very useful in the future.

This wiki page can be used as a working document to assist the discussions and/or the tasks. All discussions should be made on the [https://lists.yoctoproject.org/g/docs/ Yocto Project Docs mailing list].

== Sphinx migration for Yocto Project Documentation ==

The Sphinx migration is planned for 3.2 release, and the development is done in [https://git.yoctoproject.org/cgit/cgit.cgi/yocto-docs/log/?h=sphinx this branch], which produces the following [https://docs.yoctoproject.org/ HTML documentation]. Until the branch gets merged , patches should be sent against this branch.

A lot of work was already done to convert all the docbook files, but there is still some significant of work left to be done.

To use the proof of concept, please follow these instructions:

pip3 install sphinx
# to use the non default theme
pip3 install sphinx_rtd_theme

And then:

git clone https://git.yoctoproject.org/git/yocto-docs -b sphinx
cd yocto-docs/documentation
make -f Makefile.sphinx html

The resulting HTML index page will be _build/html/index.html.

== Sphinx proof of concept for Bitbake Documentation ==

A similar branch exists for the Bitbake documentation as well. The branch is available on [https://git.openembedded.org/bitbake/log/?h=sphinx here], and produces the following [https://people.linaro.org/~nicolas.dechesne/bitbake-docs/html/ HTML documentation].

To build the Bitbake documentation with Sphinx:

git clone https://git.openembedded.org/bitbake -b sphinx
cd bitbake/documentation
make -f Makefile.sphinx html

== Design ideas / principles ==

=== Automated conversion ===

The initial Docbook to Sphinx migration can be done with automated tools such as [https://pandoc.org/ pandoc]. While the conversion is far from being perfect, it produces some clean output markdown text file. After the initial automated conversion developers will need to fix up at least headings, images and links. In addition Sphinx has built in mechanisms (directives) that should be used to replace similar functions implemented in Docbook such as glossary, variables substitutions, notes, ...

To illustrate how Pandoc can be used, the initial conversion for bsp-guide, ref-manual and overview-manual was done with the following commands:

for i in *.xml; do pandoc -f docbook -t rst $i -o $i.rst; done

=== Headings ===

The layout of the Yocto Project manuals is organized as follows

Book
Chapter
Section
Section
Section

During the conversion with Pandoc, some of the headings are not converted properly and need to be fixed manually. Let's propose to use the following headings styles:

Book => overline ===
Chapter => overline ***
Section => ====
Section => ----
Section => ^^^^
Section => """"

At least with this proposal, we keep the same TOCs with Sphinx and Docbook.

=== Built in glossary ===

Sphinx has a [https://www.sphinx-doc.org/en/master/usage/restructuredtext/directives.html#glossary glossary directive]. From the documentation:

This directive must contain a reST definition list with terms and definitions. The definitions will then be referencable with the [https://www.sphinx-doc.org/en/master/usage/restructuredtext/roles.html#role-term 'term' role].

So anywhere in *any* manual, we can do :term:`VAR` to refer to an item from the glossary, and a link is created.

=== Global substitutions ===

The Yocto Project documentation makes heavy use of 'global' variables. In Docbook these 'variables' are stored in the file [http://git.yoctoproject.org/cgit.cgi/yocto-docs/tree/documentation/poky.ent poky.ent]. This Docbook feature is not handled automatically with Pandoc. Sphinx has builtin support for [https://www.sphinx-doc.org/en/master/usage/restructuredtext/basics.html#substitutions substitutions] however they are local to each reST file by default. They can be made global by using ''rst_prolog'':

rst_prolog
A string of reStructuredText that will be included at the beginning of every source file that is read.

As such we will keep a documentation configuration file with all global substitutions, similar to ''poky.ent''. Here is a sample code to define variables:

rst_prolog = """
.. |DISTRO| replace:: 3.1
.. |DISTRO_COMPRESSED| replace:: 31
.. |DISTRO_NAME_NO_CAP| replace:: dunfell
.. |DISTRO_NAME| replace:: Dunfell
.. |DISTRO_NAME_NO_CAP_MINUS_ONE| replace:: zeus
.. |DISTRO_NAME_MINUS_ONE| replace:: Zeus
.. |YOCTO_DOC_VERSION| replace:: 3.1
.. |YOCTO_DOC_VERSION_MINUS_ONE| replace:: 3.0.2
.. |DISTRO_REL_TAG| replace:: yocto-3.1
.. |METAINTELVERSION| replace:: 12.0
.. |REL_MONTH_YEAR| replace:: April 2020
"""

and use them:

This is the manual for version: |DISTRO|, aka |DISTRO_NAME|.

=== Note directive ===

Sphinx has a builtin ''note'' directive that produces clean Note section in the output file. There are various types of directives such as "attention", "caution", "danger", "error", "hint", "important", "tip", "warning", "admonition" that are supported, and additional directive can be added as Sphinx extension if needed.

=== Figures ===

Our documentation has many figures/images. Somehow Pandoc has completely ignored all images during the conversion. It might be worth investigating why, even though it might take more time than manually fixing all documents... Sphinx as a ''figure'' directive which is straight forward to use. To include a figure in the body of the documentation:

.. image:: figures/YP-flow-diagram.png

=== Links and References ===
From: https://sublime-and-sphinx-guide.readthedocs.io/en/latest/references.html

You can include links to other locations in the same document, to locations in other documents and to external websites.

You can link from text to a heading in any other part of the document by using the :ref: command with the heading text as the parameter. For example, this text in another part of this document would link to this section:
:ref:`Cross-References to Locations in the Same Document`

For that to work, we need sphinx.ext.autosectionlabel to be enabled in conf.py.

We can use Custom link text or create custom anchor instead of using the section text.

TODO: study https://sublime-and-sphinx-guide.readthedocs.io/en/latest/references.html#use-the-external-links-extension, looks like something we could use..

== Tasks list ==

Here is an initial tasks list for the migration:

# Proof of concept
## <s>Initial proof of concept patches</s>
## Merge initial patch set in master, along with Docbook
## <s>Update initial patches with yocto-docs recent changes </s>
# Convert all YP manuals
## <s>Automated conversion with pandoc</s>
### <s>adt-manual</s>
### <s>brief-yoctoprojectqs</s>
### <s>dev-manual</s>
### <s>kernel-dev</s>
### <s>profile-manual</s>
### <s>sdk-manual</s>
### <s>toaster-manual</s>
### <s>test-manual</s>
### <s>bitbake-manual</s>
## fix up all manuals for:
### links
### <s>figures</s>
### <s>headings</s>
### notes
### code blocks
### global substitutions variables
## Use current docs CSS file to create proper theme
### <s>initial import</s>
### tuning/optimization
## How to deal with each docbook 'first' page (license and TOC) (In progress RP and ndec looking at that)
## <s>Implement glossary</s>
# <s>Decide how to manage the mega manual. Should we create a single doc (index.rst) file per documentat, or a single one which ultimately would become the mega manual</s>
# <s.Custom Sphinx theme. The whole output can be themed using CSS and custom theme. We need to implement something that 'looks like' the Yocto Project and create beautiful rendering of our documentation</s>
# How to publish HTML online (per branch, latest, current, ...)? The whole process must be automated. (In progress, Michael on it)
# How to create PDF files?
# What's the impact of Google search and referencing?
# <s.Do we want to backport to dunfell LTS? (RP: Yes!)</s>
# <s>RP: Can we create a single page manual for ease of searching (some users need this and I personally use it a lot that way, I'm unlikely alone)</s>
# RP: Glossary headings don't appear to be externally linkable?

Documentation Sphinx

2020-09-04T23:58:53Z

Nicolas Dechesne:

== Yocto Project Documentation: Sphinx migration ==

The Yocto Project developers are currently investigating whether to move from Docbook to Sphinx (or any similar tools) for the project's documentation. The initial discussion can be found [https://lists.yoctoproject.org/g/docs/message/165 here].

Sphinx is a tool that makes it easy to create intelligent and beautiful documentation, written by Georg Brandl and licensed under the BSD license. It was originally created for the Python documentation.

See more on [https://www.sphinx-doc.org/en/master/ Sphinx website]. Furthermore Sphinx is designed to be [https://www.sphinx-doc.org/en/master/extdev/index.html extensible], and we can implement our own custom extensions, as Python modules, which will be executed during the generation of the documentation. This is a rather advance use of Sphinx, but could be very useful in the future.

This wiki page can be used as a working document to assist the discussions and/or the tasks. All discussions should be made on the [https://lists.yoctoproject.org/g/docs/ Yocto Project Docs mailing list].

== Sphinx migration for Yocto Project Documentation ==

The Sphinx migration is planned for 3.2 release, and the development is done in [https://git.yoctoproject.org/cgit/cgit.cgi/yocto-docs/log/?h=sphinx this branch], which produces the following [https://docs.yoctoproject.org/ HTML documentation]. Until the branch gets merged , patches should be sent against this branch.

A lot of work was already done to convert all the docbook files, but there is still some significant of work left to be done.

To use the proof of concept, please follow these instructions:

pip3 install sphinx
# to use the non default theme
pip3 install sphinx_rtd_theme

And then:

git clone https://git.yoctoproject.org/git/yocto-docs -b sphinx
cd yocto-docs/documentation
make -f Makefile.sphinx html

The resulting HTML index page will be _build/html/index.html.

== Sphinx proof of concept for Bitbake Documentation ==

A similar branch exists for the Bitbake documentation as well. The branch is available on [https://git.openembedded.org/bitbake/log/?h=sphinx here], and produces the following [https://people.linaro.org/~nicolas.dechesne/bitbake-docs/html/ HTML documentation].

To build the Bitbake documentation with Sphinx:

git clone https://git.openembedded.org/bitbake -b sphinx
cd bitbake/documentation
make -f Makefile.sphinx html

== Design ideas / principles ==

=== Automated conversion ===

The initial Docbook to Sphinx migration can be done with automated tools such as [https://pandoc.org/ pandoc]. While the conversion is far from being perfect, it produces some clean output markdown text file. After the initial automated conversion developers will need to fix up at least headings, images and links. In addition Sphinx has built in mechanisms (directives) that should be used to replace similar functions implemented in Docbook such as glossary, variables substitutions, notes, ...

To illustrate how Pandoc can be used, the initial conversion for bsp-guide, ref-manual and overview-manual was done with the following commands:

for i in *.xml; do pandoc -f docbook -t rst $i -o $i.rst; done

=== Headings ===

The layout of the Yocto Project manuals is organized as follows

Book
Chapter
Section
Section
Section

During the conversion with Pandoc, some of the headings are not converted properly and need to be fixed manually. Let's propose to use the following headings styles:

Book => overline ===
Chapter => overline ***
Section => ====
Section => ----
Section => ^^^^
Section => """"

At least with this proposal, we keep the same TOCs with Sphinx and Docbook.

=== Built in glossary ===

Sphinx has a [https://www.sphinx-doc.org/en/master/usage/restructuredtext/directives.html#glossary glossary directive]. From the documentation:

This directive must contain a reST definition list with terms and definitions. The definitions will then be referencable with the [https://www.sphinx-doc.org/en/master/usage/restructuredtext/roles.html#role-term 'term' role].

So anywhere in *any* manual, we can do :term:`VAR` to refer to an item from the glossary, and a link is created.

=== Global substitutions ===

The Yocto Project documentation makes heavy use of 'global' variables. In Docbook these 'variables' are stored in the file [http://git.yoctoproject.org/cgit.cgi/yocto-docs/tree/documentation/poky.ent poky.ent]. This Docbook feature is not handled automatically with Pandoc. Sphinx has builtin support for [https://www.sphinx-doc.org/en/master/usage/restructuredtext/basics.html#substitutions substitutions] however they are local to each reST file by default. They can be made global by using ''rst_prolog'':

rst_prolog
A string of reStructuredText that will be included at the beginning of every source file that is read.

As such we will keep a documentation configuration file with all global substitutions, similar to ''poky.ent''. Here is a sample code to define variables:

rst_prolog = """
.. |DISTRO| replace:: 3.1
.. |DISTRO_COMPRESSED| replace:: 31
.. |DISTRO_NAME_NO_CAP| replace:: dunfell
.. |DISTRO_NAME| replace:: Dunfell
.. |DISTRO_NAME_NO_CAP_MINUS_ONE| replace:: zeus
.. |DISTRO_NAME_MINUS_ONE| replace:: Zeus
.. |YOCTO_DOC_VERSION| replace:: 3.1
.. |YOCTO_DOC_VERSION_MINUS_ONE| replace:: 3.0.2
.. |DISTRO_REL_TAG| replace:: yocto-3.1
.. |METAINTELVERSION| replace:: 12.0
.. |REL_MONTH_YEAR| replace:: April 2020
"""

and use them:

This is the manual for version: |DISTRO|, aka |DISTRO_NAME|.

=== Note directive ===

Sphinx has a builtin ''note'' directive that produces clean Note section in the output file. There are various types of directives such as "attention", "caution", "danger", "error", "hint", "important", "tip", "warning", "admonition" that are supported, and additional directive can be added as Sphinx extension if needed.

=== Figures ===

Our documentation has many figures/images. Somehow Pandoc has completely ignored all images during the conversion. It might be worth investigating why, even though it might take more time than manually fixing all documents... Sphinx as a ''figure'' directive which is straight forward to use. To include a figure in the body of the documentation:

.. image:: figures/YP-flow-diagram.png

=== Links and References ===
From: https://sublime-and-sphinx-guide.readthedocs.io/en/latest/references.html

You can include links to other locations in the same document, to locations in other documents and to external websites.

You can link from text to a heading in any other part of the document by using the :ref: command with the heading text as the parameter. For example, this text in another part of this document would link to this section:
:ref:`Cross-References to Locations in the Same Document`

For that to work, we need sphinx.ext.autosectionlabel to be enabled in conf.py.

We can use Custom link text or create custom anchor instead of using the section text.

TODO: study https://sublime-and-sphinx-guide.readthedocs.io/en/latest/references.html#use-the-external-links-extension, looks like something we could use..

== Tasks list ==

Here is an initial tasks list for the migration:

# Proof of concept
## <s>Initial proof of concept patches</s>
## Merge initial patch set in master, along with Docbook
## <s>Update initial patches with yocto-docs recent changes </s>
# Convert all YP manuals
## <s>Automated conversion with pandoc</s>
### <s>adt-manual</s>
### <s>brief-yoctoprojectqs</s>
### <s>dev-manual</s>
### <s>kernel-dev</s>
### <s>profile-manual</s>
### <s>sdk-manual</s>
### <s>toaster-manual</s>
### <s>test-manual</s>
### <s>bitbake-manual</s>
## fix up all manuals for:
### links
### <s>figures</s?
### <s>headings</s>
### notes
### global substitutions variables
## Use current docs CSS file to create proper theme
### <s>initial import</s>
### tuning/optimization
## How to deal with each docbook 'first' page (license and TOC) (In progress RP and ndec looking at that)
## <s>Implement glossary</s>
# <s>Decide how to manage the mega manual. Should we create a single doc (index.rst) file per documentat, or a single one which ultimately would become the mega manual</s>
# <s.Custom Sphinx theme. The whole output can be themed using CSS and custom theme. We need to implement something that 'looks like' the Yocto Project and create beautiful rendering of our documentation</s>
# How to publish HTML online (per branch, latest, current, ...)? The whole process must be automated. (In progress, Michael on it)
# How to create PDF files?
# What's the impact of Google search and referencing?
# <s.Do we want to backport to dunfell LTS? (RP: Yes!)</s>
# <s>RP: Can we create a single page manual for ease of searching (some users need this and I personally use it a lot that way, I'm unlikely alone)</s>
# RP: Glossary headings don't appear to be externally linkable?

Documentation Sphinx

2020-07-01T23:16:35Z

Nicolas Dechesne: /* Tasks list */

== Yocto Project Documentation: Sphinx migration ==

The Yocto Project developers are currently investigating whether to move from Docbook to Sphinx (or any similar tools) for the project's documentation. The initial discussion can be found [https://lists.yoctoproject.org/g/docs/message/165 here].

Sphinx is a tool that makes it easy to create intelligent and beautiful documentation, written by Georg Brandl and licensed under the BSD license. It was originally created for the Python documentation.

See more on [https://www.sphinx-doc.org/en/master/ Sphinx website]. Furthermore Sphinx is designed to be [https://www.sphinx-doc.org/en/master/extdev/index.html extensible], and we can implement our own custom extensions, as Python modules, which will be executed during the generation of the documentation. This is a rather advance use of Sphinx, but could be very useful in the future.

This wiki page can be used as a working document to assist the discussions and/or the tasks. All discussions should be made on the [https://lists.yoctoproject.org/g/docs/ Yocto Project Docs mailing list].

== Sphinx proof of concept for Yocto Project Documentation ==

An experimental branch with some initial Sphinx support is available on [https://git.yoctoproject.org/cgit/cgit.cgi/yocto-docs/log/?h=sphinx here], which produces the following [https://people.linaro.org/~nicolas.dechesne/yp-docs/html HTML documentation].

The support is very minimal, and only 3 documents were ''converted'', but the idea was to share an initial look and feel about both the output and the changes to the documentation source code. There is a large amount of work to accomplish if we want to convert the whole documentation!

To use the proof of concept, please follow these instructions:

pip3 install sphinx
# to use the non default theme
pip3 install sphinx_rtd_theme

And then:

git clone https://git.yoctoproject.org/git/yocto-docs -b sphinx
cd yocto-docs/documentation
make -f Makefile.sphinx html

The resulting HTML index page will be _build/html/index.html.

== Sphinx proof of concept for Bitbake Documentation ==

A similar experimental branch exists for the Bitbake documentation as well. The branch is available on [https://git.openembedded.org/bitbake/log/?h=sphinx here], and produces the following [https://people.linaro.org/~nicolas.dechesne/bitbake-docs/html/ HTML documentation].

To build the Bitbake documentation with Sphinx:

git clone https://git.openembedded.org/bitbake -b sphinx
cd bitbake/documentation
make -f Makefile.sphinx html

== Design ideas / principles ==

=== Automated conversion ===

The initial Docbook to Sphinx migration can be done with automated tools such as [https://pandoc.org/ pandoc]. While the conversion is far from being perfect, it produces some clean output markdown text file. After the initial automated conversion developers will need to fix up at least headings, images and links. In addition Sphinx has built in mechanisms (directives) that should be used to replace similar functions implemented in Docbook such as glossary, variables substitutions, notes, ...

To illustrate how Pandoc can be used, the initial conversion for bsp-guide, ref-manual and overview-manual was done with the following commands:

for i in *.xml; do pandoc -f docbook -t rst $i -o $i.rst; done

=== Headings ===

The layout of the Yocto Project manuals is organized as follows

Book
Chapter
Section
Section
Section

During the conversion with Pandoc, some of the headings are not converted properly and need to be fixed manually. Let's propose to use the following headings styles:

Book => overline ===
Chapter => overline ***
Section => ====
Section => ----
Section => ^^^^
Section => """"

At least with this proposal, we keep the same TOCs with Sphinx and Docbook.

=== Built in glossary ===

Sphinx has a [https://www.sphinx-doc.org/en/master/usage/restructuredtext/directives.html#glossary glossary directive]. From the documentation:

This directive must contain a reST definition list with terms and definitions. The definitions will then be referencable with the [https://www.sphinx-doc.org/en/master/usage/restructuredtext/roles.html#role-term 'term' role].

So anywhere in *any* manual, we can do :term:`VAR` to refer to an item from the glossary, and a link is created.

=== Global substitutions ===

The Yocto Project documentation makes heavy use of 'global' variables. In Docbook these 'variables' are stored in the file [http://git.yoctoproject.org/cgit.cgi/yocto-docs/tree/documentation/poky.ent poky.ent]. This Docbook feature is not handled automatically with Pandoc. Sphinx has builtin support for [https://www.sphinx-doc.org/en/master/usage/restructuredtext/basics.html#substitutions substitutions] however they are local to each reST file by default. They can be made global by using ''rst_prolog'':

rst_prolog
A string of reStructuredText that will be included at the beginning of every source file that is read.

As such we will keep a documentation configuration file with all global substitutions, similar to ''poky.ent''. Here is a sample code to define variables:

rst_prolog = """
.. |DISTRO| replace:: 3.1
.. |DISTRO_COMPRESSED| replace:: 31
.. |DISTRO_NAME_NO_CAP| replace:: dunfell
.. |DISTRO_NAME| replace:: Dunfell
.. |DISTRO_NAME_NO_CAP_MINUS_ONE| replace:: zeus
.. |DISTRO_NAME_MINUS_ONE| replace:: Zeus
.. |YOCTO_DOC_VERSION| replace:: 3.1
.. |YOCTO_DOC_VERSION_MINUS_ONE| replace:: 3.0.2
.. |DISTRO_REL_TAG| replace:: yocto-3.1
.. |METAINTELVERSION| replace:: 12.0
.. |REL_MONTH_YEAR| replace:: April 2020
"""

and use them:

This is the manual for version: |DISTRO|, aka |DISTRO_NAME|.

=== Note directive ===

Sphinx has a builtin ''note'' directive that produces clean Note section in the output file. There are various types of directives such as "attention", "caution", "danger", "error", "hint", "important", "tip", "warning", "admonition" that are supported, and additional directive can be added as Sphinx extension if needed.

=== Figures ===

Our documentation has many figures/images. Somehow Pandoc has completely ignored all images during the conversion. It might be worth investigating why, even though it might take more time than manually fixing all documents... Sphinx as a ''figure'' directive which is straight forward to use. To include a figure in the body of the documentation:

.. image:: figures/YP-flow-diagram.png

=== Links and References ===
From: https://sublime-and-sphinx-guide.readthedocs.io/en/latest/references.html

You can include links to other locations in the same document, to locations in other documents and to external websites.

You can link from text to a heading in any other part of the document by using the :ref: command with the heading text as the parameter. For example, this text in another part of this document would link to this section:
:ref:`Cross-References to Locations in the Same Document`

For that to work, we need sphinx.ext.autosectionlabel to be enabled in conf.py.

We can use Custom link text or create custom anchor instead of using the section text.

TODO: study https://sublime-and-sphinx-guide.readthedocs.io/en/latest/references.html#use-the-external-links-extension, looks like something we could use..

== Tasks list ==

Here is an initial tasks list for the migration:

# Proof of concept
## <s>Initial proof of concept patches</s>
## Merge initial patch set in master, along with Docbook
## <s>Update initial patches with yocto-docs recent changes </s>
# Convert all YP manuals
## Automated conversion with pandoc
### <s>adt-manual</s>
### <s>brief-yoctoprojectqs</s>
### <s>dev-manual</s>
### <s>kernel-dev</s>
### <s>profile-manual</s>
### <s>sdk-manual</s>
### <s>toaster-manual</s>
### <s>test-manual</s>
### <s>bitbake-manual</s>
## fix up for links, figures, headings, notes
### adt-manual
### brief-yoctoprojectqs
### bsp-guide
### dev-manual
### kernel-dev
### overview-manual
### profile-manual
### ref-manual
### sdk-manual
### toaster-manual
### test-manual
### bitbake-manual
## What to do for *.css and *.xsl Docbook files
## How to deal with each docbook 'first' page (license and TOC)
## Implement global substitutions variables
## Implement glossary
# Decide how to manage the mega manual. Should we create a single doc (index.rst) file per documentat, or a single one which ultimately would become the mega manual
# Custom Sphinx theme. The whole output can be themed using CSS and custom theme. We need to implement something that 'looks like' the Yocto Project and create beautiful rendering of our documentation
# How to publish HTML online (per branch, latest, current, ...)? The whole process must be automated.
# How to create PDF files?
# What's the impact of Google search and referencing?
# Do we want to backport to dunfell LTS? (RP: Yes!)
# RP: Can we create a single page manual for ease of searching (some users need this and I personally use it a lot that way, I'm unlikely alone)?
# RP: Glossary headings don't appear to be externally linkable?

Documentation Sphinx

2020-07-01T23:16:10Z

Nicolas Dechesne: /* Sphinx proof of concept */

== Yocto Project Documentation: Sphinx migration ==

The Yocto Project developers are currently investigating whether to move from Docbook to Sphinx (or any similar tools) for the project's documentation. The initial discussion can be found [https://lists.yoctoproject.org/g/docs/message/165 here].

Sphinx is a tool that makes it easy to create intelligent and beautiful documentation, written by Georg Brandl and licensed under the BSD license. It was originally created for the Python documentation.

See more on [https://www.sphinx-doc.org/en/master/ Sphinx website]. Furthermore Sphinx is designed to be [https://www.sphinx-doc.org/en/master/extdev/index.html extensible], and we can implement our own custom extensions, as Python modules, which will be executed during the generation of the documentation. This is a rather advance use of Sphinx, but could be very useful in the future.

This wiki page can be used as a working document to assist the discussions and/or the tasks. All discussions should be made on the [https://lists.yoctoproject.org/g/docs/ Yocto Project Docs mailing list].

== Sphinx proof of concept for Yocto Project Documentation ==

An experimental branch with some initial Sphinx support is available on [https://git.yoctoproject.org/cgit/cgit.cgi/yocto-docs/log/?h=sphinx here], which produces the following [https://people.linaro.org/~nicolas.dechesne/yp-docs/html HTML documentation].

The support is very minimal, and only 3 documents were ''converted'', but the idea was to share an initial look and feel about both the output and the changes to the documentation source code. There is a large amount of work to accomplish if we want to convert the whole documentation!

To use the proof of concept, please follow these instructions:

pip3 install sphinx
# to use the non default theme
pip3 install sphinx_rtd_theme

And then:

git clone https://git.yoctoproject.org/git/yocto-docs -b sphinx
cd yocto-docs/documentation
make -f Makefile.sphinx html

The resulting HTML index page will be _build/html/index.html.

== Sphinx proof of concept for Bitbake Documentation ==

A similar experimental branch exists for the Bitbake documentation as well. The branch is available on [https://git.openembedded.org/bitbake/log/?h=sphinx here], and produces the following [https://people.linaro.org/~nicolas.dechesne/bitbake-docs/html/ HTML documentation].

To build the Bitbake documentation with Sphinx:

git clone https://git.openembedded.org/bitbake -b sphinx
cd bitbake/documentation
make -f Makefile.sphinx html

== Design ideas / principles ==

=== Automated conversion ===

The initial Docbook to Sphinx migration can be done with automated tools such as [https://pandoc.org/ pandoc]. While the conversion is far from being perfect, it produces some clean output markdown text file. After the initial automated conversion developers will need to fix up at least headings, images and links. In addition Sphinx has built in mechanisms (directives) that should be used to replace similar functions implemented in Docbook such as glossary, variables substitutions, notes, ...

To illustrate how Pandoc can be used, the initial conversion for bsp-guide, ref-manual and overview-manual was done with the following commands:

for i in *.xml; do pandoc -f docbook -t rst $i -o $i.rst; done

=== Headings ===

The layout of the Yocto Project manuals is organized as follows

Book
Chapter
Section
Section
Section

During the conversion with Pandoc, some of the headings are not converted properly and need to be fixed manually. Let's propose to use the following headings styles:

Book => overline ===
Chapter => overline ***
Section => ====
Section => ----
Section => ^^^^
Section => """"

At least with this proposal, we keep the same TOCs with Sphinx and Docbook.

=== Built in glossary ===

Sphinx has a [https://www.sphinx-doc.org/en/master/usage/restructuredtext/directives.html#glossary glossary directive]. From the documentation:

This directive must contain a reST definition list with terms and definitions. The definitions will then be referencable with the [https://www.sphinx-doc.org/en/master/usage/restructuredtext/roles.html#role-term 'term' role].

So anywhere in *any* manual, we can do :term:`VAR` to refer to an item from the glossary, and a link is created.

=== Global substitutions ===

The Yocto Project documentation makes heavy use of 'global' variables. In Docbook these 'variables' are stored in the file [http://git.yoctoproject.org/cgit.cgi/yocto-docs/tree/documentation/poky.ent poky.ent]. This Docbook feature is not handled automatically with Pandoc. Sphinx has builtin support for [https://www.sphinx-doc.org/en/master/usage/restructuredtext/basics.html#substitutions substitutions] however they are local to each reST file by default. They can be made global by using ''rst_prolog'':

rst_prolog
A string of reStructuredText that will be included at the beginning of every source file that is read.

As such we will keep a documentation configuration file with all global substitutions, similar to ''poky.ent''. Here is a sample code to define variables:

rst_prolog = """
.. |DISTRO| replace:: 3.1
.. |DISTRO_COMPRESSED| replace:: 31
.. |DISTRO_NAME_NO_CAP| replace:: dunfell
.. |DISTRO_NAME| replace:: Dunfell
.. |DISTRO_NAME_NO_CAP_MINUS_ONE| replace:: zeus
.. |DISTRO_NAME_MINUS_ONE| replace:: Zeus
.. |YOCTO_DOC_VERSION| replace:: 3.1
.. |YOCTO_DOC_VERSION_MINUS_ONE| replace:: 3.0.2
.. |DISTRO_REL_TAG| replace:: yocto-3.1
.. |METAINTELVERSION| replace:: 12.0
.. |REL_MONTH_YEAR| replace:: April 2020
"""

and use them:

This is the manual for version: |DISTRO|, aka |DISTRO_NAME|.

=== Note directive ===

Sphinx has a builtin ''note'' directive that produces clean Note section in the output file. There are various types of directives such as "attention", "caution", "danger", "error", "hint", "important", "tip", "warning", "admonition" that are supported, and additional directive can be added as Sphinx extension if needed.

=== Figures ===

Our documentation has many figures/images. Somehow Pandoc has completely ignored all images during the conversion. It might be worth investigating why, even though it might take more time than manually fixing all documents... Sphinx as a ''figure'' directive which is straight forward to use. To include a figure in the body of the documentation:

.. image:: figures/YP-flow-diagram.png

=== Links and References ===
From: https://sublime-and-sphinx-guide.readthedocs.io/en/latest/references.html

You can include links to other locations in the same document, to locations in other documents and to external websites.

You can link from text to a heading in any other part of the document by using the :ref: command with the heading text as the parameter. For example, this text in another part of this document would link to this section:
:ref:`Cross-References to Locations in the Same Document`

For that to work, we need sphinx.ext.autosectionlabel to be enabled in conf.py.

We can use Custom link text or create custom anchor instead of using the section text.

TODO: study https://sublime-and-sphinx-guide.readthedocs.io/en/latest/references.html#use-the-external-links-extension, looks like something we could use..

== Tasks list ==

Here is an initial tasks list for the migration:

# Proof of concept
## <s>Initial proof of concept patches</s>
## Merge initial patch set in master, along with Docbook
## <s>Update initial patches with yocto-docs recent changes </s>
# Convert all YP manuals
## Automated conversion with pandoc
### <s>adt-manual</s>
### <s>brief-yoctoprojectqs</s>
### <s>dev-manual</s>
### <s>kernel-dev</s>
### <s>profile-manual</s>
### <s>sdk-manual</s>
### <s>toaster-manual</s>
### <s>test-manual</s>
### bitbake-manual
## fix up for links, figures, headings, notes
### adt-manual
### brief-yoctoprojectqs
### bsp-guide
### dev-manual
### kernel-dev
### overview-manual
### profile-manual
### ref-manual
### sdk-manual
### toaster-manual
### test-manual
### bitbake-manual
## What to do for *.css and *.xsl Docbook files
## How to deal with each docbook 'first' page (license and TOC)
## Implement global substitutions variables
## Implement glossary
# Decide how to manage the mega manual. Should we create a single doc (index.rst) file per documentat, or a single one which ultimately would become the mega manual
# Custom Sphinx theme. The whole output can be themed using CSS and custom theme. We need to implement something that 'looks like' the Yocto Project and create beautiful rendering of our documentation
# How to publish HTML online (per branch, latest, current, ...)? The whole process must be automated.
# How to create PDF files?
# What's the impact of Google search and referencing?
# Do we want to backport to dunfell LTS? (RP: Yes!)
# RP: Can we create a single page manual for ease of searching (some users need this and I personally use it a lot that way, I'm unlikely alone)?
# RP: Glossary headings don't appear to be externally linkable?

Documentation Sphinx

2020-06-30T18:28:00Z

Nicolas Dechesne: /* Tasks list */

Documentation Sphinx

2020-06-26T11:10:18Z

Nicolas Dechesne: /* Sphinx proof of concept */

Documentation Sphinx

2020-06-19T17:04:47Z

Nicolas Dechesne: /* Design ideas / principles */

Documentation Sphinx

2020-06-17T16:46:59Z

Nicolas Dechesne: /* Tasks list */

Documentation Sphinx

2020-06-17T15:39:07Z

Nicolas Dechesne: /* Tasks list */

Documentation Sphinx

2020-06-17T15:35:39Z

Nicolas Dechesne: /* Sphinx proof of concept */

Documentation Sphinx

2020-06-17T15:26:50Z

Nicolas Dechesne: /* Sphinx proof of concept */

Project Users

2020-06-04T18:29:29Z

Nicolas Dechesne:

It's hard to know which companies or projects use the Yocto Project since there is no requirement to tell us. This list is here to informally collate the companies, projects and products that use the Yocto Project in some way.

= Companies using the Yocto Project =

== Semiconductor Vendors ==
* AMD (Silver Member)
* ARM (Platinum Member)
* Intel (Platinum Member)
* Microchip
* NXP (Silver Member)
* Qualcomm
* Renesas (Gold Member)
* STMicroelectronics (Silver Member)
* Texas Instruments (Platinum Member)
* Xilinx (Platinum Member)

== Operating System Vendors ==
* ENEA (Silver Member)
* Linaro (Silver Member)
* Lineo (Silver Member)
* Mentor Graphics (Gold Member)
* Montavista (Silver Member)
* Wind River (Gold Member)

== Others ==
* Cisco (Platinum Member)
* Comcast (Platinum Member)
* Dell (Silver Member)
* Facebook (Platinum Member)
* Juniper (Gold Member)
* [https://koansoftware.com/ KOAN sas]
* Lexmark
* LG (Silver Member)
* Microsoft
* StreamUnlimited Engineering GmbH
* Fujitsu
* BMW
* BMW Car IT
* National Instruments
* Dynamic Devices
* Axis
* Atlas Copco
* Kodak
* Smile
* General Electric
* [https://www.witekio.com/values-expertise/ Witekio]

= Products that use the Yocto Project =

* BMW cars
* Comcast set top boxes
* LG TVs
* Lexmark Printers
* [https://www.streamunlimited.com/hardware-modules/ StreamUnlimited hardware modules for voice assistants and connected speakers]
* Vernier LabQuest

= Projects that use the Yocto Project =

* [https://www.automotivelinux.org/ Automotive Grade Linux (AGL)]
* Comcast RDK
* OpenBMC
* Windows Subsystem Linux (v1+v2)
* [https://oryx-linux.org/ Oryx Linux]
* [https://www.streamunlimited.com/stream-sdk/ StreamSDK for voice assistants and connected speakers]
* Nvidia vibrante SDK [https://docs.nvidia.com/drive/archive/4.1.8.0L/nvoss_docs/index.html]

Documentation Sphinx

2020-05-26T17:07:27Z

Nicolas Dechesne: /* Tasks list */

Documentation Sphinx

2020-05-26T14:53:23Z

Nicolas Dechesne: /* Sphinx proof of concept */

Documentation Sphinx

2020-05-26T09:32:28Z

Nicolas Dechesne:

Documentation Sphinx

2020-05-26T09:31:35Z

Nicolas Dechesne:

Documentation Sphinx

2020-05-26T09:01:16Z

Nicolas Dechesne: Created page with "== Yocto Project Documentation: Sphinx migration == The Yocto Project developers are currently investigating whether to move from Docbook to Sphinx (or any similar tools) fo..."

Project Users

2020-05-15T06:35:05Z

Nicolas Dechesne: /* Companies using the Yocto Project */

It's hard to know which companies or projects use the Yocto Project since there is no requirement to tell us. This list is here to informally collate the companies, projects and products that use the Yocto Project in some way.

= Companies using the Yocto Project =

== Semiconductor Vendors ==
* AMD (Silver Member)
* ARM (Platinum Member)
* Intel (Platinum Member)
* Microchip
* NXP (Silver Member)
* Qualcomm
* Renesas (Gold Member)
* STMicroelectronics (Silver Member)
* Texas Instruments (Platinum Member)
* Xilinx (Platinum Member)

== Operating System Vendors ==
* ENEA (Silver Member)
* Linaro (Silver Member)
* Lineo (Silver Member)
* Mentor Graphics (Gold Member)
* Montavista (Silver Member)
* Wind River (Gold Member)

== Others ==
* Cisco (Platinum Member)
* Comcast (Platinum Member)
* Dell (Silver Member)
* Facebook (Platinum Member)
* Juniper (Gold Member)
* [https://koansoftware.com/ KOAN sas]
* Lexmark
* LG (Silver Member)
* Microsoft
* StreamUnlimited Engineering GmbH
* Fujitsu
* BMW
* BMW Car IT
* National Instruments
* Dynamic Devices
* Axis
* Atlas Copco
* Kodak
* Smile
* General Electric
* Witekio

= Products that use the Yocto Project =

* BMW cars
* Comcast set top boxes
* LG TVs
* Lexmark Printers
* [https://www.streamunlimited.com/hardware-modules/ StreamUnlimited hardware modules for voice assistants and connected speakers]
* Vernier LabQuest

= Projects that use the Yocto Project =

* [https://www.automotivelinux.org/ Automotive Grade Linux (AGL)]
* Comcast RDK
* OpenBMC
* Windows Subsystem Linux (v1+v2)
* [https://oryx-linux.org/ Oryx Linux]
* [https://www.streamunlimited.com/stream-sdk/ StreamSDK for voice assistants and connected speakers]
* Nvidia vibrante SDK [https://docs.nvidia.com/drive/archive/4.1.8.0L/nvoss_docs/index.html]

SeasonOfDocs

2020-04-29T11:47:25Z

Nicolas Dechesne: /* Yocto Project Season of Docs */

SeasonOfDocs

2020-04-29T11:47:05Z

Nicolas Dechesne: /* Yocto Project Season of Docs */

SeasonOfDocs

2020-04-29T11:42:21Z

Nicolas Dechesne: /* Yocto Project Season of Docs */

SeasonOfDocs

2020-04-29T07:30:33Z

Nicolas Dechesne: /* Format Overhaul */

SeasonOfDocs

2020-04-28T21:15:52Z

Nicolas Dechesne: Created page with "= Yocto Project Season of Code ="

= Yocto Project Season of Code =

User talk:Joshua Watt

2020-02-05T17:45:49Z

Nicolas Dechesne: Welcome!

'''Welcome to ''Yocto Project''!'''
We hope you will contribute much and well.
You will probably want to read the [[Help:Contents|help pages]].
Again, welcome and have fun! [[User:Nicolas Dechesne|Nicolas Dechesne]] ([[User talk:Nicolas Dechesne|talk]]) 09:45, 5 February 2020 (PST)

User:Joshua Watt

2020-02-05T17:45:49Z

Nicolas Dechesne: Creating user page for new user.

Embedded Software Engineer working for Garmin

User talk:Mcfrisk

2020-02-05T17:45:30Z

Nicolas Dechesne: Welcome!

User:Mcfrisk

2020-02-05T17:45:30Z

Nicolas Dechesne: Creating user page for new user.

Embedded SW developer. Contributor to Yocto Project.

GSoC

2020-02-04T22:00:20Z

Nicolas Dechesne: /* Links */

= Yocto Project Summer of Code =
This is the Yocto Project specific page for [https://summerofcode.withgoogle.com/ Google Summer of Code]. The Yocto Project is applying to be a mentoring organization for 2020 and is recruiting mentors and students.

Prospective mentors are requested to assist with the ideas page, add themselves to the list, join the dedicated [https://lists.yoctoproject.org/g/gsoc/ mailing list] and contact Nicolas Dechesne to be invited to be a mentor on Google's tool.

Students are requested to engage possible mentors and may begin submitting applications for Yocto Project related projects starting now. This is the first time the project is engaging with Google Summer Of Code, and we are hoping it will be successful attempt. We recommend all students to engage with us now to help make your application strong!

Here are some links to help students with their applications:
* [https://summerofcode.withgoogle.com/ GSoC home]
* [TBD Yocto Project GSoC page]
* [http://drupal.org/node/59037 Guide from drupal.org]

== Projects ==
* Current ideas page: [[GSoC/Ideas|Project Ideas for 2020]]
* [https://lists.yoctoproject.org/g/gsoc/ Yocto Project GSoC Mailing list]

== Key Dates for 2020 ==
{| class="wikitable"
| January 14 19:00 UTC || Mentoring organizations can begin submitting applications to Google
|-
| February 5 19:00 UTC || Mentoring organization application deadline
|-
| February 5 - February 19 || Google program administrators review organization applications
|-
| February 20 18:00 UTC || List of accepted mentoring organizations published
|-
| February 20 - March 16 || Potential student participants discuss application ideas with mentoring organizations
|-
| March 16 18:00 UTC || Student application period begins
|-
| March 31 18:00 UTC || Student application deadline
|-
| April 14 18:00 UTC || Student slot requests due from Org Admins
|-
| April 23 18:00 UTC || Student Project selections due from Org Admins
|-
| April 27 18:00 UTC || Accepted student projects announced
|-
| Community Bonding Period || Students get to know mentors, read documentation, get up to speed to begin working on their projects
|-
| May 18 || Coding officially begins!
|-
| June 15 18:00 UTC || Mentors and students can begin submitting Phase 1 evaluations
|-
| June 19 18:00 UTC || Phase 1 Evaluation deadline
|-
| Work Period || Students work on their project with guidance from Mentors
|-
| July 13 18:00 UTC || Mentors and students can begin submitting Phase 2 evaluations
|-
| July 17 18:00 UTC || Phase 2 Evaluation deadline
|-
| Work Period || Students continue working on their project with guidance from Mentors
|-
| August 10 - 17 18:00 UTC || Final week: Students submit their final work product and their final mentor evaluation
|-
| August 17 - 24 18:00 UTC || Mentors submit final student evaluations
|-
| August 25 || Final results of Google Summer of Code 2020 announced
|-
| October || Mentor Summit
|}

== Links ==
* [https://lists.yoctoproject.org/g/gsoc/ Yocto Project GSoC Mailing list]
* [[GSoC/Ideas|Summer of Code Ideas]]
* [[GSoC/Application|Our project proposal guidelines]]
* Our application to become a mentoring organization. [[GSoC/Application/2020|Application]]
* [http://video.fosdem.org/2009/maintracks/google_soc.xvid.avi FOSDEM video from 2009 about the Summer of Code]