Documentation Sphinx

Yocto Project Documentation: Sphinx migration

The Yocto Project developers are currently investigating whether to move from Docbook to Sphinx (or any similar tools) for the project's documentation. The initial discussion can be found here.

   Sphinx is a tool that makes it easy to create intelligent and beautiful documentation, written by Georg Brandl and licensed under the BSD license. It was originally created for the Python documentation.

See more on Sphinx website. Furthermore Sphinx is designed to be extensible, and we can implement our own custom extensions, as Python modules, which will be executed during the generation of the documentation. This is a rather advance use of Sphinx, but could be very useful in the future.

This wiki page can be used as a working document to assist the discussions and/or the tasks. All discussions should be made on the Yocto Project Docs mailing list.

Sphinx migration for Yocto Project Documentation

The Sphinx migration is planned for 3.2 release, and the development is done in this branch, which produces the following HTML documentation. Until the branch gets merged , patches should be sent against this branch.

A lot of work was already done to convert all the docbook files, but there is still some significant of work left to be done.

To use the proof of concept, please follow these instructions:

   pip3 install sphinx
   # to use the non default theme
   pip3 install sphinx_rtd_theme
   # your system may also need
   pip3 install pyyaml

And then:

   git clone https://git.yoctoproject.org/git/yocto-docs -b sphinx
   cd yocto-docs/documentation
   make -f Makefile.sphinx html

The resulting HTML index page will be _build/html/index.html.

Sphinx proof of concept for Bitbake Documentation

A similar branch exists for the Bitbake documentation as well. The branch is available on here, and produces the following HTML documentation.

To build the Bitbake documentation with Sphinx:

   git clone https://git.openembedded.org/bitbake -b sphinx
   cd bitbake/doc
   make -f Makefile.sphinx html

Design ideas / principles

Automated conversion

The initial Docbook to Sphinx migration can be done with automated tools such as pandoc. While the conversion is far from being perfect, it produces some clean output markdown text file. After the initial automated conversion developers will need to fix up at least headings, images and links. In addition Sphinx has built in mechanisms (directives) that should be used to replace similar functions implemented in Docbook such as glossary, variables substitutions, notes, ...

To illustrate how Pandoc can be used, the initial conversion for bsp-guide, ref-manual and overview-manual was done with the following commands:

   for i in *.xml; do pandoc -f docbook -t rst $i -o $i.rst; done

Headings

The layout of the Yocto Project manuals is organized as follows

   Book
     Chapter
       Section
         Section
           Section

During the conversion with Pandoc, some of the headings are not converted properly and need to be fixed manually. Let's propose to use the following headings styles:

   Book              => overline ===
     Chapter         => overline ***
       Section       => ====
         Section     => ----
           Section   => ^^^^
             Section => """"

At least with this proposal, we keep the same TOCs with Sphinx and Docbook.

Built in glossary

Sphinx has a glossary directive. From the documentation:

   This directive must contain a reST definition list with terms and definitions. The definitions will then be referencable with the 'term' role.

So anywhere in *any* manual, we can do :term:`VAR` to refer to an item from the glossary, and a link is created.

Global substitutions

The Yocto Project documentation makes heavy use of 'global' variables. In Docbook these 'variables' are stored in the file poky.ent. This Docbook feature is not handled automatically with Pandoc. Sphinx has builtin support for substitutions however they are local to each reST file by default. They can be made global by using rst_prolog:

   rst_prolog
       A string of reStructuredText that will be included at the beginning of every source file that is read.

As such we will keep a documentation configuration file with all global substitutions, similar to poky.ent. Here is a sample code to define variables:

   rst_prolog = """
   .. |DISTRO| replace:: 3.1
   .. |DISTRO_COMPRESSED| replace:: 31
   .. |DISTRO_NAME_NO_CAP| replace:: dunfell
   .. |DISTRO_NAME| replace:: Dunfell
   .. |DISTRO_NAME_NO_CAP_MINUS_ONE| replace:: zeus
   .. |DISTRO_NAME_MINUS_ONE| replace:: Zeus
   .. |YOCTO_DOC_VERSION| replace:: 3.1
   .. |YOCTO_DOC_VERSION_MINUS_ONE| replace:: 3.0.2
   .. |DISTRO_REL_TAG| replace:: yocto-3.1
   .. |METAINTELVERSION| replace:: 12.0
   .. |REL_MONTH_YEAR| replace:: April 2020
   """

and use them:

   This is the manual for version: |DISTRO|, aka |DISTRO_NAME|.

However, there are serious shortcomings with Sphinx built-in, for example they can be used/nested inside code-block sections. Another mechanism was implemented so that we can continue to use 'variables' within our documentation. A Sphinx extension was implemented to support variable substitutions while "parsing" the .rst files. The pattern for variables substitutions is: {VAR}. The implementation of the extension can be found here: https://git.yoctoproject.org/cgit/cgit.cgi/yocto-docs/tree/documentation/sphinx/yocto-vars.py?h=sphinx. All variables are set in a file call poky.yaml, which was generated from poky.ent: https://git.yoctoproject.org/cgit/cgit.cgi/yocto-docs/tree/documentation/poky.yaml?h=sphinx. For example, the following .rst content will produce the 'expected' content:

   .. code-block:: 
       $ mkdir ~/poky-{DISTRO}
       or 
       $ git clone {YOCTO_GIT_URL}/git/poky -b {DISTRO_NAME_NO_CAP}

Variables can be nested, like it was the case for DocBook.

Note directive

Sphinx has a builtin note directive that produces clean Note section in the output file. There are various types of directives such as "attention", "caution", "danger", "error", "hint", "important", "tip", "warning", "admonition" that are supported, and additional directive can be added as Sphinx extension if needed.

Figures

Our documentation has many figures/images. Somehow Pandoc has completely ignored all images during the conversion. It might be worth investigating why, even though it might take more time than manually fixing all documents... Sphinx as a figure directive which is straight forward to use. To include a figure in the body of the documentation:

   .. image:: figures/YP-flow-diagram.png

Links and References

Please read: https://sublime-and-sphinx-guide.readthedocs.io/en/latest/references.html

You can include links to other locations in the same document, to locations in other documents and to external websites.

references

We enable this extension by default: sphinx.ext.autosectionlabel.

You can link from text to a heading in any other part of the document by using the :ref: command with the heading text as the parameter. For example, this text in another part of this document would link to this section:

   :ref:`Cross-References to Locations in the Same Document`

We can use Custom link text or create custom anchor instead of using the section text:

   Please check this :ref:`section <Cross-References to Locations in the Same Document>`

You can also create links/references in another .rst file, using the file name:

   :ref:`overview-manual/overview-manual-development-environment:yocto project source repositories`

or with a custom link text:

   :ref:`here <overview-manual/overview-manual-development-environment:yocto project source repositories>`

Here is a good tip: You can use the following command to get Sphinx to dump all the references that exist:

   python -msphinx.ext.intersphinx <path to build folder>/html/objects.inv

In this dump, you will find all links and for each link you will get the default "Link Text" that Sphinx will use. If the default link text is not appropriate, you can provide a custom link text in the :ref: link.

external links

We enable the 'extlinks' extension by default, and it's configured with the following ``extlinks``:

   'yocto_home': ('https://yoctoproject.org%s', None),
   'yocto_wiki': ('https://wiki.yoctoproject.org%s', None),
   'yocto_dl': ('https://downloads.yoctoproject.org%s', None),
   'yocto_lists': ('https://lists.yoctoproject.org%s', None),
   'yocto_bugs': ('https://bugzilla.yoctoproject.org%s', None),
   'yocto_ab': ('https://autobuilder.yoctoproject.org%s', None),
   'yocto_docs': ('https://docs.yoctoproject.org%s', None),
   'yocto_git': ('https://git.yoctoproject.org%s', None),
   'oe_home': ('https://www.openembedded.org%s', None),
   'oe_lists': ('https://lists.openembedded.org%s', None),

In the documentation rst files, we can then do:

   Please check this :yocto_wiki:`wiki page </Weekly_Status>`

intersphinx links

We also enable the intersphinx extension so that we can cross reference content from the bitbake manual for example. When building the YP docs, Sphinx will first download the objects.inv file from the Bitbake manual, so that it knows all the existing references/links set in the bitbake documentation. Then references to the bitbake manual can be done like this:

   See the ":ref:`-D <bitbake:bitbake-user-manual/bitbake-user-manual-intro:usage and syntax>`" option

or

   :term:`bitbake:BB_NUMBER_PARSE_THREADS`

Tasks list

Here is an initial tasks list for the migration:

1. Proof of concept
   1. Initial proof of concept patches
   2. Merge initial patch set in master, along with Docbook
   3. Update initial patches with yocto-docs recent changes
2. Convert all YP manuals
   1. Automated conversion with pandoc
      1. adt-manual
      2. brief-yoctoprojectqs
      3. dev-manual
      4. kernel-dev
      5. profile-manual
      6. sdk-manual
      7. toaster-manual
      8. test-manual
      9. bitbake-manual
      10. overview-manual
   2. fix up all manuals for: links, figures, headings, notes, code blocks, global substitutions variables
      1. adt-manual
      2. brief-yoctoprojectqs
      3. overview-manual
      4. dev-manual
      5. kernel-dev
      6. profile-manual
      7. sdk-manual
      8. toaster-manual
      9. test-manual
      10. bitbake-manual
   3. Use current docs CSS file to create proper theme
      1. initial import
      2. tuning/optimization
   4. How to deal with each docbook 'first' page (license and TOC) (In progress RP and ndec looking at that)
   5. Implement glossary
3. Decide how to manage the mega manual. Should we create a single doc (index.rst) file per documentat, or a single one which ultimately would become the mega manual
4. Custom Sphinx theme. The whole output can be themed using CSS and custom theme. We need to implement something that 'looks like' the Yocto Project and create beautiful rendering of our documentation
5. How to publish HTML online (per branch, latest, current, ...)? The whole process must be automated. (In progress, Michael on it)
6. How to create PDF files?
7. What's the impact of Google search and referencing?
8. Do we want to backport to dunfell LTS? (RP: Yes!)
9. RP: Can we create a single page manual for ease of searching (some users need this and I personally use it a lot that way, I'm unlikely alone)
10. RP: Glossary headings don't appear to be externally linkable?