Documentation Sphinx

Yocto Project Documentation: Sphinx migration

The Yocto Project developers are currently investigating whether to move from Docbook to Sphinx (or any similar tools) for the project's documentation. The initial discussion can be found here.

   Sphinx is a tool that makes it easy to create intelligent and beautiful documentation, written by Georg Brandl and licensed under the BSD license. It was originally created for the Python documentation.

See more on Sphinx website. Furthermore Sphinx is designed to be extensible, and we can implement our own custom extensions, as Python modules, which will be executed during the generation of the documentation. This is a rather advance use of Sphinx, but could be very useful in the future.

This wiki page can be used as a working document to assist the discussions and/or the tasks. All discussions should be made on the Yocto Project Docs mailing list.

Sphinx migration for Yocto Project Documentation

The Sphinx migration is planned for 3.2 release, and the development is done in this branch, which produces the following HTML documentation. Until the branch gets merged , patches should be sent against this branch.

A lot of work was already done to convert all the docbook files, but there is still some significant of work left to be done.

To use the proof of concept, please follow these instructions:

   pip3 install sphinx
   # to use the non default theme
   pip3 install sphinx_rtd_theme

And then:

   git clone https://git.yoctoproject.org/git/yocto-docs -b sphinx
   cd yocto-docs/documentation
   make -f Makefile.sphinx html

The resulting HTML index page will be _build/html/index.html.

Sphinx proof of concept for Bitbake Documentation

A similar branch exists for the Bitbake documentation as well. The branch is available on here, and produces the following HTML documentation.

To build the Bitbake documentation with Sphinx:

   git clone https://git.openembedded.org/bitbake -b sphinx
   cd bitbake/doc
   make -f Makefile.sphinx html

Design ideas / principles

Automated conversion

The initial Docbook to Sphinx migration can be done with automated tools such as pandoc. While the conversion is far from being perfect, it produces some clean output markdown text file. After the initial automated conversion developers will need to fix up at least headings, images and links. In addition Sphinx has built in mechanisms (directives) that should be used to replace similar functions implemented in Docbook such as glossary, variables substitutions, notes, ...

To illustrate how Pandoc can be used, the initial conversion for bsp-guide, ref-manual and overview-manual was done with the following commands:

   for i in *.xml; do pandoc -f docbook -t rst $i -o $i.rst; done

Headings

The layout of the Yocto Project manuals is organized as follows

   Book
     Chapter
       Section
         Section
           Section

During the conversion with Pandoc, some of the headings are not converted properly and need to be fixed manually. Let's propose to use the following headings styles:

   Book              => overline ===
     Chapter         => overline ***
       Section       => ====
         Section     => ----
           Section   => ^^^^
             Section => """"

At least with this proposal, we keep the same TOCs with Sphinx and Docbook.

Built in glossary

Sphinx has a glossary directive. From the documentation:

   This directive must contain a reST definition list with terms and definitions. The definitions will then be referencable with the 'term' role.

So anywhere in *any* manual, we can do :term:`VAR` to refer to an item from the glossary, and a link is created.

Global substitutions

The Yocto Project documentation makes heavy use of 'global' variables. In Docbook these 'variables' are stored in the file poky.ent. This Docbook feature is not handled automatically with Pandoc. Sphinx has builtin support for substitutions however they are local to each reST file by default. They can be made global by using rst_prolog:

   rst_prolog
       A string of reStructuredText that will be included at the beginning of every source file that is read.

As such we will keep a documentation configuration file with all global substitutions, similar to poky.ent. Here is a sample code to define variables:

   rst_prolog = """
   .. |DISTRO| replace:: 3.1
   .. |DISTRO_COMPRESSED| replace:: 31
   .. |DISTRO_NAME_NO_CAP| replace:: dunfell
   .. |DISTRO_NAME| replace:: Dunfell
   .. |DISTRO_NAME_NO_CAP_MINUS_ONE| replace:: zeus
   .. |DISTRO_NAME_MINUS_ONE| replace:: Zeus
   .. |YOCTO_DOC_VERSION| replace:: 3.1
   .. |YOCTO_DOC_VERSION_MINUS_ONE| replace:: 3.0.2
   .. |DISTRO_REL_TAG| replace:: yocto-3.1
   .. |METAINTELVERSION| replace:: 12.0
   .. |REL_MONTH_YEAR| replace:: April 2020
   """

and use them:

   This is the manual for version: |DISTRO|, aka |DISTRO_NAME|.

Note directive

Sphinx has a builtin note directive that produces clean Note section in the output file. There are various types of directives such as "attention", "caution", "danger", "error", "hint", "important", "tip", "warning", "admonition" that are supported, and additional directive can be added as Sphinx extension if needed.

Figures

Our documentation has many figures/images. Somehow Pandoc has completely ignored all images during the conversion. It might be worth investigating why, even though it might take more time than manually fixing all documents... Sphinx as a figure directive which is straight forward to use. To include a figure in the body of the documentation:

   .. image:: figures/YP-flow-diagram.png

Links and References

From: https://sublime-and-sphinx-guide.readthedocs.io/en/latest/references.html

You can include links to other locations in the same document, to locations in other documents and to external websites.

You can link from text to a heading in any other part of the document by using the :ref: command with the heading text as the parameter. For example, this text in another part of this document would link to this section:

   :ref:`Cross-References to Locations in the Same Document`

For that to work, we need sphinx.ext.autosectionlabel to be enabled in conf.py.

We can use Custom link text or create custom anchor instead of using the section text.

TODO: study https://sublime-and-sphinx-guide.readthedocs.io/en/latest/references.html#use-the-external-links-extension, looks like something we could use..

Tasks list

Here is an initial tasks list for the migration:

1. Proof of concept
   1. Initial proof of concept patches
   2. Merge initial patch set in master, along with Docbook
   3. Update initial patches with yocto-docs recent changes
2. Convert all YP manuals
   1. Automated conversion with pandoc
      1. adt-manual
      2. brief-yoctoprojectqs
      3. dev-manual
      4. kernel-dev
      5. profile-manual
      6. sdk-manual
      7. toaster-manual
      8. test-manual
      9. bitbake-manual
   2. fix up all manuals for:
      1. links
      2. figures
      3. headings
      4. notes
      5. code blocks
      6. global substitutions variables
   3. Use current docs CSS file to create proper theme
      1. initial import
      2. tuning/optimization
   4. How to deal with each docbook 'first' page (license and TOC) (In progress RP and ndec looking at that)
   5. Implement glossary
3. Decide how to manage the mega manual. Should we create a single doc (index.rst) file per documentat, or a single one which ultimately would become the mega manual
4. Custom Sphinx theme. The whole output can be themed using CSS and custom theme. We need to implement something that 'looks like' the Yocto Project and create beautiful rendering of our documentation
5. How to publish HTML online (per branch, latest, current, ...)? The whole process must be automated. (In progress, Michael on it)
6. How to create PDF files?
7. What's the impact of Google search and referencing?
8. Do we want to backport to dunfell LTS? (RP: Yes!)
9. RP: Can we create a single page manual for ease of searching (some users need this and I personally use it a lot that way, I'm unlikely alone)
10. RP: Glossary headings don't appear to be externally linkable?