Documentation Sphinx

Yocto Project Documentation: Sphinx migration

The Yocto Project developers are currently investigating whether to move from Docbook to Sphinx (or any similar tools) for the project's documentation. The initial discussion can be found here.

   Sphinx is a tool that makes it easy to create intelligent and beautiful documentation, written by Georg Brandl and licensed under the BSD license. It was originally created for the Python documentation.

See more on Sphinx website. Furthermore Sphinx is designed to be extensible, and we can implement our own custom extensions, as Python modules, which will be executed during the generation of the documentation. This is a rather advance use of Sphinx, but could be very useful in the future.

This wiki page can be used as a working document to assist the discussions and/or the tasks. All discussions should be made on the Yocto Project Docs mailing list.

Sphinx proof of concept

An experimental branch with some initial Sphinx support was published here, which produces the following HTML documentation.

The support is very minimal, and only 3 documents were converted, but the idea was to share an initial look and feel about both the output and the changes to the documentation source code. There is a large amount of work to accomplish if we want to convert the whole documentation!

To use the proof of concept, please follow these instructions:

   pip3 install sphinx
   # to use the non default theme
   pip3 install sphinx_rtd_theme

And then:

   git clone https://git.linaro.org/people/nicolas.dechesne/yocto-docs.git -b sphinx
   cd sphinx/documentation
   make -f Makefile.sphinx html

The resulting HTML index page will be _build/html/index.html.

Note: the branch https://git.linaro.org/people/nicolas.dechesne/yocto-docs.git/log/?h=sphinx2 is a rewrite of the initial branch with additional changes.

Design ideas / principles

Automated conversion

The initial Docbook to Sphinx migration can be done with automated tools such as pandoc. While the conversion is far from being perfect, it produces some clean output markdown text file. After the initial automated conversion developers will need to fix up at least headings, images and links. In addition Sphinx has built in mechanisms (directives) that should be used to replace similar functions implemented in Docbook such as glossary, variables substitutions, notes, ...

To illustrate how Pandoc can be used, the initial conversion for bsp-guide, ref-manual and overview-manual was done with the following commands:

   for i in *.xml; do pandoc -f docbook -t rst $i -o $i.rst; done

Headings

The layout of the Yocto Project manuals is organized as follows

   Book
     Chapter
       Section
         Section
           Section

During the conversion with Pandoc, some of the headings are not converted properly and need to be fixed manually. Let's propose to use the following headings styles:

   Book              => overline ===
     Chapter         => overline ***
       Section       => ====
         Section     => ----
           Section   => ^^^^
             Section => """"

At least with this proposal, we keep the same TOCs with Sphinx and Docbook.

Built in glossary

Sphinx has a glossary directive. From the documentation:

   This directive must contain a reST definition list with terms and definitions. The definitions will then be referencable with the 'term' role.

So anywhere in *any* manual, we can do :term:`VAR` to refer to an item from the glossary, and a link is created.

Global substitutions

The Yocto Project documentation makes heavy use of 'global' variables. In Docbook these 'variables' are stored in the file poky.ent. This Docbook feature is not handled automatically with Pandoc. Sphinx has builtin support for substitutions however they are local to each reST file by default. They can be made global by using rst_prolog:

   rst_prolog
       A string of reStructuredText that will be included at the beginning of every source file that is read.

As such we will keep a documentation configuration file with all global substitutions, similar to poky.ent. Here is a sample code to define variables:

   rst_prolog = """
   .. |DISTRO| replace:: 3.1
   .. |DISTRO_COMPRESSED| replace:: 31
   .. |DISTRO_NAME_NO_CAP| replace:: dunfell
   .. |DISTRO_NAME| replace:: Dunfell
   .. |DISTRO_NAME_NO_CAP_MINUS_ONE| replace:: zeus
   .. |DISTRO_NAME_MINUS_ONE| replace:: Zeus
   .. |YOCTO_DOC_VERSION| replace:: 3.1
   .. |YOCTO_DOC_VERSION_MINUS_ONE| replace:: 3.0.2
   .. |DISTRO_REL_TAG| replace:: yocto-3.1
   .. |METAINTELVERSION| replace:: 12.0
   .. |REL_MONTH_YEAR| replace:: April 2020
   """

and use them:

   This is the manual for version: |DISTRO|, aka |DISTRO_NAME|.

Note directive

Sphinx has a builtin note directive that produces clean Note section in the output file. There are various types of directives such as "attention", "caution", "danger", "error", "hint", "important", "tip", "warning", "admonition" that are supported, and additional directive can be added as Sphinx extension if needed.

Figures

Our documentation has many figures/images. Somehow Pandoc has completely ignored all images during the conversion. It might be worth investigating why, even though it might take more time than manually fixing all documents... Sphinx as a figure directive which is straight forward to use. To include a figure in the body of the documentation:

   .. image:: figures/YP-flow-diagram.png

Tasks list

Here is an initial tasks list for the migration:

1. Proof of concept
   1. Initial proof of concept patches
   2. Merge initial patch set in master, along with Docbook
   3. Update initial patches with yocto-docs recent changes
2. Convert all YP manuals
   1. Automated conversion with pandoc
      1. adt-manual
      2. brief-yoctoprojectqs
      3. dev-manual
      4. kernel-dev
      5. profile-manual
      6. sdk-manual
      7. toaster-manual
      8. test-manual
   2. fix up for links, figures, headings, notes
      1. adt-manual
      2. brief-yoctoprojectqs
      3. bsp-guide
      4. dev-manual
      5. kernel-dev
      6. overview-manual
      7. profile-manual
      8. ref-manual
      9. sdk-manual
      10. toaster-manual
      11. test-manual
   3. What to do for *.css and *.xsl Docbook files
   4. How to deal with each docbook 'first' page (license and TOC)
   5. Implement global substitutions variables
   6. Implement glossary
3. Decide how to manage the mega manual. Should we create a single doc (index.rst) file per documentat, or a single one which ultimately would become the mega manual
4. Custom Sphinx theme. The whole output can be themed using CSS and custom theme. We need to implement something that 'looks like' the Yocto Project and create beautiful rendering of our documentation
5. How to publish HTML online (per branch, latest, current, ...)? The whole process must be automated.
6. How to create PDF files?
7. What's the impact of Google search and referencing?
8. Do we want to backport to dunfell LTS? (RP: Yes!)
9. RP: Can we create a single page manual for ease of searching (some users need this and I personally use it a lot that way, I'm unlikely alone)?
10. RP: Glossary headings don't appear to be externally linkable?