Documentation Sphinx

Yocto Project Documentation: Sphinx migration

The Yocto Project developers are currently investigating whether to move from Docbook to Sphinx (or any similar tools) for the project's documentation. The initial discussion can be found here.

   Sphinx is a tool that makes it easy to create intelligent and beautiful documentation, written by Georg Brandl and licensed under the BSD license. It was originally created for the Python documentation.

See more on Sphinx website. Furthermore Sphinx is designed to be extensible, and we can implement our own custom extensions, as Python modules, which will be executed during the generation of the documentation. This is a rather advance use of Sphinx, but could be very useful in the future.

This wiki page can be used as a working document to assist the discussions and/or the tasks.

Sphinx proof of concept

An experimental branch with some initial Sphinx support was published here, which produces the following HTML documentation.

The support is very minimal, and only 3 documents were converted, but the idea was to share an initial look and feel about both the output and the changes to the documentation source code. There is a large amount of work to accomplish if we want to convert the whole documentation!

Design ideas / principles

Automated conversion

The initial Docbook to Sphinx migration can be done with automated tools such as pandoc. While the conversion is far from being perfect, it produces some clean output markdown text file. After the initial automated conversion developers will need to fix up at least headings, images and links. In addition Sphinx has built in mechanisms (directives) that should be used to replace similar functions implemented in Docbook such as glossary, variables substitutions, notes, ...

To illustrate how Pandoc can be used, the initial conversion for bsp-guide, ref-manual and overview-manual was done with the following commands:

   for i in *.xml; do pandoc -f docbook -t rst $i -o $i.rst; done

Headings

The layout of the Yocto Project manuals is organized as follows

   Book
     Chapter
       Section
         Section
           Section

During the conversion with Pandoc, some of the headings are not converted properly and need to be fixed manually. Let's propose to use the following headings styles:

   Book              => overline ===
     Chapter         => overline ***
       Section       => ====
         Section     => ----
           Section   => ^^^^
             Section => """"

At least with this proposal, we keep the same TOCs with Sphinx and Docbook.

Built in glossary

Sphinx has a glossary directive. From the documentation:

   This directive must contain a reST definition list with terms and definitions. The definitions will then be referencable with the 'term' role.

So anywhere in *any* manual, we can do :term:`VAR` to refer to an item from the glossary, and a link is created.

Global substitutions

The Yocto Project documentation makes heavy use of 'global' variables. In Docbook these 'variables' are stored in the file poky.ent. This Docbook feature is not handled automatically with Pandoc. Sphinx has builtin support for substitutions however they are local to each reST file by default. They can be made global by using rst_prolog:

   rst_prolog
       A string of reStructuredText that will be included at the beginning of every source file that is read.

As such we will keep a documentation configuration file with all global substitutions, similar to poky.ent. Here is a sample code to define variables:

   rst_prolog = """
   .. |DISTRO| replace:: 3.1
   .. |DISTRO_COMPRESSED| replace:: 31
   .. |DISTRO_NAME_NO_CAP| replace:: dunfell
   .. |DISTRO_NAME| replace:: Dunfell
   .. |DISTRO_NAME_NO_CAP_MINUS_ONE| replace:: zeus
   .. |DISTRO_NAME_MINUS_ONE| replace:: Zeus
   .. |YOCTO_DOC_VERSION| replace:: 3.1
   .. |YOCTO_DOC_VERSION_MINUS_ONE| replace:: 3.0.2
   .. |DISTRO_REL_TAG| replace:: yocto-3.1
   .. |METAINTELVERSION| replace:: 12.0
   .. |REL_MONTH_YEAR| replace:: April 2020
   """

and use them:

   This is the manual for version: |DISTRO|, aka |DISTRO_NAME|.