Project:Python/Sphinx race conditions

Packages using Sphinx to generate documentation, e.g. via Makefiles are frequently subject to race conditions.

Symptoms

CODE Example error message

Running Sphinx v1.6.3
making output directory...
loading pickled environment... not yet created

Exception occurred:
  File "/usr/lib64/python3.5/os.py", line 241, in makedirs
    mkdir(name, mode)
FileExistsError: [Errno 17] File exists: '/var/tmp/portage/dev-python/psycopg-2.7.3.2/work/psycopg2-2.7.3.2/doc/src/_build/doctrees'
The full traceback has been saved in /var/tmp/portage/dev-python/psycopg-2.7.3.2/temp/sphinx-err-toqwpu8i.log, if you want to report the issue to the developers.
Please also report this if it was a user error, so that a better error message can be provided next time.
A bug report can be filed in the tracker at <https://github.com/sphinx-doc/sphinx/issues>. Thanks!
loading intersphinx inventory from http://docs.python.org/3/objects.inv...
make: *** [Makefile:48: text] Error 1

This happens because multiple Sphinx instances are running in parallel, and they happen to be attempting to create the same directory.

Problem

Sphinx is not designed to be parallel-safe. This has been reported upstream as issue 2946, however no fix is expected soon. If you run two Sphinx instances using the same doctree in parallel, they will race for it, and either one will overwrite the other's one work, or they will hit a failing mkdir or like.

Workarounds

Use -j1

The easy workaround is… not to run Sphinx in parallel. For example, when you're calling make to build docs, pass -j1:

FILE example-1.ebuildForcing serial doc-build

python_compile_all() {
    emake -j1 -C docs
}

Force separate doctrees

This is the hard solution. Each sphinx-build invocation is passed a -d parameter that specifies the doctree directory. You need to alter the Makefiles to make sure that every invocation that can happen in parallel uses a different directory.

Good news is that if you do that, you have a good chance of getting it merged upstream.