Project:Python/Tests
This page lists tips and good practices for handling test suites in Python packages.
Adding test support to Python packages
Determine how to run tests
Every Python package should enable tests if they are provided upstream. In order to enable tests, you need to determine the correct dependencies and test runner. In order to determine those, bear in mind the following advices:
- Consult the associated documentation. Upstreams sometimes mention how to run tests. However, note that the documentation can be sometimes outdated if upstream changed the test system.
- Look at common test program/CI configuration files — tox.ini, .travis.yml. Note that they usually include a lot more than strictly necessary for Gentoo packaging needs; however, they can be useful in figuring out what is actually being done upstream.
- Look at setup.py. Setuptools packages often define test dependencies there (e.g. via tests_require). Some packages may also support running tests via the test command which is usually preferable over calling other test runners directly.
- Look at imports in test files. The table in test suites section lists common packages providing test suites, and appropriate runners for them.
Do not forget to add appropriate test dependencies. If tests require additional packages, add them.
Always run verbosely
Please always add -v (single, not more) to the test runner if possible. This ensures that the test runner outputs all test names verbosely, and therefore makes it easier to determine where stray output is coming from.
...but avoid unnecessary test dependencies
Upstream frequently run tests via common tools such as py.test or nosetests, at the same time using only the standard unittest module. In that case, it is reasonable to try to use the built-in test runner and avoid any dependencies at all.
"${EPYTHON}" -m unittest discover -v
If upstream uses non-standard test file names (that are caught e.g. by nosetests only), you can override the default pattern using the -p option.
"${EPYTHON}" -m unittest discover -v -p '*_tests.py'
Skip coverage testing and PEP-8 testing
Some of the packages us special test runners or plugins to enable extra features such as:
- code analysis (sic!),
- coverage testing,
- PEP-8 testing, etc.
Those are generally only useful upstream. They usually provide only informatory output and statistics, or (in case of PEP-8) complain about pure upstream issues that do not affect users.
If the packages do any of that, please do not run those tests and avoid the unnecessary dependencies caused by them.
Do not ever use tox!
Tox is just a wrapper to run tests for multiple Python implementations. We have our own way of running tests for multiple Python implementations. DO NOT USE TOX!
Test with network-sandbox
Use FEATURES="network-sandbox" to ensure that tests (and the setup script) work correctly without Internet connection. Note that some packages will attempt to fetch missing dependencies over the network — you either need to add those dependencies to ensure that they are installed, or otherwise remove them from setup.py.
If some tests require Internet connection, those tests should be disabled to avoid possible data transfer fees on our users. However, please mark this with an explicit comment so that developers willing to run all the tests can re-enable them easily.
Deal with failing tests
If some tests fail, always investigate the issue. You should work with upstream on ensuring that all tests pass correctly. You may disable tests that are clearly broken but ensure that they are re-enabled once the issue is resolved. If you restrict any tests, leave a clear comment why you're doing that, so that others know how to proceed.
Useful tips for solving test failures:
- run tests manually,
- try a newer version (possibly upstream git),
- try using different versions of dependencies (upstreams frequently declare specific versions in requirements or tox.ini).
setuptools (and distutils) test commands
The setuptools build system (dev-python/setuptools) provides some support for running different test systems via the test command. Furthermore, some packages define their own variants of the test command.
python_test() {
esetup.py test
}
Note that many packages do not use those test commands — and in case of setuptools the test command does not fail and is a no-op. Make sure that any tests are actually run.
Test suites
The following table lists the packages installing common test suites, their module names and packages installing them.
| Package name | Python package | Test runner executable | Notes |
|---|---|---|---|
| dev-python/logilab-common | logilab.common.test | pytest -v | - |
| dev-python/nose | nose | nosetests -v | - |
| dev-python/pytest | pytest or py | py.test -v | - |
| dev-python/twisted (was: dev-python/twisted-core) | twisted.trial | trial | - |
| (built-in) | unittest | "${EPYTHON}" -m unittest discover -v | since Python 2.7/3.2 |
| dev-python/unittest2 | unittest2 | unit2.py discover -v | see below: unittest and unittest2 |
unittest and unittest2
unittest is Python's built-in test suite module. This module is constantly developed and new features are added in every version of Python. Especially, Python 2.7/3.2 have seen many improvements that a number of packages rely on.
unittest2 aims to be a backport of those new features to the older Python versions. In this common case, the affected packages use either unittest2 or unittest conditionally, similarly to the following snippet:
try:
import unittest2 as unittest
except ImportError:
import unittest
In this case, the developer is responsible for assuring which Python versions require unittest2, and which work fine with plain unittest. Often, requiring dev-python/unittest2 as an unconditional dependency is unnecessary.
In Gentoo, we consider two cases:
- If the tests work fine with plain unittest in all supported implementations, no dependency is used,
- If the tests require unittest2, dependency on dev-python/unittest2 should be added. It can be specifically versioned and made conditional to some of the Python implementations if only some of the new features are needed.
pytest plugins
pytest has a number of plugins, some of them automatically affecting tests, some of them used only when explicitly requested.
pytest-cov
Enables coverage testing. Used only when explicitly enabled (--cov). Do not use it, skip it from dependencies.
pytest-relaxed
Enables more relaxed rules for finding tests. Enables itself automatically and breaks stuff. Since v1.1.4, we patch it to avoid autoloading so that it doesn't break all pytest-related tests. If your package depends on it, you have to add -p pytest_relaxed.plugin to your pytest invocation.
pytest-xdist
Enables distributed (parallel) testing. Used only when explicitly enabled (-n). At the moment, we don't do parallel testing consistently, so skip it.