Recently I had to add
python3.8 for our Python linter (the strictest one in existence):
wemake-python-styleguide. And during this straight-forward (at first look) task, I have found several problems with test coverage that were not solved in Python community at all.
Let's dig into it.
The thing about this update was that
python3.8 introduced a new API.
Previously we had
visit_NameConstant methods (we don't really care about what they do, only names are important), but now we have to use
visit_Constant method instead.
Well, we will have to write some compatibility layer in our app to support both new
My first attempt was to create a
route_visit function to do the correct thing for all releases. To make it work I also had to define
PY38 constant like so:
import sys PY38 = sys.version_info >= (3, 8) if PY38: route_visit = ... # new logic: 3.8+ else: route_visit = ... # old logic: 3.6, 3.7
And it worked pretty well. I was able to run my tests successfully. The only thing broken was coverage. We use
pytest-cov to measure coverage in our apps, we also enforce
--cov-fail-under=100 policies. Which enforce us to cover all our code and all branches inside it.
Here's the problem with my solution: I was either covering
if PY38: branch on
python3.8 build or
else: branch on other releases. I was never covering 100% of my program. Because it is literally impossible.
Open-source libraries usually face this problem. They are required to work with different python versions, 3rd party API changes, backward compatibility, etc. Here are some examples that you probably have already seen somewhere:
try: import django HAS_DJANGO = True except ImportError: HAS_DJANGO = False
Or this was a popular hack during
try: range_ = xrange # python2 except NameError: range_ = range # python3
With all these examples in mind, one can be sure that 100% of coverage is not possible.
The common scenario to still achieve a feeling of 100% coverage for these cases was:
# pragma: no covermagic comment to exclude a single line or a whole block from coverage
- Or writing every compatibility related check in a special
compat.pythat were later ommited from coverage
Here's how the first way looks like:
try: # pragma: no cover import django HAS_DJANGO = True except ImportError: # pragma: no cover HAS_DJANGO = False
Let's be honest: these solutions are dirty hacks. But, they do work. And I personally used both of them countless times in my life.
Here's the interesting thing: aren't we supposed to test these complex integrational parts with the most precision and observability? Because that's where our application breaks the most: integration parts. And currently, we are just ignoring them from coverage and pretending that this problem does not exist.
And for this reason, this time I felt like I am not going to simply exclude my compatibility logic. I got an idea for a new project.
My idea was that
# pragma comments can have more information inside them. Not just
no cover, but
no cover when?. That's how
coverage-conditional-plugin was born. Let's use it and see how it works!
First, we would need to install it:
pip install coverage-conditional-plugin # pip works, but I prefer poetry
And then we would have to configure
coverage and the plugin itself:
[coverage:run] # Here we specify plugins for coverage to be used: plugins = coverage_conditional_plugin [coverage:coverage_conditional_plugin] # Here we specify our pragma rules: rules = # we are going to define them later.
rules key. It is the most important thing here. The rule (in this context) is some predicate that tells: should we include lines behind this specific
pragma in our coverage or not. Here are some examples:
[coverage:coverage_conditional_plugin] # Here we specify our pragma rules: rules = "sys_version_info >= (3, 8)": py-gte-38 "sys_version_info < (3, 8)": py-lt-38 "is_installed('django')": has-django "not is_installed('django')": has-no-django
It is pretty clear what we are doing here: we are defining pairs of predicates to include this code if some condition is true and another code in the opposite case.
Here's how our previous examples would look like with these magic comments:
import sys PY38 = sys.version_info >= (3, 8) if PY38: # pragma: py-lt-38 route_visit = ... # new logic: 3.8+ else: # pragma: py-gte-38 route_visit = ... # old logic: 3.6, 3.7
What does it say? If we are running on
if PY38: part. But, cover
else: case. Because it is going to be executed and we know it. And we need to know how good did we cover it. On the other hand, if we are running on
py-gte-38 then cover
if PY38: case and leave
And we can test that everything works correctly. Let's add some nonsense into our
PY38 branch to see if it is going to be covered by
As we can see: green signs show which lines were fully covered, the yellow line indicates that branch coverage was not full, and the red line indicates that the line was not covered at all. And here's an example of grey or ignored lines under the opposed condition:
Here you can find the full real-life source code for this sample.
And here's one more example with
django to show you how external packages can be handled:
try: # pragma: has-no-django import django HAS_DJANGO = True except ImportError: # pragma: has-django HAS_DJANGO = False
We use the same logic here. Do we have
django installed during tests (we have a little helper function
is_installed to tell us)? If so, cover
try:. If not, cover
except ImportError: branch. But always cover something.
I hope you got the idea. Conditional coverage allows you to add or ignore lines based on predicates and collecting required bits of coverage from every run, not just ignoring complex conditions and keeping our eyes wide shut. Remember, that the code we need to cover the most!
By the way, we have all kinds of helpers to query your environment:
os_environfor env variables
platform_versionfor OS-based values
pkg_versionthat returns package version information (as its name says)
- and many others!
This little plugin is going to be really helpful for library authors that have to deal with compatibility and unfixed environments. And
coverage-conditional-plugin will surely cover their backs! Give it a star on Github if you like this idea. And read the project docs to learn more.