Welcome to Flake8-AAA’s documentation¶

Flake8-AAA¶

A linter for Python tests.

Pytest and unittest styles supported.
Tests are linted against the Arrange Act Assert pattern.
Provides a Flake8 interface to automatically lint test files as part of your Flake8 run.
Provides a command line interface for custom (non-Flake8) usage and debugging.

Installation¶

Install with pip:

$ pip install flake8-aaa

Integration with Flake8¶

Given that you already have Flake8 installed in the same environment, check that Flake8-AAA was installed correctly by asking flake8 for its version signature:

$ flake8 --version
3.6.0 (aaa: 0.4.0, mccabe: 0.6.1, pycodestyle: 2.4.0, pyflakes: 2.0.0) CPython 3.6.7 on Linux

The (aaa: 0.4.0, ... part of that output tells you flake8 found this plugin. Now you can run flake8 as usual against your project and Flake8-AAA will lint your tests via its plugin:

$ flake8

Resources¶

Tested on Pythons 3.5 and 3.6.

Python 2 supported up to v0.4.0: pypi, docs, tag.

Test discovery¶

The flake8-aaa plugin is triggered for files that look to it like test modules - anything that does not look like a test module is skipped.

The following rules are applied by flake8-aaa when discovering tests:

The module’s filename must start with “test_” and have been collected for linting by Flake8.
Every function in the module that has a name that starts with “test” is checked.
Test functions can be class methods.
Test functions that contain only comments, docstrings or pass are skipped.

These rules are aimed to mirror pytest’s default collection strategy as closely as possible.

If you find that flake8-aaa is giving false positives (you have checks that you expected to fail, but they did not), then you should check that the plugin did not ignore or skip those tests which you expected to fail.

Note

flake8-aaa does not check doctests.

Processing¶

For each test found, Flake8-aaa runs the following process:

Check for no-op¶

Skip test if it is considered “no-op” (pass, docstring, etc).

Process Act Block¶

Search the test for an “Action Node” which will indicate the existence of an Act Block. There are four recognised types of Action Node:

marked_act

Action is marked with Marked with # act comment:

do_thing()  # act

pytest_raises

Action is wrapped in pytest.raises context manager:

with pytest.raises(ValueError):
    do_thing()

result_assignment

Simple result = action:

result = do_thing()

unittest_raises

Action is wrapped in unittest’s assertRaises context manager:

with self.assertRaises(ValueError):
    do_thing()

If no Action Node is found then TODO link: “AAA01: no Act block found in test” is raised.

Next the footprint of the Action Node is generated to create the Act Block. Most Act Block footprints will be the same as the Action Node’s footprint, for example:

ACT     with pytest.raises(ValueError):   # <- Action Node
ACT         do_thing()                    # <- Action Node

However, Action Nodes can be wrapped in loops and context managers which must be checked and incorporated when appropriate, for example:

ACT     with mock.patch(thing_doer) as mock_thing_doer:
ACT         result = do_thing()           # <- Action Node

Rules and error codes¶

The rules applied by Flake8-AAA are from the Arrange Act Assert pattern for Python developers.

Note

The rules applied by Flake8-AAA are only a subset of the rules and guidelines of the Arrange Act Assert pattern itself. Please see the published guidelines for the pattern and read these rules in the context of the definition there.

AAA01: no Act block found in test¶

An Act block is usually a line like result = or a check that an exception is raised. Flake8-AAA could not find an Act block in the indicated test function.

Resolution¶

Add an Act block to the test or mark a line that should be considered the action.

Even if the result of a test action is None, assign that result and pin it with a test:

result = action()

assert result is None

If you can not assign a result, then mark the end of the line considered the Act block with # act (case insensitive):

data['new_key'] = 1  # act

Code blocks wrapped in pytest.raises() and unittest.assertRaises() context managers are recognised as Act blocks.

AAA02: multiple Act blocks found in test¶

There must be one and only one Act block in every test but Flake8-AAA found more than one potential Act block. This error is usually triggered when a test contains more than one result = statement or more than one line marked # act. Multiple Act blocks create ambiguity and raise this error code.

Resolution¶

Split the failing test into multiple tests. Where there is complicated or reused set-up code then apply the DRY principle and extract the reused code into one or more fixtures.

AAA03: expected 1 blank line before Act block, found none¶

For tests that have an Arrange block, there must be a blank line between the Arrange and Act blocks, but Flake8-AAA could not find one.

This blank line creates separation between the arrangement and the action and makes the Act block easy to spot.

This rule works best with pycodestyle’s E303 rule enabled because it ensures that there are not multiple blank lines between the blocks.

Resolution¶

Add a blank line before the Act block.

AAA04: expected 1 blank line before Assert block, found none¶

For tests that have an Assert block, there must be a blank line between the Act and Assert blocks, but Flake8-AAA could not find one.

This blank line creates separation between the action and the assertions and makes the Act block easy to spot.

As with rule AAA03, this rule works best with E303 enabled.

Resolution¶

Add a blank line before the Assert block.

AAA99: collision when marking this line as NEW_CODE, was already OLD_CODE¶

This is an error code that is raised when Flake8 tries to mark a single line as occupied by two different types of block. It should never happen. The values for NEW_CODE and OLD_CODE are from the list of Line markers.

Resolution¶

Please open a new issue containing the output for the failing test as generated by the Command line tool.

You could hack around with your test to see if you can get it to work while waiting for someone to reply to your issue. If you’re able to adjust the test to get it to work, that updated test would also be helpful for debugging.

Controlling Flake8-AAA¶

In code¶

Flake8-AAA can be controlled using some special comments in your test code.

Explicitly marking blocks¶

One can set the act block explicitly using the # act comment. This is necessary when there is no assignment possible.

Disabling Flake8-AAA selectively¶

When Flake8-AAA finds the # noqa comment at the end of the line that defines a test function, it will ignore it.

Command line¶

Flake8-AAA has a simple command line interface to assist with development and debugging. Its goal is to show the state of analysed test functions, which lines are considered to be parts of which blocks and any errors that have been found.

Invocation, output and return value¶

With Flake8-AAA installed, it can be called as a Python module:

$ python -m flake8_aaa [test_file]

Where [test_file] is the path to a file to be checked.

The return value of the execution is the number of errors found in the file, for example:

$ python -m flake8_aaa ../some_test.py
------+------------------------------------------------------------------------
 1 DEF|def test():
 2 ARR|    x = 1
 3 ARR|    y = 1
 4 ACT|    result = x + y
           ^ AAA03 expected 1 blank line before Act block, found none
 5 BL |
 6 ASS|    assert result == 2
------+------------------------------------------------------------------------
    1 | ERROR
$ echo "$?"
1

And once the error above is fixed, the return value returns to zero:

$ python -m flake8_aaa ../some_test.py
------+------------------------------------------------------------------------
 1 DEF|def test():
 2 ARR|    x = 1
 3 ARR|    y = 1
 4 BL |
 5 ACT|    result = x + y
 6 BL |
 7 ASS|    assert result == 2
------+------------------------------------------------------------------------
    0 | ERRORS
$ echo "$?"
0

noqa and command line¶

The # noqa comment marker works slightly differently when Flake8-AAA is called on the command line rather than invoked through flake8. When called on the command line, to skip linting a test function, mark the function definition with # noqa on the same line as the def.

For example:

def test_to_be_ignored(  # noqa
    arg_1,
    arg_2,
):
    ...

Line markers¶

Each test found in the passed file is displayed. Each line is annotated with its line number in the file and a marker to show how Flake8-AAA classified that line. Line markers are as follows:

ACT: Line is part of the Act Block.
ARR: Line is part of an Arrange Block.
ASS: Line is part of the Assert Block.
BL: Line is considered a blank line for layout purposes.
DEF: Test function definition.

???: Unprocessed line. Flake8-AAA has not categorised this line.

Release checklist¶

Items to be completed before and after each release.

Pre-release steps¶

Given a new version called x.y.z:

Create a branch for the new release. Usually called something like bump-vx.y.z.
Update __version__ in __about__.py with the new version number 'x.y.z'.
Update CHANGELOG.
- Add a new subtitle below Unreleased after the note about latest documentation, in the format x.y.z_ - yyyy/mm/dd, where yyyy/mm/dd is the reverse formatted date of the day the release is created.
- Update the .. _Unreleased: link at the bottom of the page to compare vx.y.z...HEAD.
- Under the _Unreleased link, create a new link for the release .. _x.y.z: https:/[...]/compare/va.b.c...vx.y.z, where va.b.c is the previous release.
Commit changes and push bump-vx.y.z branch for testing.
Now is a good time to build and check the documentation locally.
When branch bump-vx.y.z is green, then merge it to master.
Update master locally and ensure that you remain on master for the rest of the process.
Test that a build can be shipped to test PyPI with make testpypi. (Every build runs the full clean test suite locally to ensure that nothing has broken before building)
After successful push, check the TestPyPI page.
Then tag the repo with make tag. Add a short message about what the key change is.
Make the new tag public with git push origin --tags.
Build and push to PyPI with make pypi.
After successful push, check the PyPI page.

Post release checks¶

Visit the CHANGELOG and ensure that the new release’s comparison link works with the new tag.
Check the RTD builds to ensure that the latest documentation version has been picked up and that the stable docs are pointed at it.

A new docs release will not have been created for the new tag as per this issue. Click “Build Version:” on the builds page for the new tag to be picked up.