How to make tests AWESOME

Some people may argue that automated tests are not worth of effort. But here is a kicker: time spent on creating and maintaining tests in many cases is completely justified by time saved on manual testing and re-testing. Another toll of not having automated test is cost of bugs (in time and money) caught late in the process, sometimes even in production. Of course, automated tests are not 100% guarantee of bug-free code, however properly create test suite significantly reduces probability of bug slipping into production.

So, how do we craft superior test suite?

Test real thing

Test must validate reality against expected values, thus tests without asserts are not tests. If you are testing for certain value, assert the value against expected.

import pytest
import src.division as dv


def test_division():
    # Do assert for expected value
    assert 5 == dv.divide(10, 2)

Once exception to this rule is when we test for expected exception to be raised. If using pytest,  the syntax is such that it does not require assert to validate that certain exception is raised.

def test_division__divided_by_0__raises_exception():
    # pytest syntax to test for expected Exception
    with pytest.raises(ZeroDivisionError):
        dv.divide(10, 0)

Make tests repeatable

No matter how many times we run a test, it should produce same results. Imagine a test that fails half of the times. How easy would it be to work with such a test? What does it tell us about validity of our application? Can we rely on it? Test that is not repeatable is like a friend that fails you every once in a while. You can’t trust the friend, neither you can trust non-repeatable test.

Make tests independent

Order of tests should not matter. Test frameworks may not guarantee order of tests execution. Each test should be self-contained and run independently. Do all necessary setup before a test runs and clean up artifacts after. Multiple testing frameworks support setup-run-cleanup in different ways, for example pytest has fixtures that are super helpful in this matter.

import pytest
import src.user as usr
import src.user_repository as u_repo


@pytest.fixture
def tmp_user():
    user = usr.User()
    # Add user to the database before test runs.
    repo = u_repo.UserRepository()
    repo.add_user(user)
    yield user
    # Delete user from the database after test completes,
    # either fails or passes.
    repo.delete_user(user)


def test_user_authentication(tmp_user: usr.User):
    tmp_user.authenticate()
    assert tmp_user.is_authenticcated

Make tests focused

Decide what exactly you are testing and stay focused on that. Do not assert irrelevant conditions just in case, other conditions can be tested by their own dedicated tests. Test “the thing” under test, not the whole universe. Otherwise with a little bug introduced in an application, we will have far too many tests failing, that would make it hard to narrow down and fix a problem.

Some people recommend to go as extreme as a single assert per test, but this is arbitrary. If we have multiple data fields changed as result of a state transition it is not possible to verify it with just a single assert, so limiting yourself this way may not be practical.

Make tests short

Keep each test short, make it fit into a “single screen”. Imagine yourself trying to fix a failing test that is couple hundreds of lines of code, not a nice place to be in. If a test needs a lot of preparation, move helping code that setups and dismantles environment into a helping functions, classes or use fixtures. And if you stay focused on a single thing under test, the test code would be short and comprehensible. If not, then most likely the test does too much, try to split it.

Make tests fast

If tests are too slow, developers will be hesitant to run them often, loosing advantage of verifying code in small increments. Slow tests mean long feedback time, whether it’s CI/CD or just running tests locally. Some IDEs, like Visual Studio have Live Unit Testing feature, where tests run automatically while code is being modified providing immediate feedback.

Of course, the time entire test suite runs greatly depends on the size of the test suite itself, which certainly depends on an application size being tested. Some tests are slow by nature, e.g. tests that span across multiple external dependencies whether it’s a database of microservices. That leads to next advice.

Organize tests

Follow test structure

A test usually have three phases: Given, When, Then, and sometimes Cleanup. You can literally imprint the phases in test code with comments. Believe it or not, but this little thing improves readability significantly.

1. GIVEN creates a state of an application before running a test. In other words, it setups a stage for the test to run.
2. WHEN applies predetermined action or raises an event that triggers state transition under test.
3. THEN validates whether new state of the application corresponds to expected. Technically we assert the state and verify that expected events indeed occurred.

import src.user as usr
import src.user_repository as u_repo


def test_user_authentication():
    # GIVEN
    user = usr.User()
    repo = u_repo.UserRepository()
    repo.add_user(user)
    try:
        # WHEN:
        user.authenticate()
        # THEN
        assert user.is_authenticcated
    finally:
        # CLEANUP
        repo.delete_user(user)

We may need optional cleanup step to erase all artifacts being created during tests, for example records created in a database, temporary and etc. Cleanup phase is mainly applicable to integration and system tests.

Pro Tips

• If using pytest, fixtures can greatly help with managing creation and cleanup of resources. You can create your own fixtures or take advantage of pytest built in fixtures e.g. tmp_path.
• If setup or cleanup fails for any reason it should fail exactly that test, other tests should not fail.

Treat tests as application code

Keep test code up to the same standards as application code. Even though end users do not run test code directly they still benefit from it by increased quality of the application. Developers read test code too, especially when tests fail after code being changed. Test code should be easy to comprehend and reason about. When test fails after making a change it should be fairly easy to fix code or adjust the test so it reflects new changes in the code.

When application code evolves tests should follow. Tests also require refactoring to stay up to date with the application code.

Make test automated

Having test suite is great advantage. However, completely relying on people to run tests before committing code is a bit chancy. You may end up pulling code and finding out that tests fail. Is it a sign of broken code or broken tests?

Run test suite automatically as part of CI/CD, best on every pull request or before code is merged into main branch. Block code from being merged if tests fail. Tests automation reduces probability of bugs slipping into production. Production bugs costs much more to deal with than if were caught earlier.

Make use of tools

Use tools that allow to collect metrics like code coverage and report them back to developers. Show how new changes in the code improve or reduce percentage of code being auto-tested and validated.

Some tools, for example Visual Studio Live Unit Testing feature can show paths in the code that are not hit by tests, hence not tested at all. Visualizations like this help to identify important spots that are never verified and write tests to cover it.

Summary

Automated tests suite is no doubt a great tool, but it needs to be crafted and used properly. Having poor tests is the same as having no tests, or often even worse.

Consider practices above as empirically proven recommendations that increase usefulness of tests, hence the quality of software. I personally apply them every day and I do see real benefits.

If you feel I missed something or you have your own set of practices, please let me know in comments down below.