How to make your tests AWESOME

Some people may argue that automated tests are not worth the effort. But here is a kicker: time spent on creating and maintaining tests in many cases is completely justified by time saved on manual testing and re-testing. Another toll of not having automated tests is the cost of bugs (in time and money) caught late in the process, sometimes even in production. Of course, automated tests are not 100% a guarantee of bug-free code; however, properly created test suite significantly reduces the probability of bugs slipping into production.

So, how do we craft a superior test suite?

Test the real thing

Test must validate reality against expected values; thus, tests without asserts are not tests. If you are testing for a certain value, assert the value against the expected.

import pytest
import src.division as dv


def test_division():
    # Do assert for expected value
    assert 5 == dv.divide(10, 2)

Once exception to this rule is when we test for an expected exception to be raised. If using pytest, the syntax is such that it does not require assert to validate that a certain exception is raised.

Tip 💡: Take a look at this post to learn more about pytest in 5 minutes!

def test_division__divided_by_0__raises_exception():
    # pytest syntax to test for expected Exception
    with pytest.raises(ZeroDivisionError):
        dv.divide(10, 0)

Make the tests repeatable

No matter how many times we run a test, it should produce the same results. Imagine a test that fails half of the time. How easy would it be to work with such a test? What does it tell us about the validity of our application? Can we rely on it? A test that is not repeatable is like a friend that fails you every once in a while. You can’t trust the friend, nor can you trust a non-repeatable test.

Make the tests independent

Order of tests should not matter. Test frameworks may not guarantee the order of tests’ execution. Each test should be self-contained and run independently. Do all necessary setup before a test runs and clean up artifacts after. Multiple testing frameworks support setup-run-cleanup in different ways; for example, pytest has fixtures that are super helpful in this matter.

import pytest
import src.user as usr
import src.user_repository as u_repo


@pytest.fixture
def tmp_user():
    user = usr.User()
    # Add user to the database before test runs.
    repo = u_repo.UserRepository()
    repo.add_user(user)
    yield user
    # Delete user from the database after test completes,
    # either fails or passes.
    repo.delete_user(user)


def test_user_authentication(tmp_user: usr.User):
    tmp_user.authenticate()
    assert tmp_user.is_authenticcated

Make the tests focused

Decide what exactly you are testing and stay focused on that. Do not assert irrelevant conditions just in case; other conditions can be tested by their own dedicated tests. Test “the thing” under test, not the whole universe. Otherwise, with a little bug introduced in an application, we will have far too many tests failing, that would make it hard to narrow down and fix a problem.

Some people recommend going as extreme as a single assert per test, but this is arbitrary. If we have multiple data fields changed as a result of a state transition, it is not possible to verify it with just a single assert, so limiting yourself this way may not be practical.

Make tests short

Keep each test short, make it fit into a “single screen”. Imagine yourself trying to fix a failing test that is a couple of hundred lines of code, not a nice place to be in. If a test needs a lot of preparation, move helping code that sets up and dismantles the environment into helping functions, classes, or use fixtures. And if you stay focused on a single thing under test, the test code would be short and comprehensible. If not, then most likely the test does too much; try to split it.

Make the tests fast

If tests are too slow, developers will be hesitant to run them often, losing the advantage of verifying code in small increments. Slow tests mean long feedback time, whether it’s CI/CD or just running tests locally. Some IDEs, like Visual Studio, have a Live Unit Testing feature, where tests run automatically while code is being modified, providing immediate feedback.

Of course, the time the entire test suite runs greatly depends on the size of the test suite itself, which certainly depends on the application size being tested. Some tests are slow by nature, e.g. tests that span across multiple external dependencies, whether it’s a database or microservices. That leads to the next advice.

Organize the tests

Follow the test structure

A test usually has three phases: Given, When, Then, and sometimes Cleanup. You can literally imprint the phases in test code with comments. Believe it or not, but this little thing improves readability significantly.

1. GIVEN creates a state of an application before running a test. In other words, it sets up a stage for the test to run.
2. WHEN applies predetermined action or raises an event that triggers state transition under test.
3. Then, it validates whether the new state of the application corresponds to the expected one. Technically, we assert the state and verify that the expected events indeed occurred.

import src.user as usr
import src.user_repository as u_repo


def test_user_authentication():
    # GIVEN
    user = usr.User()
    repo = u_repo.UserRepository()
    repo.add_user(user)
    try:
        # WHEN:
        user.authenticate()
        # THEN
        assert user.is_authenticcated
    finally:
        # CLEANUP
        repo.delete_user(user)

We may need an optional cleanup step to erase all artifacts being created during tests, for example, records created in a database, temporary files, etc. The cleanup phase is mainly applicable to integration and system tests.

Pro Tips

• If using pytest, fixtures can greatly help with managing creation and cleanup of resources. You can create your own fixtures or take advantage of pytest built-in fixtures, e.g. tmp_path.
• If setup or cleanup fails for any reason, it should fail exactly that test; other tests should not fail.

Treat the tests as application code

Keep test code up to the same standards as application code. Even though end users do not run test code directly, they still benefit from it by the increased quality of the application. Developers read test code too, especially when tests fail after code being changed. Test code should be easy to comprehend and reason about. When a test fails after making a change, it should be fairly easy to fix the code or adjust the test so it reflects the new changes in the code.

When application code evolves, tests should follow. Tests also require refactoring to stay up-to-date with the application code.

Make the test automated

Having a test suite is a great advantage. However, completely relying on people to run tests before committing code is a bit chancy. You may end up pulling code and finding out that tests fail. Is it a sign of broken code or broken tests?

Run test suite automatically as part of CI/CD, best on every pull request or before code is merged into the main branch. Block code from being merged if tests fail. Test automation reduces the probability of bugs slipping into production. Production bugs cost much more to deal with than if they were caught earlier.

Make use of tools

Use tools that allow you to collect metrics like code coverage and report them back to developers. Show how new changes in the code improve or reduce the percentage of code being auto-tested and validated.

Some tools, for example, Visual Studio Live Unit Testing feature, can show paths in the code that are not hit by tests, hence not tested at all. Visualizations like this help to identify important spots that are never verified and write tests to cover them.

Summary

Automated test suite is no doubt a great tool, but it needs to be crafted and used properly. Having poor tests is the same as having no tests, or often even worse.

Consider practices above as empirically proven recommendations that increase the usefulness of tests, hence the quality of software. I personally apply them every day, and I do see real benefits.

If you feel I missed something or you have your own set of practices, please let me know in the comments down below.