Testing with pytest

A test is a small program that runs your code with known inputs and checks the result. Once you have a set of them, you can change your code with confidence: if something that used to work breaks, a test tells you within seconds, long before a user finds out. Tests also document behaviour, because each one is a concrete example of what your code is meant to do.

This unit starts with the standard library's unittest and doctest, then spends most of its time on pytest, the third party framework most Python projects use. You will write tests, share setup code through fixtures, run one test against many inputs, and replace slow or external parts of a program with mocks.

You will be able to:

• Explain what automated tests are for and how unit, integration and end to end tests differ
• Write and run tests with unittest and pytest, and read a pytest failure report
• Share setup and cleanup code with fixtures, including tmp_path, monkeypatch and capsys
• Run one test over many inputs with parametrize, and use the skip, skipif and xfail markers correctly
• Check exceptions and floating point results, and replace dependencies with unittest.mock
• Organise a test suite, select which tests run, and measure coverage

Why test, and at which level?

Running your program by hand and reading the output is testing too, but it is slow and you will not repeat it after every change. Automated tests are cheap to repeat, so they catch regressions: bugs where something that worked yesterday stops working today.

Tests are usually grouped by how much of the system they exercise:

A healthy project has many fast unit tests, fewer integration tests and a handful of end to end tests. This unit concentrates on unit tests, but pytest runs all of them the same way.

unittest: the standard library framework

The unittest module ships with Python. You group tests as methods of a class that inherits from unittest.TestCase. Every method whose name starts with test is a test, and you check results with assertion methods such as assertEqual. A setUp method runs before each test.

Your timing on the Ran line will differ. Notice:

• unittest.main() normally reads the command line and then exits the program. Passing argv and exit=False lets the script carry on afterwards, which is handy inside a single file.
• The report goes to standard error, not standard output.
• In a project you put the tests in files named test_*.py and run python -m unittest in a terminal, which finds and runs them.

Other common assertion methods are assertTrue, assertIn, assertIsNone and assertAlmostEqual. pytest runs unittest test classes unchanged, so an old suite can move to pytest gradually.

doctest: examples that check themselves

doctest finds interactive examples inside docstrings, runs them and compares the output. It keeps the examples in your documentation honest, but it is a poor fit for detailed tests.

pytest can collect these examples too, with pytest --doctest-modules.

Your first pytest tests

pytest is a third party package, so install it into your project's virtual environment (see Setting up Python). The command is the same in PowerShell:

With pytest, a test is a plain function whose name starts with test, and you check results with Python's ordinary assert statement. The code under test lives in its own module, and the test file imports it.

The examples on this page are single scripts: each one writes a module and a test file into the current folder, then runs pytest in process with pytest.main(). That is the same as typing pytest -qq test_textstats.py in a terminal. Two options keep the examples tidy: -p no:cacheprovider stops pytest from creating a .pytest_cache folder, and -qq (very quiet) leaves out the final line of counts and timings, so the output on the page matches yours.

The third test below catches a real bug: str.title() treats an apostrophe as the end of a word.

Read the report from the top:

• Each . is a passing test and each F a failing one.
• The > line is the statement that failed, and the E lines explain why, here with a character level diff (- is the expected text, + is what you got).
• test_textstats.py:14 is the file and line number. (The file that the script writes starts with an empty line, so its line numbers are one higher than in the listing.)
• The summary names the test by its node ID, file::function. You can pass a node ID to pytest to rerun that test alone. Long messages are shortened to fit the terminal.

Plain Python would only say AssertionError. pytest rewrites the assert statements in test modules as it imports them, so it can show the values involved. In a terminal, pytest -q also prints a last line such as 1 failed, 2 passed in 0.05s. pytest.main() returns the same exit code that the pytest command gives the shell:

Pitfall. Assertion rewriting applies to test modules and conftest.py files, not to your application code. Do not use assert to validate input in application code either: running Python with -O removes every assert statement. See Errors and exceptions.

How pytest finds tests

When you run pytest with no arguments, it searches the current folder and its subfolders for tests. This is called discovery:

The default rules are:

• Files: test_*.py or *_test.py. Hidden folders, virtual environments and build folders are skipped.
• Functions: module level functions whose names start with test.
• Classes: classes whose names start with Test and that have no __init__ method; their methods that start with test are collected. Subclasses of unittest.TestCase are collected as well.

Saved as test_shop.py, this file holds two tests:

pytest --collect-only -q lists what would run without running it: here test_shop.py::test_tea_price and test_shop.py::TestBasket::test_total.

Running and selecting tests

These are the options you will use most. Each one also works as a list item in pytest.main().

-k matches substrings of test names and combines them with and, or, not and parentheses. -m does the same with marker names, which you will meet shortly. --lf and --ff rely on the cache folder that the examples here switch off; in your own projects leave it on (and keep .pytest_cache/ out of version control).

Fixtures: setup and cleanup you can share

Most tests need something prepared first: sample data, a temporary folder, a database connection. A fixture is a function, decorated with @pytest.fixture, that prepares that thing. A test asks for a fixture by naming it as a parameter; pytest calls the fixture and passes in its value. A fixture that uses yield instead of return can clean up afterwards: the code after yield runs when the fixture is no longer needed, even if the test failed.

Fixtures in a file named conftest.py are available to every test in that folder and its subfolders, without an import. The example puts a database fixture there, and the --setup-show option prints each fixture's setup and teardown.

Notice:

• names asks for db, so fixtures can build on each other.
• db has scope="module", shown as M: it is created once and shared by both tests in the file. names has the default function scope (F), so it is rebuilt for each test that uses it.
• The connection is closed when its scope ends. For a module scope that is after the last test in the file has run.

Pitfall. A wider scope is faster, but tests that share a mutable fixture can affect each other. If one test inserts a row into the shared db, the next test sees it. Keep shared fixtures read only, or use function scope.

Built in fixtures

pytest provides many fixtures, and pytest --fixtures lists them. Three are worth knowing from the start:

The tests never touch your real environment variables or files: monkeypatch restores APP_USER afterwards, and tmp_path is a new folder for every test.

Running one test over many inputs

Copying a test to try another input is tedious. @pytest.mark.parametrize takes a string of parameter names and a list of value tuples, and turns one test function into one test per tuple. ids gives each case a readable name. The last case below fails on purpose:

The id in square brackets tells you at once which case failed. Yet the formula is right, and so is the expected value: 36.6 °C really is 97.88 °F. The failure comes from binary floating point, which cannot store 36.6 exactly, so the result is off in the 14th decimal place. The bug is in the test, which compares floats with ==; the fix is pytest.approx, covered below.

To give a single case its own id or marker, wrap it in pytest.param, for example pytest.param(36.6, 97.88, id="body").

Markers: skip, skipif, xfail and your own

A marker is a label attached to a test with @pytest.mark.<name>. Three built in markers change how a test runs, and they are often confused:

xfail does not mean "this test does not matter". It records a known failure so the suite stays green, and an X tells you the bug has been fixed and the marker can go. With strict=True (or the strict_xfail setting) an unexpected pass counts as a failure, so nobody forgets to remove the marker.

You can also invent markers, such as slow, and select by them with -m. Register them in pyproject.toml, as the script below does; otherwise pytest warns about an unknown marker. strict = true turns on all of pytest's strictness checks, so a typo such as @pytest.mark.slwo becomes an error and unexpected xpasses fail. pytest 9.0 added this native [tool.pytest] table; older versions use [tool.pytest.ini_options] instead.

test_big_sum does not appear at all: -m "not slow" deselected it. With -q in a terminal, the last line would read 2 passed, 1 skipped, 1 deselected, 1 xfailed plus a time.

Checking exceptions and approximate values

pytest.raises is a context manager that passes only if the code inside it raises the given exception. Its match argument is a regular expression searched for in the exception's message, and the object it returns holds the exception. pytest.approx compares numbers with a tolerance, by default a relative difference of one in a million. Both work outside test files too, which makes them easy to try:

approx also accepts lists, tuples and dictionaries of numbers, and abs= or rel= set the tolerance yourself. Changing the failing test above to assert to_fahrenheit(celsius) == pytest.approx(expected) makes all four cases pass.

Pitfall. Keep the with pytest.raises(...) block to the one line that should raise. If an earlier line in the block raises the same exception type by accident, the test still passes. Avoid pytest.raises(Exception): it accepts almost any bug.

Replacing dependencies with mocks

Code that calls a web service, reads the clock or sends email is hard to test: the result changes, the call is slow, or it has side effects. A test double stands in for the real dependency during the test. The standard library's unittest.mock provides Mock objects, which accept any call, return what you configure and record how they were called:

return_value sets what every call returns. side_effect can be an exception to raise, or a list of results (and exceptions) to produce one per call.

unittest.mock.patch swaps a real object for a mock for the duration of a with block or a decorated test, then puts the original back. Below, describe() calls fetch_json(), which would make a real HTTP request. The tests patch fetch_json, so no network is needed:

The string passed to patch names the place where the object is looked up, not where it was defined. describe() looks up fetch_json in the weather module, so you patch "weather.fetch_json". If another module did from weather import fetch_json, it would hold its own reference, and you would patch "that_module.fetch_json" instead.

monkeypatch.setattr(weather, "fetch_json", fake) does the same job; use whichever reads better. Adding autospec=True to patch makes the mock reject calls that do not match the real function's signature.

Tip. Needing many mocks for one test is a design smell. Passing dependencies in as arguments (a fetch function, a clock) often makes code easier to test than patching does.

Coverage and parallel runs

Coverage measures which lines of your code ran during the tests. The pytest-cov plugin wraps the coverage.py tool:

The report lists, per module, the number of statements, how many were missed and the line numbers of the missed ones. Add --cov-branch to also check that both sides of each if ran. Coverage shows what your tests did not exercise; it cannot tell you whether the tests that did run check anything useful. Aim to cover the important behaviour rather than to hit 100%.

pytest runs tests one after another. The pytest-xdist plugin spreads them over several worker processes:

-n auto starts as many workers as you have CPUs, and -n 4 starts four. Parallel runs only work if tests are independent: they must not rely on running in a particular order, or write to the same files. tmp_path gives every test its own folder, which helps.

Laying out a test suite

For anything bigger than a script, keep the tests in a tests/ folder next to a src/ folder that holds your package (see Modules and packages):

Your tests should import the package the way its users will. The usual way is to install the project into the virtual environment in editable mode, with python -m pip install -e .. For a quick start, the pythonpath setting adds src to the import path instead. testpaths tells pytest where to look when you run it with no arguments:

A few habits keep a suite healthy:

• Name tests after behaviour: test_word_count_ignores_extra_spaces says what broke when it fails.
• Arrange, act, assert: prepare the inputs, call the code once, then check the result.
• Keep tests independent: each test sets up what it needs, so any test can run alone or in any order.
• Run the suite automatically on every push. This is called continuous integration (CI).

A CI job is a short script that your code host (GitHub Actions and GitLab CI are two common services) runs on a clean machine each time you push. For a Python project it repeats what you do by hand: fetch the code, install Python, create a virtual environment, install the project with its test dependencies, and run python -m pytest. Many projects run Ruff and a type checker in the same job, and repeat the job on every Python version they support. If a step fails, the commit is marked as failed, so a broken change is caught before it is merged. Each service describes these steps in its own file format, and its documentation has a ready template for Python.

Check yourself

1. What is the difference between @pytest.mark.skip and @pytest.mark.xfail?
2. Why does pytest show the values inside a failing assert, while plain Python only raises AssertionError?
3. In a fixture that uses yield, when does the code after yield run?
4. A fixture has scope="module". How many times is it created for a test file with five tests that all use it?
5. Your module does from time import monotonic. A test patches "time.monotonic", but the code still sees the real clock. Why?
6. Why does assert 0.1 + 0.2 == 0.3 fail, and what should a test use instead?
7. A test passes when run alone but fails when the whole suite runs. What is a likely cause?
8. Your coverage report says 100%. What does that guarantee?

Suggested answers

1. A skipped test does not run at all. An xfail test runs; a failure is reported as expected (x), and a pass is reported as X (or as a failure in strict mode), which tells you the known bug is gone.
2. pytest rewrites assert statements in test modules as it imports them, adding code that records the intermediate values for the report.
3. When the fixture's scope ends, whether the tests passed or failed. For the default function scope that is after each test; for a module scope it is after the last test in the file. It is the teardown step.
4. Once. Every test in the file receives the same object.
5. from time import monotonic copies a reference into your module's namespace at import time. Patching time.monotonic replaces the attribute on the time module only. Patch the name where your code looks it up: "your_module.monotonic".
6. Binary floating point cannot represent 0.1, 0.2 or 0.3 exactly, so the sum is 0.30000000000000004. Use pytest.approx(0.3) (or math.isclose).
7. The tests are not independent: an earlier test leaves state behind (a shared mutable fixture, a global variable, a file, an environment variable) that the failing test depends on.
8. Only that every line ran at least once during the tests. It says nothing about whether the results were checked, or about combinations of inputs and branches that were never tried.

Exercises

These exercises are optional practice. Pick the ones that help you.

1. Test a word counter. This function is meant to count words case insensitively and ignore punctuation, while keeping apostrophes inside words:

   Write at least four pytest tests for it: empty text gives {}; "Tea tea TEA" gives {"tea": 3}; "Stop. Stop!" gives {"stop": 2}; "don't panic" gives {"don't": 1, "panic": 1}. Run them, see which fail, then fix the function until they all pass. Hint: keep only letters, digits and apostrophes.

2. A temporary file fixture. Write count_nonblank_lines(path), which returns how many lines in a UTF-8 text file contain something other than whitespace. Write a fixture notes_file that uses tmp_path to create a file with the lines "first", "", "second", " " and "third", and returns its path. Test that the count is 3, and that a missing file raises FileNotFoundError.

3. A parametrised palindrome test. Write is_palindrome(text), which ignores case and every character that is not a letter or digit. Test it with one parametrised test and readable ids: "" → True, "Level" → True, "A man, a plan, a canal: Panama" → True, "12321" → True, "python" → False, "ab" → False.

4. Mock the clock. A web session expires after 300 seconds without activity:

   Write three tests that never wait: a session checked 200 seconds after creation is not idle; one checked 301 seconds after creation is idle; a session touched at 250 seconds and checked at 500 seconds is not idle. Use unittest.mock.patch with side_effect to supply the clock readings.

Solutions

1. Test a word counter.

Against the original function, only the punctuation test fails: count_words("Stop. Stop!") returns {'stop.': 1, 'stop!': 1}. That is why it pays to test several kinds of input. The fixed function replaces every other character with a space before splitting:

2. A temporary file fixture.

chr(10) is the newline character. It is spelt that way only because this test file is itself written from inside a Python string; in a real test file you would write "\n".

3. A parametrised palindrome test.

4. Mock the clock.

Each value in side_effect is returned by one call to monotonic(), in order: creation, then touch(), then the check inside is_idle(). The patch target is "session.monotonic" because that is where Session looks the name up.