Packaging and tooling

So far you have installed the odd package with pip. Real projects need more: a record of exactly which packages they depend on, a description that lets others install them, and tools that catch mistakes before the code runs. This unit covers where packages come from, how pip installs them, how to describe a project in pyproject.toml and pin its dependencies, and the everyday tools around that: pipx and uv, Ruff, type checkers, pre-commit, Jupyter and conda.

Python's packaging tools change faster than the language. This unit sticks to what is standardised in PEPs and what the main tools have offered for some time, and it notes where something is new. When a command here differs from what you see, check the tool's own documentation: python -m pip help install and uv help are always current for the version you have.

You will be able to:

• Explain what PyPI, wheels and source distributions are, and how pip chooses between them.
• Install, inspect, upgrade and remove packages with python -m pip, using version specifiers correctly.
• Record dependencies in requirements.txt and in a pyproject.toml with a [project] table, and explain what a lock file adds.
• Install command line tools with pipx or uv without breaking any environment, and explain the "externally-managed-environment" error.
• Set up Ruff, a type checker and pre-commit for a project, and know when Jupyter or conda fits.

Where packages come from

The Python Package Index, PyPI, is the public repository of Python packages, run by the Python Software Foundation. Anyone can publish there, so read a package's page (its maintainers, release history, source repository and downloads) before you trust it, and type names carefully: attackers publish lookalike names hoping for a typo.

A package on PyPI is published as one or more distributions, files of two kinds:

A wheel's file name ends with three tags: Python version, ABI and platform. py3-none-any means pure Python that runs on any Python 3 and any operating system. A package with compiled code publishes one wheel per platform, with names such as example_ext-2.1.0-cp313-cp313-win_amd64.whl (CPython 3.13, 64 bit Windows). pip picks the best wheel that matches your interpreter and falls back to building the sdist only if none does. A build failure that mentions a compiler usually means no wheel exists yet for your Python version or platform.

The distribution name you install is not always the name you import: python -m pip install PyYAML gives you import yaml, and scikit-learn gives you import sklearn. This program shows facts about your interpreter that matter for installing; your output will differ:

Installing with pip

pip is the standard installer and comes with Python. Run it as python -m pip (or py -m pip on Windows outside a virtual environment): that way it installs into the interpreter you name, not into whichever Python owns the first pip on your PATH. Install into an active virtual environment, never into the system Python.

Pitfall. Put version specifiers in double quotes. Unquoted, < and > are redirection operators in bash, zsh and the Windows Command Prompt. There, pip install requests>=2.32 installs the latest version and sends pip's output to a file (named =2.32 in bash and zsh, 2.32 in the Command Prompt). Windows PowerShell happens to pass this command through unchanged, but that is not something to rely on. Single quotes work in bash and PowerShell but not in the Windows Command Prompt; double quotes work everywhere.

pip uninstall removes only the package you name, not the dependencies it brought in. Since version 20.3 (2020), pip's resolver backtracks: when two packages need different versions of a third, it searches for a combination that satisfies both, and fails with an explanation if none exists.

Tip. Set the environment variable PIP_REQUIRE_VIRTUALENV=true in your shell profile, and pip refuses to install anything outside a virtual environment.

Version specifiers

A requirement specifier is a package name plus rules for acceptable versions, as defined by the Python packaging specifications:

The third party packaging library implements these rules; pip uses it internally. Install it (python -m pip install packaging) to experiment:

The last line is why versions are not compared as strings: as text, "2.10" sorts before "2.9".

requirements.txt

A requirements file lists requirement specifiers, one per line, for pip install -r. Comments start with #, and a line can carry an environment marker after ; to apply only on some systems.

There are two quite different kinds of requirements file, and it pays to keep them apart:

• Handwritten, listing only what your code imports directly, with loose ranges. This says what the project needs.
• Frozen, the output of pip freeze: every installed package, including dependencies of dependencies, each pinned with ==. This records what one environment contained, so that another can be made identical.

pip freeze captures everything in the environment, including tools you installed by hand and forgot about. Tools such as pip-tools (pip-compile) and uv pip compile produce a cleaner pinned file from a handwritten list. For extra safety, pinned files can carry --hash values that pip checks before installing.

pyproject.toml

pyproject.toml is the standard configuration file of a Python project, written in TOML (Files and data formats). It has three kinds of table:

• [build-system] names the build back end, the tool that turns your source into an sdist and a wheel (PEP 517 and PEP 518).
• [project] holds the project's metadata and dependencies (PEP 621).
• [tool.<name>] tables hold settings for individual tools such as Ruff, mypy and pytest.

Here is a complete file for a small library with a command line entry point, laid out in the src/ layout from Modules and packages:

Hatchling is one of several build back ends; setuptools, Flit, PDM's, Poetry's and uv's own are others, and the [project] table works the same with all of them. With the file in place you can:

Publishing to PyPI is a separate step done with a tool such as twine or uv publish; practise on TestPyPI first.

Applications and libraries use pyproject.toml differently. A library is installed into other people's environments, so its dependencies should be as loose as is safe (lower bounds, upper bounds only for known breakage): exact pins would clash with everyone else's. An application such as a web service or a script you deploy is the end of the chain, so you also want every version fixed. That is the job of a lock file.

You can read any pyproject.toml with tomllib, which is how tools read it too:

Lock files

pyproject.toml says what the project needs (requests>=2.32). A lock file records what was actually chosen: the exact version of every package in the dependency tree, where it came from, and the hashes of the files, so that every install of the project, on a laptop, in CI or in production, gets the same bytes.

For years each tool had its own format: uv.lock, poetry.lock, pdm.lock, Pipfile.lock. PEP 751, accepted in 2025, defines a standard, tool independent format in files named pylock.toml. Tools are adopting it: at the time of writing, pip has an experimental pip lock command that writes one, and uv can export one with uv export --format pylock.toml.

• Commit the lock file of an application to version control, and update it deliberately.
• A library is resolved in its users' environments, so its lock file, if any, only serves its own development and CI.

Command line tools: PEP 668, pipx and uv

Some Python packages are mainly programs: Ruff, Black, pre-commit and httpie, for example. Installing them into each project's environment is wasteful, and installing them into the system Python is exactly what PEP 668 blocks: on many Linux distributions and on Homebrew, pip install outside a virtual environment stops with error: externally-managed-environment, as Setting up Python showed. Do not override it with --break-system-packages; that can break tools of the operating system itself.

The marker is a file named EXTERNALLY-MANAGED in the standard library folder, and pip ignores it inside a virtual environment. This program checks both for your interpreter; your output will differ:

pipx installs each command line application into its own hidden virtual environment and puts the command on your PATH. Install pipx itself with your system's package manager where one offers it (sudo apt install pipx on Debian and Ubuntu, brew install pipx with Homebrew), or on Windows with py -m pip install --user pipx. Then:

uv, from Astral, is a fast package installer, resolver and project manager written in Rust. It can stand in for several tools at once:

Other project managers you will meet are Poetry (poetry.lock) and PDM (pdm.lock), both built around pyproject.toml, and Hatch, a PyPA project manager whose build back end, hatchling, appeared above. Pipenv (Pipfile and Pipfile.lock) predates the [project] table; you will find it in older projects. Every one of them installs from the same PyPI and reads the same specifiers, so the knowledge carries over. pip and venv are always available and are what this course uses.

Code quality tools

Three kinds of tool catch problems before your tests even run.

Ruff (also from Astral) is a linter and a formatter. The linter flags unused imports, undefined names, shadowed builtins, outdated idioms and hundreds of other problems, and can fix many automatically; the formatter rewrites code into a consistent style so reviews can focus on content. Configure it in pyproject.toml:

Type checkers read your type hints and report code that contradicts them, without running it. The main ones are mypy and pyright (which powers the Pylance extension in VS Code). Python itself ignores hints at run time, so this program runs, although its last call breaks the annotation:

A type checker spots the mistake before anyone runs the code:

pre-commit runs checks such as Ruff automatically every time you git commit, so problems never reach the repository. It only works inside a Git repository. Git is outside this course; References and further reading points to its documentation. Install pre-commit as a command line tool, for example with pipx install pre-commit. Then list the hooks in a .pre-commit-config.yaml file at the project root. This minimal file runs Ruff's linter and formatter:

Most other tools publish a snippet like this in their documentation. Then:

Jupyter, conda and the py launcher

Jupyter notebooks (.ipynb files) mix code, its output, charts and formatted notes in one document, which suits exploration, data analysis and teaching (Working with data). Install JupyterLab in your virtual environment and start it from there; VS Code and PyCharm can also open notebooks directly.

To use a project's environment from a Jupyter installed elsewhere, register it as a kernel with python -m ipykernel install --user --name my-project (the ipykernel package must be installed in that environment). Notebooks are good for exploring, but hard to test and review: move code you keep into modules and test it.

conda is a separate package and environment manager, popular in data science. It installs packages from its own channels, most commonly the community run conda-forge, and it manages non Python dependencies too, such as C and Fortran libraries and GPU toolkits. Miniforge is a small installer preconfigured for conda-forge. If you use conda, install packages with conda first and use pip inside a conda environment only for what conda does not provide.

On Windows, the py launcher chooses between installed Python versions:

Inside an activated virtual environment, plain python already means the environment's interpreter.

Check yourself

1. What is the difference between a wheel and a source distribution, and why does pip prefer wheels?
2. Why run python -m pip instead of plain pip?
3. What goes wrong with pip install requests>=2.32 in a shell, and how do you fix it?
4. Which versions does ~=2.32.0 accept, and how is that different from ~=2.32?
5. What is the difference between a handwritten requirements.txt and the output of pip freeze?
6. Why should a library's dependencies avoid exact pins, while an application uses a lock file?
7. What does error: externally-managed-environment mean, and what are the right responses?
8. What does Ruff check that a type checker does not, and the other way round?

Suggested answers

1. A wheel is prebuilt and installs by copying files; an sdist must be built first, which can be slow and may need a compiler. pip prefers a wheel that matches your interpreter and platform.
2. python -m pip installs into the interpreter you named. A bare pip may belong to a different Python on your PATH.
3. In bash, zsh and the Command Prompt the shell treats > as output redirection, so pip installs the latest version and its output goes to a file. Quote the specifier: "requests>=2.32".
4. ~=2.32.0 means >=2.32.0 and ==2.32.*, so 2.32.x only; ~=2.32 means >=2.32 and ==2.*, so 2.33 is allowed but 3.0 is not.
5. A handwritten file lists the direct dependencies with loose ranges: what the project needs. A frozen file pins every installed package exactly, including indirect ones and anything else in that environment: what one environment contained.
6. A library is installed alongside other packages, so exact pins would conflict with them. An application is the final consumer and wants every install to be identical, which a lock file guarantees.
7. The system Python belongs to the operating system's package manager (PEP 668), so pip refuses to change it. Use a virtual environment, the distribution's own packages, or pipx or uv tool for command line programs; avoid --break-system-packages.
8. Ruff checks style, unused or undefined names, risky patterns and formatting; a type checker checks that values are used consistently with their type hints, such as passing an int where a str is expected.

Exercises

These exercises are optional practice. Pick the ones that help you.

1. An isolated environment. In a new project folder, create a virtual environment and activate it. Install pandas and NumPy, check from Python that both import and print their versions, then save the exact versions to requirements.txt and read the file. Deactivate the environment and show that pandas is not available to the Python outside it. Finally, recreate the environment from requirements.txt in a second folder.
2. A minimal pyproject.toml. Write a pyproject.toml for a project called word-stats that requires Python 3.13 or newer, depends on PyYAML 6.0 or newer, offers a word-stats command that runs main() in word_stats/cli.py, and has a dev dependency group with pytest and Ruff. Then write a Python script that loads the file with tomllib and checks those facts with assert.
3. Read the specifiers. For the candidate versions 1.9.9, 2.0.0, 2.4.1, 2.10.0 and 3.0.0, predict which ones each of >=2,<3, ~=2.4, ~=2.4.0 and !=2.4.1,>=2 accepts. Then check your predictions with packaging.specifiers.SpecifierSet.

Solutions

1. An isolated environment.

On Windows (PowerShell):

On macOS or Linux (bash or zsh):

requirements.txt lists more than the two packages you asked for: pandas pulls in dependencies of its own, such as python-dateutil, and pip freeze pins them all. After deactivate, the last command fails with ModuleNotFoundError: No module named 'pandas' (unless you once installed pandas outside any environment, which is worth knowing too). To recreate the environment elsewhere:

On Windows, in PowerShell:

You can also run this check inside and outside the environment; your output will differ:

find_spec() looks a module up without importing it, so the check is quick even for a large package.

2. A minimal pyproject.toml.

The checking script writes the same text to a file first, so it runs on its own:

The distribution name uses a hyphen (word-stats), while the import package uses an underscore (word_stats), because a hyphen is not allowed in a Python identifier.

3. Read the specifiers.

The surprise for most people is 2.10.0 under ~=2.4: ~=2.4 means >=2.4 and ==2.*, and 2.10 is a later 2.x release than 2.4. Versions compare part by part as numbers, so 10 is greater than 4.