Functional tools and decorators

Because functions are objects (see Functions), you can hand them to other functions. Python builds a whole toolbox on that idea: map and filter apply a function across data, functools adapts and caches functions, and the operator module supplies ready made small functions so you write fewer lambdas.

The same idea powers decorators, the @name lines you see above many function definitions. A decorator wraps a function to add behaviour such as timing, logging, retrying or caching without touching the function's own code. This unit builds one step by step, so that you understand every line instead of copying a recipe.

You will be able to:

Transform and filter data with map, filter and functools.reduce, and judge when a comprehension reads better
Use any/all, operator.itemgetter/attrgetter and functools.partial to write compact key functions and callbacks
Build a decorator step by step, keeping the wrapped function's metadata with functools.wraps
Write decorators that take arguments, and predict the order in which stacked decorators run
Apply practical decorators for timing, logging, retrying and memoisation, including functools.cache and lru_cache
Recognise functools.singledispatch and total_ordering and know when they help

`map`, `filter` and `reduce`

map(func, iterable) applies func to every item and yields the results.
filter(pred, iterable) yields only the items for which pred(item) is true. filter(None, iterable) keeps the truthy items.
functools.reduce(func, iterable, initial) combines the items pairwise into one value: it calls func(acc, item) for each item, feeding each result back in as acc.

map and filter are lazy iterators, like generators (see Iterators and generators); reduce consumes its input and returns a single value.

Rendering diagram…

The second version, a generator expression inside sum, gives the same answer and is how most Python programmers would write it. Squaring keeps parity, so filtering n instead of n * n is equivalent here.

A few more details, shown below: map accepts several iterables and stops at the shortest (since Python 3.14, map(..., strict=True) raises ValueError if their lengths differ, like zip). reduce without an initial value uses the first item as the start and raises TypeError on an empty iterable, so pass an initial value whenever the input might be empty.

Most reductions already have a built in name: sum, max, min, math.prod, str.join, any and all. Prefer those, and keep reduce for combinations without one, such as merging a list of dicts with reduce(operator.or_, dicts, {}).

Pitfall. A filter predicate should return True or False. A function that returns the item itself "works" until an item is falsy, such as 0:

Comprehension or `map`/`filter`?

You want to...	Clearer	Less clear
apply an existing named function	`map(str, nums)`	`[str(n) for n in nums]` (fine too)
apply an expression	`[n * 2 for n in nums]`	`map(lambda n: n * 2, nums)`
filter and transform	`[n * 2 for n in nums if n > 0]`	`map(lambda n: n * 2, filter(lambda n: n > 0, nums))`
keep it lazy	`(n * 2 for n in nums)`	`map(lambda n: n * 2, nums)`

A useful rule: if you would need a lambda, write a comprehension or generator expression instead.

`any`, `all` and the `operator` module

any(iterable) is true if at least one item is truthy; all(iterable) is true if every item is. Both stop as soon as the answer is known. For an empty iterable, any is False and all is True, because no item breaks the rule.

The operator module provides functions for Python's operators (operator.add, operator.mul, ...) and two factories for key functions:

itemgetter(i) builds a function that returns x[i]; with several indexes or keys, it returns a tuple.
attrgetter("name") builds a function that returns x.name; dotted paths such as "address.city" work too.

itemgetter(1) does the same as lambda row: row[1], but it states the intent in one word and runs slightly faster.

`functools.partial`

partial(func, *args, **kwargs) returns a new callable with some arguments already filled in. It is a tidy way to adapt a general function to a callback or key that expects fewer arguments.

Unlike a lambda, a partial stores its arguments when it is created, shows them in .func, .args and .keywords, and doesn't suffer from the late binding trap described in Functions.

Decorators, step by step

Steps 1 and 2: a wrapper, then the `@` syntax

A function is an object you can pass to another function and return from one. So a decorator starts as an ordinary higher order function: it takes a function and returns a new function, the wrapper, that calls the original and adds something. Writing @shout on the line above a def means "define the function, then replace it with shout(function)". It is shorthand for cheer = shout(cheer), and it runs once, when the def statement executes, not on every call.

shout(greet) wraps by hand and leaves greet itself unchanged; @shout does the same wrapping and rebinds the name cheer to the wrapper.

Pitfall. The decorator must return the wrapper, not call it. This broken version calls wrapper() while decorating, so the name run_report ends up bound to None, and the wrapper never calls the original function either:

It prints calling run_report while the def is still being decorated, before anything calls the function, and then fails with TypeError: 'NoneType' object is not callable on the last line. The fix is return wrapper, plus a call to func() inside the wrapper.

Step 3: accept any arguments and pass the result back

A general purpose wrapper accepts *args, **kwargs and forwards them unchanged, and it returns whatever the original function returned.

It works, but the last line shows a problem: the decorated function now calls itself wrapper and has lost its docstring. Tracebacks, help() and test reports will all show the wrong name.

Step 4: keep the metadata with `functools.wraps`

@functools.wraps(func) on the wrapper copies the original's name, qualified name, module, docstring, annotations and attribute dict onto the wrapper, and stores the original in __wrapped__. Use it in every decorator you write.

Step 5: decorators that take arguments

@repeat(times=3) first calls repeat(times=3), and the result is the decorator that is then applied to the function. So a decorator with arguments has three levels: a factory that receives the arguments, the decorator that receives the function, and the wrapper that receives the call's arguments.

Step 6: stacking decorators

Decorators stack from the bottom up: the one nearest the def is applied first. @tag("b") above @tag("i") means title = tag("b")(tag("i")(title)). At call time it is the other way round: the outermost wrapper runs first and calls the next one in.

Practical decorators

Timing

It prints a line such as sum_of_squares took 9.81 ms and then the result, 2666646666700000; your output will differ in the timing. time.perf_counter() is the right clock for measuring durations, and the finally block reports the time even when the function raises (Errors and exceptions covers try and finally).

Logging calls

A logging decorator records every call and its result. It uses the logging module rather than print, so you can turn it up, down or off without editing the decorator (see A tour of the standard library).

To record when each call happened, add %(asctime)s to the format string instead of formatting timestamps yourself.

Retrying a flaky operation

Network calls and other I/O sometimes fail for a moment and then succeed. A retry decorator with arguments makes the policy explicit: how many attempts, which exceptions count as temporary, and how long to wait. The example fakes a flaky service with an iterator of outcomes, so it runs instantly.

Note. Retry only operations that are safe to repeat, and only on exceptions that signal a temporary problem; retrying a ValueError wastes time. Real code usually waits longer after each failure (exponential backoff), as Errors and exceptions and HTTP and web APIs show.

Memoisation, by hand and with `functools`

Memoisation stores each result of a function under its arguments, so a repeated call returns the stored answer instead of recomputing it. It is safe only for pure functions, whose result depends on nothing but their arguments. The handwritten version below shows the idea. Counting the paths across a grid, moving only right or down, calls itself on overlapping subproblems: a 16 × 16 grid takes more than a billion calls without a cache and about five hundred with one.

The standard library already has this, better tested and thread safe: @functools.cache (since 3.9) keeps every result, and @functools.lru_cache(maxsize=n) keeps the n most recently used results and discards the least recently used. Both add cache_info() and cache_clear(). Replacing @memoise with @cache above gives the same answer.

The second square(2) was a hit. Then square(4) pushed out 3, the least recently used entry, so the last square(3) had to be computed again.

Pitfall. Cached functions use their arguments as dict keys, so every argument must be hashable: passing a list raises TypeError: unhashable type: 'list'. Never cache a function whose result depends on the clock, a file or other changing state.

Two more `functools` helpers

@singledispatch turns a function into a generic function that picks an implementation from the type of its first argument. You register implementations with .register, using the parameter's type hint; union types such as list | tuple are accepted since Python 3.11.

True dispatches to the int version because bool is a subclass of int. For methods, use functools.singledispatchmethod (see Classes and objects).

@total_ordering is a class decorator: define __eq__ and one ordering method such as __lt__, and it fills in <=, > and >=. The example is a small class. Classes and objects explains __eq__, __lt__ and the NotImplemented value they return for other types.

Comparing the parts as tuples of ints puts 3.10 after 3.9, which comparing the strings would get wrong.

Check yourself

What does map return, and when does it do its work?
Why does reduce(operator.add, []) fail, and how do you avoid it?
Rewrite list(map(lambda w: w.upper(), filter(lambda w: len(w) > 3, words))) as a comprehension.
What do any([]) and all([]) return?
What is @log_calls above def f(): ... shorthand for, and when does it run?
Why should every decorator use functools.wraps?
With @a on the first line and @b on the second, above def f(), which wrapper runs first when you call f()?
Which functions are safe to cache, and what must be true of their arguments?

Exercises

These exercises are optional practice. Pick the ones that help you.

Compose. Using functools.reduce, write compose(*funcs) that returns a function applying the given functions from right to left: compose(f, g, h)(x) == f(g(h(x))). compose() with no functions returns a function that gives back its argument unchanged. Example: compose(str, abs, int)("-42") gives '42'.
Sorted report. Given records (name, department, salary), write report_order(records) that returns the names sorted by department A to Z and, within each department, by salary from highest to lowest. Use operator.itemgetter. Example: for [("Ann", "IT", 50), ("Bo", "HR", 40), ("Cy", "IT", 70), ("Di", "HR", 45)] the result is ['Di', 'Bo', 'Cy', 'Ann'].
Call counter. Write a decorator count_calls that counts how many times the decorated function has been called and exposes the count as an attribute calls. The decorated function must keep its name and docstring. Example: after two calls to a decorated ping, ping.calls == 2 and ping.__name__ == "ping".
Call limit. Write a decorator factory limit_calls(max_calls) whose wrapper raises RuntimeError("<name> may be called at most <max_calls> times") on any call beyond the limit. max_calls below 0 raises ValueError when the decorator is created. Example: with @limit_calls(2) on launch, two calls work and the third raises RuntimeError: launch may be called at most 2 times.

Solutions

1. Compose.

2. Sorted report.

Python's sort is stable: items with equal keys keep their existing order. So you sort by the secondary key first and the primary key last. With numeric salaries, key=lambda r: (r[1], -r[2]) does it in one pass.

3. Call counter.

4. Call limit.

The counter lives in the closure, one per decorated function, so two functions decorated with @limit_calls(2) have independent limits.

map, filter and reduce

Comprehension or map/filter?

any, all and the operator module

functools.partial