Regular expressions

A regular expression (regex) is a small pattern language for describing text: "four digits, a dash, two digits", or "a word that appears twice in a row". Python's re module searches strings for such patterns, extracts the parts you care about, and replaces or splits text around them.

Regexes are compact and powerful, and they are also easy to get subtly wrong. This unit builds them up one feature at a time, with a runnable example for each, and ends with the cases where a regex is the wrong tool.

You will be able to:

• Decide when a regex is the right tool and when string methods or a real parser are better
• Find, extract, replace and split text with the re functions, and read match objects
• Build patterns from character classes, anchors, quantifiers, groups and alternation
• Use backreferences and lookarounds
• Apply flags, write readable verbose patterns, and escape user input with re.escape
• Recognise patterns that risk catastrophic backtracking

Regex or string methods?

For fixed text, string methods are simpler and faster: in, startswith, split, replace (see Strings and text formatting). Reach for a regex when the text varies in a structured way:

Write patterns as raw strings, r"...". In a normal string literal Python processes backslash escapes first, so "\b" becomes a backspace character before re ever sees it, while r"\b" keeps the two characters \ and b, which re reads as "word boundary":

The first search looked for backspace characters and found nothing. Since Python 3.12, an unknown escape such as "\d" in a normal string also triggers a SyntaxWarning. Raw strings avoid both problems.

The re functions

re.match anchors at the start of the string, not of each line, even with the MULTILINE flag. It returns None when there is no match, like the other functions:

Note. Python 3.15 adds re.prefixmatch() as a clearer name for re.match(), which is now soft deprecated: it keeps working and is not scheduled for removal. Use re.match while your code must run on older versions.

findall returns groups when the pattern has them

findall returns whole matches only when the pattern has no capturing groups. With one group it returns that group's text; with several it returns tuples:

If you need grouping without capturing, use (?:...), as the fourth line does. If you need both the whole match and its groups, use finditer.

Replacing and splitting

In the replacement string of sub, \1 or \g<1> inserts group 1 and \g<name> a named group. Instead of a string you can pass a function: it receives each match object and returns the replacement text:

Pitfall. Pass count, maxsplit and flags by keyword. Passing them positionally to sub, subn and split has been deprecated since Python 3.13, and it is easy to put a flag where the count belongs.

A capturing group in a split pattern keeps the separators in the result.

Compiling a pattern

re.compile turns a pattern into a Pattern object with the same methods. The module caches recently used patterns, so compiling is mainly about readability: give the pattern a name and define it once:

Match objects

A successful search returns a Match object. Group 0 is the whole match; groups 1, 2, ... are the parenthesised parts, numbered by their opening parenthesis from the left:

Pitfall. When nothing matches you get None, and None.group() raises AttributeError. Test the result first, as the if m := ... line above does.

Character classes

For str patterns, \d, \w and \s are Unicode aware. \d is not the same as [0-9]: it also matches Arabic-Indic digits, and \w matches é. Add the re.ASCII flag to restrict them to ASCII:

The last pattern is a useful idiom: "a word character that is not a digit and not an underscore" means "a letter", in any script. Text, bytes and Unicode explains what counts as a digit or letter.

Anchors and boundaries

Anchors match a position, not a character:

In Perl and PCRE, \Z also matches before a final newline and \z means the very end. Python's \Z has always meant the very end; 3.14 added \z so you can write it the way other engines do. The next example uses \z, so it needs Python 3.14 or later. On 3.13, replace \z with \Z.

Pitfall. ^\d+$ accepts "42\n", because $ matches before a final newline. To validate a whole value, use re.fullmatch, or anchor with \A and \Z. On Python 3.14 and later you can write \z instead of \Z; on 3.13 \z is an error.

Quantifiers: greedy and lazy

Quantifiers are greedy: they take as much as they can, then give back one character at a time if the rest of the pattern fails. Adding ? (*?, +?, ??, {m,n}?) makes them lazy: they take as little as possible and extend only when needed:

The greedy .* ran to the last quote on the line. The lazy version and the negated class [^"]* both stop at the next quote; the negated class says exactly what you mean and never needs to backtrack. Since Python 3.11, re also supports possessive quantifiers (*+, ++, ?+) and atomic groups (?>...), which never give characters back.

Groups and backreferences

A backreference matches the exact text a group captured earlier, which lets a pattern find repetition:

The quote pattern accepts text that ends with the same kind of quote it started with, because (?P=q) must repeat whatever q matched.

Alternation and lookarounds

| means "or", and it has the lowest precedence of all: ^cat|dog$ means ^cat or dog$. Group the alternatives to limit them. The engine also takes the first alternative that leads to a match, not the longest:

In the last line, Java matched first, but the following \b failed, so the engine backtracked and tried JavaScript.

A lookaround checks what comes before or after the current position without consuming it:

The code_rule uses two lookaheads at the start to demand "somewhere there is a digit" and "somewhere there is a capital", then matches the six characters. A lookbehind in Python must have a fixed width: (?<=\$) is fine, but (?<=\$|EUR ) is rejected because its alternatives differ in length.

Flags

Combine flags with |, as in flags=re.I | re.M. An inline flag must come at the very start of the pattern, or you can scope it to a group with (?i:...):

VERBOSE makes long patterns readable. Whitespace in the pattern is ignored and # starts a comment, so match a literal space with \s, [ ] or \ :

Escaping text with re.escape

Characters such as ., +, ?, ( and ) have special meanings in a pattern. When you build a pattern from text you do not control, such as user input or a file name, escape it with re.escape. Otherwise the pattern may match the wrong thing, fail to compile, or run for a very long time:

Catastrophic backtracking

When a match fails, the engine backtracks and tries every other way the quantifiers could have divided the text. Usually there are few. But a pattern with nested quantifiers over the same characters, such as (a+)+$, can split a run of n as in about 2ⁿ ways. On a string that almost matches, like "aaaa…ab", the engine tries them all before giving up.

This demo uses short strings only. Your output will differ, but each step adds two characters and the time grows about fourfold:

At this rate, 40 characters would take hours. If an attacker can send such input to your server, a single request can keep a CPU busy: this is called ReDoS (regular expression denial of service). The standard re module has no timeout. Protect yourself:

The first three rewrites match exactly the same strings as the patterns they replace. The last one gives the same yes or no answer when you search one line, but its match ends at the first z instead of the last, and its character classes also cross line breaks.

Also keep untrusted input out of patterns (use re.escape), limit the length of the text you match, and consider possessive quantifiers or atomic groups to forbid backtracking where you know it cannot help.

When not to use a regex

• HTML and XML nest, and regexes cannot track nesting. re.findall(r"<b>(.*?)</b>", "<b>a <b>b</b></b>") returns ['a <b>b']. Use html.parser, xml.etree.ElementTree (Files and data formats) or a third party parser.
• Email addresses: the full grammar is far too complex for a readable regex, and a well formed address can still be undeliverable. Check the basic shape (text, one @, a domain with a dot), then confirm by sending a message.
• JSON, CSV, TOML: use the json, csv and tomllib modules.
• Dates and numbers: datetime.fromisoformat, datetime.strptime, and int() or float() inside try.
• Fixed text: in, str.startswith, str.split and str.replace are clearer and faster.

Check yourself

1. Why should patterns be raw strings?
2. What is the difference between re.search, re.match and re.fullmatch?
3. What does re.findall return for a pattern with one capturing group, and for one with two?
4. re.match(r"\d+$", "42\n") succeeds. Why, and how do you reject the newline?
5. What does ".*?" match in 'say "hi" and "bye"', and why does ".*" behave differently?
6. What does \1 mean inside a pattern, and inside a replacement string?
7. Why do you pass user input through re.escape before putting it in a pattern?
8. What makes (a+)+$ slow on "aaaaaaaaaaaaaaaaaaaaaaab"?

Suggested answers

1. In a normal string literal Python processes backslash escapes first: "\b" becomes a backspace, and unknown escapes such as "\d" trigger a SyntaxWarning. A raw string passes the backslashes to re unchanged.
2. search finds a match anywhere; match only at the start of the string (not of each line); fullmatch only if the entire string matches.
3. With one group, a list of that group's strings. With two or more, a list of tuples, one item per group.
4. $ also matches before a final newline. Use re.fullmatch(r"\d+", ...) or the anchors \A and \Z (\z also works on Python 3.14 and later).
5. "hi" and then "bye". The lazy .*? stops at the first closing quote; the greedy .* runs to the last quote, giving "hi" and "bye" as one match.
6. In a pattern, \1 matches the same text that group 1 captured. In a replacement, it inserts group 1's text.
7. So that special characters such as ., + or ( are matched literally. Unescaped input can change the pattern's meaning, make it invalid, or make it catastrophically slow.
8. The nested quantifiers can divide the run of as in exponentially many ways, and because the final b makes $ fail, the engine tries them all.

Exercises

These exercises are optional practice. Pick the ones that help you.

Each solution keeps its sample data inside the block.

1. "bin" twice. From a list of lines, keep those that contain the letters bin at least twice anywhere, even inside other words. "binary data in the bin" qualifies; "one bin only" does not.
2. "bin" as a word. Keep the lines that contain bin as a whole word: no letter, digit or underscore directly before or after it, matching case exactly. "/usr/bin/env" and "the bin." qualify; "binary", "bin_old" and "BIN" do not.
3. Tandem repeats. Find the words that consist of one chunk written twice in a row, such as blabla, 123123 or aa, and report the chunk. A word is a run of \w characters. hello does not count: ll is only part of the word.
4. Replace a word. Replace every whole word bin with storage, leave words such as binary alone, and report how many replacements were made. For "Put it in the bin. The bin is full; binary files stay. See /usr/bin too." the result is "Put it in the storage. The storage is full; binary files stay. See /usr/storage too." with 3 replacements.
5. Swap two letters. A word is a maximal run of letters (in any script; digits, underscores and punctuation are not letters). In every word of two or more letters, swap the first two letters; leave everything else unchanged. "Hello, world!" becomes "eHllo, owrld!", "it's" becomes "ti's" and "R2-D2" stays the same.
6. Unique emails. Extract the email addresses from a text, in order of first appearance, dropping later duplicates that differ only in case. An address here is one or more of A-Z a-z 0-9 . _ % + -, then @, then one or more dot terminated labels of letters, digits or hyphens, then a top level domain of two or more letters. This is a practical approximation, not full validation. For the text Write to ana@example.com or Ben.Lee@mail.example.org. Copy: ANA@example.com, support@example.invalid; not addresses: @example.com, ana@localhost the result is ["ana@example.com", "Ben.Lee@mail.example.org", "support@example.invalid"].

Solutions

1. "bin" twice.

The assertion shows that str.count gives the same answer without a regex. For a fixed substring it is the better choice.

2. "bin" as a word.

bin_old does not match because _ is a word character, so there is no boundary between n and _.

3. Tandem repeats.

The \b on both sides makes the repeat cover the whole word. Without them, hello would match because of ll.

4. Replace a word.

5. Swap two letters. [^\W\d_] is "a letter". The negative lookbehind (?<![^\W\d_]) means "not preceded by a letter", so the match can only start at the beginning of a word.

6. Unique emails.

The pattern has only a non capturing group, so findall returns whole addresses. The dictionary keeps the first spelling of each address and, like every dict, remembers insertion order. The sentence ending dot after org is not part of the address, because the top level domain must be letters only.