Text, bytes and Unicode

Computers store bytes, but people read text. Every time a program reads a file, receives a web page or prints a name, something has to translate between the two. When that translation uses the wrong rules, you get errors such as UnicodeDecodeError, or garbled text like привет where привет should be.

This unit explains how text becomes bytes and back: code points, encodings such as UTF-8 and UTF-16, error handlers, and the Unicode details that make two equal looking strings compare unequal. Strings and text formatting covered what you can do with a str; this unit covers what a str is.

You will be able to:

• Explain the difference between a character, a code point and a byte
• Encode and decode text with an explicit codec, and choose an error handler on purpose
• Describe how UTF-8 and UTF-16 store code points, including surrogate pairs and the byte order mark
• Inspect characters with ord, chr, escapes and unicodedata
• Normalise and casefold strings before comparing them, and count user visible characters correctly
• Read and write files with an explicit encoding

Characters, code points and bytes

Three ideas are easy to mix up:

• A character is a unit of text as a reader sees it, such as é or €.
• A code point is the number Unicode assigns to a character, written U+ plus at least four hex digits: é is U+00E9.
• A byte is eight bits, a number from 0 to 255. An encoding (also called a codec) is a rule for turning code points into bytes and back.

In Python, a str is a sequence of code points and a bytes object is a sequence of bytes. They are separate types, and you convert between them explicitly:

Four code points became five bytes, because UTF-8 needs two bytes for é. Indexing a bytes object gives an int, not a one character string. In the bytes display, Python shows printable ASCII bytes as characters and everything else as \x.. escapes.

Tip. Decode bytes into str as soon as they enter your program, work with str inside it, and encode only when the text leaves again. Keep the encoding name next to the I/O call, where readers can see it.

Before Unicode: ASCII and code pages

ASCII (1963) uses 7 bits, so it has 128 codes. Codes 0 to 31 and 127 are control codes such as newline and tab. The other 95, from 32 (space) to 126 (~), are printable: punctuation, digits, uppercase letters, then lowercase letters. Placing the letters in order made sorting by code straightforward:

ASCII has no é, no € and no Cyrillic. Computers stored text in 8 bit bytes anyway, so vendors and standards bodies filled the upper 128 values (0x80 to 0xFF) in different ways. Each mapping is a code page:

• ISO-8859-1 (latin-1) covers Western European letters and keeps 0x80 to 0x9F for invisible C1 control codes.
• cp1252 (Windows Western) is nearly the same, but puts 27 printable characters, such as € and curly quotes, in 0x80 to 0x9F.
• cp1250 and ISO-8859-2 cover Central European languages. Windows calls ISO-8859-2 code page 28592.
• For Cyrillic there are three incompatible tables: cp1251 (Windows), KOI8-R (common on older Unix systems) and ISO-8859-5 (Windows code page 28595).
• IBM mainframes use EBCDIC, a separate family that does not even agree with ASCII.

The same byte means different things under different code pages:

A bytes object does not record which code page produced it. Decode it with the wrong one and you get mojibake, text garbled by the wrong codec:

Here the damage could be reversed, because cp1251 assigns a character to every byte involved. That is not always true, so fix the decoding instead of repairing its output.

Unicode: one number for every character

Unicode gives every character in every script a single code point, independent of any byte layout. The code space runs from U+0000 to U+10FFFF and is divided into 17 planes of 65,536 code points each. Plane 0, the Basic Multilingual Plane (BMP), holds most characters in everyday use. Planes 1 to 16 are the supplementary planes, home to emoji, historic scripts and rarer CJK ideographs.

The 2,048 code points from U+D800 to U+DFFF are surrogates. They are reserved for UTF-16's internal use (see below) and are never characters on their own. That leaves 1,114,112 − 2,048 = 1,112,064 scalar values that any Unicode encoding can store:

A Python str can hold a lone surrogate, but a correct encoder refuses to write one.

UTF-8

UTF-8 stores each code point in one to four bytes. It is the dominant encoding on the web and the default for Python source files. Its design has three useful properties:

• ASCII text is unchanged: every ASCII character is one byte with the same value.
• No byte of a multibyte sequence is below 0x80, so an ASCII byte such as / or \n never appears inside another character.
• The first byte says how long the sequence is, so a reader can resynchronise after damage, and there is no byte order question.

A code point must use the shortest form that fits. Longer "overlong" forms are invalid, and Python's decoder rejects them: b"\xc0\xaf".decode("utf-8") raises UnicodeDecodeError instead of returning /.

Here is the table turned into code, checked against str.encode:

Each continuation byte carries six payload bits behind the marker 10; the first byte carries the rest behind its length marker. In real code, use str.encode: this function is for understanding the layout.

UTF-16 and UTF-32

UTF-16 works in 16 bit code units. A BMP code point that is not a surrogate fits in one unit. A code point above U+FFFF becomes a surrogate pair: subtract 0x10000 to get a 20 bit number, then put the high 10 bits after 0xD800 and the low 10 bits after 0xDC00:

Pitfall. The split must be exactly 10 bits and 10 bits. Bit shifts (>> 10 and & 0x3FF) make that hard to get wrong. Code that slices a binary string by hand and takes 9 bits for the high half gives the right answer only up to U+103FF and a wrong high surrogate from U+10400 upwards, so a test with one small code point will not catch it. Test with values such as U+10400 and U+1F600, and compare with str.encode as above.

A UTF-16 unit is two bytes, so the byte order matters. utf-16-le and utf-16-be fix the order; the plain utf-16 codec writes a byte order mark (next section) first so that readers can tell. Windows APIs, Java and JavaScript represent strings as UTF-16 code units, which is why surrogates matter even if you never write UTF-16 files.

UTF-32 stores every code point in exactly four bytes. It is simple but large, and it still does not give you one unit per user visible character, as the section on grapheme clusters shows:

The byte order mark and utf-8-sig

The byte order mark (BOM) is the character U+FEFF placed at the very start of a text stream. In UTF-16 and UTF-32 it tells the reader the byte order. UTF-8 has no byte order, but some Windows tools still write the UTF-8 form of the BOM, EF BB BF, at the start of files. Examples are spreadsheet programs saving "CSV UTF-8" and Windows PowerShell 5.1's -Encoding utf8.

Read such a file with utf-8 and the BOM stays in your text as an invisible \ufeff, often glued to the first column name. The utf-8-sig codec removes a leading BOM if there is one and works normally if there is not:

Use utf-8-sig for reading files that may come from such tools. When you write files, use plain utf-8 unless the receiving program needs the BOM.

Encoding, decoding and error handlers

str.encode(encoding="utf-8", errors="strict") turns text into bytes, and bytes.decode(encoding="utf-8", errors="strict") does the reverse. The default is UTF-8, but name the encoding anyway, so that readers do not have to remember the default.

With errors="strict", a character the codec cannot represent raises UnicodeEncodeError, and bytes it cannot interpret raise UnicodeDecodeError. Both are subclasses of UnicodeError and therefore of ValueError (see Errors and exceptions). The errors argument picks another behaviour:

The doubled backslashes are only how bytes displays a single backslash. Now decoding, with bytes cut off in the middle of the last letter:

Note the asymmetry of replace: ? when encoding, U+FFFD REPLACEMENT CHARACTER when decoding. ignore and replace lose data silently, so use them only when a damaged character is acceptable, such as in log output.

surrogateescape is the one lossless handler. It smuggles each undecodable byte through the str as a lone surrogate and restores it on encoding. Python uses it for file names and command line arguments on Linux and macOS, where a name may not be valid UTF-8:

Code points in source code

ord() gives a character's code point and chr() does the reverse. String literals accept several escapes for code points:

ascii() and the !a conversion show any string with ASCII only escapes, which helps when you need to see exactly which code points a string holds. A raw string such as r"\u20ac" keeps the backslash, so it is six characters.

Python reads source files as UTF-8 by default (PEP 3120, since Python 3.0), so you can type "€" directly. A # -*- coding: latin-1 -*- line at the top selects another encoding; you only need it for old files that were saved in one.

unicodedata: names, categories and normalisation

The unicodedata module reads the Unicode Character Database. Python 3.15 ships version 17.0.0 of it:

The two letter general category tells you what kind of character it is: L letters (Lu upper, Ll lower), N numbers (Nd decimal digit), M marks (Mn a combining mark that attaches to the previous character), S symbols (Sc currency, So other), Z separators, and C control and other codes.

Normalisation

Some characters can be written in more than one way. é is either one code point (U+00E9) or e followed by U+0301 COMBINING ACUTE ACCENT. Both look identical, but they are different strings. Normalisation converts text to one agreed form:

NFC is the usual choice for storing text. The K forms also replace "compatibility" characters, such as the fi ligature or a superscript two, with plain equivalents. That loses formatting, so use NFKC for matching and search keys, not for text you will display.

Comparing strings safely

casefold() is a stronger lower() meant for caseless comparison: it turns German ß into ss, for example. But it does not fix composed versus decomposed characters. Normalise as well. The Unicode recipe for caseless matching normalises, casefolds, then normalises again, because casefolding can itself produce unnormalised text:

For identifiers such as user names, normalise with NFKC instead, so that fiona and fiona collide. Normalisation does not merge lookalike letters from different scripts: Cyrillic а (U+0430) and Latin a (U+0061) stay different.

Grapheme clusters

What a reader perceives as one character, a grapheme cluster, can be several code points: a letter plus combining marks, an emoji plus a skin tone modifier, or several emoji joined with U+200D ZERO WIDTH JOINER. len() counts code points, and slicing can cut a cluster in half:

The family emoji is five code points, and text[:3] kept only its first person. Since Python 3.15, unicodedata.iter_graphemes() splits text into clusters using the Unicode segmentation rules. On older versions, the third party regex package offers the same through its \X pattern. Count clusters when you limit the length of user visible text, such as a display name; count bytes when you limit storage.

isdecimal, isdigit and isnumeric

Three string methods test for "number like" characters, from strictest to loosest. Each accepts everything the previous one accepts:

Pitfall. isdigit() is not a safe test before int(): "²".isdigit() is True, but int("²") fails. Use isdecimal(), or better, call int() inside try (EAFP). Note also that int() and float() accept decimal digits from any script, which you may or may not want for input such as an account number.

Encodings and files

Text files are bytes on disk, so open() needs an encoding. Always pass one:

Reading UTF-8 bytes as cp1252 gave no error at all, only mojibake. That is the dangerous case: a wrong codec often "works".

What happens when you leave encoding out depends on the Python version:

• Since Python 3.15, UTF-8 mode is on by default (PEP 686), so open() without an encoding uses UTF-8 on every platform, and sys.flags.utf8_mode is 1. You can turn it off with PYTHONUTF8=0 or python -X utf8=0.
• Before 3.15, the default was the locale's encoding, unless you switched UTF-8 mode on yourself. On many Windows systems that is a code page such as cp1252, so the same script could write different bytes on different machines.

Pass encoding="utf-8" anyway: your code then behaves the same on every version and every setting, and readers do not need to know the defaults. If you really want the locale's encoding, say so with encoding="locale" (3.10+). Running python -X warn_default_encoding (3.10+) emits an EncodingWarning wherever an encoding is missing. locale.getencoding() (3.11+) reports the locale's encoding even when UTF-8 mode is on:

On one Windows machine with Python 3.15 this printed 1, cp1252 and utf-8; your output will differ with your system, locale and version.

Check yourself

1. What is the difference between a character, a code point and a byte?
2. Why can Unicode encodings store 1,112,064 values rather than all 1,114,112 code points?
3. A bytes object decodes without error as cp1252. Does that prove it was cp1252? Why or why not?
4. How many bytes does UTF-8 use for U+0041, U+00E9, U+20AC and U+1F600?
5. What does the replace handler produce when encoding, and what when decoding?
6. "é" == "e\u0301" is False. Why, and does casefold() help?
7. Why is len() of the family emoji 5, and how can you count what a reader sees?
8. Python 3.15 defaults to UTF-8 for open(). Why pass encoding="utf-8" anyway?

Suggested answers

1. A character is the unit a reader sees; a code point is the number Unicode assigns to it (U+00E9); a byte is an 8 bit value that an encoding produces from code points.
2. The 2,048 surrogate code points U+D800 to U+DFFF are reserved for UTF-16's pairs and are not characters, so they cannot be encoded on their own.
3. No. Single byte code pages assign a character to almost every byte, so wrong guesses often decode "successfully" into mojibake. The encoding must come from the source, a header or a specification.
4. 1, 2, 3 and 4 bytes.
5. Encoding writes ? for each unencodable character; decoding inserts U+FFFD � for undecodable bytes.
6. The first is one code point and the second is e plus a combining accent. casefold() does not change that; normalise both strings (for example with NFC or NFD) before comparing.
7. It is three emoji joined by two zero width joiners: five code points, one grapheme cluster. Use unicodedata.iter_graphemes() (3.15+) or the regex package's \X.
8. The code then works the same on older versions, where the default was the locale encoding, and on systems with UTF-8 mode switched off. It also documents the file format for readers.

Exercises

These exercises are optional practice. Pick the ones that help you.

1. Byte budget. Write truncate_utf8(text, max_bytes) that returns the longest prefix of text whose UTF-8 encoding fits in max_bytes, never cutting a code point in half. Reject a negative budget with ValueError. Examples: truncate_utf8("naïve café", 4) is "naï" and truncate_utf8("😀😀", 5) is "😀".
2. UTF-16 by hand. Write to_utf16_units(text) -> list[int] that returns the UTF-16 code units of text, and from_utf16_units(units) -> str that reverses it. The decoder raises ValueError for a high surrogate without a following low one, or for a lone low surrogate. Check to_utf16_units against text.encode("utf-16-be").
3. User name keys. Write username_key(name) that strips surrounding whitespace, applies NFKC and casefolds. Then group the names ["Zoë", " ZOE\u0308", "fiona", "Fiona", "Bob"] by key, and report which keys have duplicates. Expected: zoë groups 'Zoë' and ' ZOE\u0308', fiona groups 'fiona' and 'Fiona', and the key bob holds 'Bob' alone.
4. Undo mojibake. Write repair(text, wrong="cp1251") that assumes text was UTF-8 decoded with the codec wrong. It returns the repaired text, or text unchanged if encoding again or decoding fails. repair("привет") is "привет", repair("Café", "cp1252") is "Café", and repair("Grüße") stays "Grüße".

Solutions

1. Byte budget. A prefix of valid UTF-8 can only be broken at its end, so decoding with errors="ignore" drops exactly the incomplete last sequence.

2. UTF-16 by hand.

3. User name keys.

' ZOË' prints as one accented letter, but the stored string still has the separate combining mark; only the key is normalised.

4. Undo mojibake.

Repairing is a last resort for data that is already damaged. The real fix is to decode with the right codec in the first place.