Files and data formats

Most useful programs read data from somewhere and write results somewhere else. This unit starts with the basics, paths and the open() function, and then works through the formats you will meet most often: CSV, JSON, TOML, YAML and XML. It finishes with pickle, the columnar Parquet format, and compressed archives.

Each format comes with a library and a few traps. Some of the traps are about correctness, such as line endings in CSV or dictionary keys turning into strings in JSON. Others are about security: some loaders will run code hidden in a file if you let them. You will learn the safe default for each one.

You will be able to:

• Build, inspect and search file paths with pathlib, and read and write text with an explicit encoding.
• Choose the right open() mode, and explain how Python handles line endings and binary data.
• Read and write CSV, JSON, TOML, YAML and XML with the standard tool for each, using safe defaults.
• Explain what pickle is for and why you must never unpickle data you do not trust.
• Store a table in Parquet with pyarrow, and compress or bundle files with gzip and zipfile.

Paths with pathlib

A pathlib.Path represents a location in the file system. You join parts with the / operator, and the path object knows its name, suffix and parent. Creating a Path does not touch the disk; only methods such as exists() or read_text() do.

print(report) would show data\2026\sales.csv on Windows and data/2026/sales.csv elsewhere, because a Path uses the separator of the operating system. The examples call .as_posix() so that the output is the same everywhere. Your code never needs to care: build paths with / and let pathlib do the rest.

The next program creates a small project tree and then looks around it. mkdir(parents=True, exist_ok=True) creates missing parent folders and does not complain if the folder is already there. read_text() and write_text() open, read or write, and close the file in one call.

iterdir() lists one folder, glob("*.md") matches names in one folder, and rglob("*.py") searches every folder below. All three return paths in whatever order the file system gives them, so the examples sort them.

Pitfall. Never build paths by gluing strings together, as in folder + "\\" + name. That breaks on other operating systems, and backslashes inside normal string literals start escape sequences. Use Path(folder) / name.

Opening files: modes, encodings and newlines

open() returns a file object. Use it in a with block so the file is closed even if an error happens halfway through (Errors and exceptions explains how with works). The mode string says what you want to do:

The "+" keeps the meaning of the letter it follows. "r+" reads and writes a file that must already exist, and does not empty it. "w+" empties the file first, then lets you read back what you write. "a+" appends, and can read as well. When you switch between reading and writing on the same file object, call seek() first so that you know where the next operation happens.

"t" (text) is the default, so "r" means the same as "rt". Two keyword arguments matter for text mode:

• encoding="utf-8" says how characters become bytes. Python 3.15 turned on UTF-8 mode by default (PEP 686), so UTF-8 is used when you leave it out. Older versions use the system's locale encoding instead, for example cp1252 on many Windows machines. Pass the encoding every time: your code then behaves the same on every version and every machine. Text, bytes and Unicode covers encodings in depth.
• newline controls line endings. By default, reading turns \r\n and \r into \n, and writing turns \n into the platform's line ending (\r\n on Windows). newline="" switches the translation off.

What to notice:

• write() takes one string and nothing else. The mode belongs to open(), not to write(). Neither write() nor writelines() adds newlines for you; print(..., file=f) does.
• for line in f reads one line at a time, so it works on files far bigger than your memory. Each line keeps its \n, hence rstrip("\n"). f.read() returns the whole file and f.readlines() returns a list of all lines; use them only for small files.
• Mode "x" is the safe way to say "create this file, but never overwrite an existing one".

Binary mode works with bytes and exact byte positions. tell() reports the current position and seek() moves it:

Note. In text mode tell() returns an opaque number, not a character count. Only seek() to 0 or to a value that tell() gave you. Use binary mode when you need real byte offsets.

CSV

CSV (comma separated values) looks simple until a value contains a comma, a quote or a line break. The csv module handles all of that, as long as you give it a file object opened with newline="".

The writer quoted only the fields that needed it, doubled the quote inside said "hi", and ended every row with \r\n, the line ending the CSV standard asks for.

Pitfall. Forget newline="" when writing on Windows and the text layer turns each \r\n into \r\r\n, so spreadsheet programs show a blank line between rows. When reading, a missing newline="" can break quoted values that contain line breaks. csv.reader and csv.writer take a file object, never a file name.

Most real files have a header row. csv.DictReader turns each row into a dict keyed by the header, and csv.DictWriter does the reverse:

CSV has no types: everything you read is a string, so convert what you need. The quoting argument changes when fields are quoted:

Files exported with a European locale often use semicolons; pass delimiter=";" to both reader and writer.

JSON

JSON is the usual format for web APIs and configuration. The json module has four functions: dumps (object to string), loads (string to object), and dump and load, which do the same with a file. The types map like this:

The mapping is not a perfect round trip. A tuple comes back as a list, and JSON object keys are always strings: int, float, bool and None keys are converted, and any other key type raises TypeError.

By default dumps escapes every non ASCII character (\u00eb). That is valid JSON, but ensure_ascii=False keeps the text readable; write such files with encoding="utf-8". Note the last line too: loads returns whatever the JSON text contains, not always a dict.

Dates, Decimal and sets have no JSON equivalent, so json.dumps(date.today()) raises TypeError: Object of type date is not JSON serializable. Pass a default= function: json calls it for every object it cannot handle, and writes whatever the function returns.

indent=2 makes the file readable for humans. When reading, the dates are plain strings again: converting them back is your job. When a file stores money as JSON numbers, json.load(f, parse_float=Decimal) reads them as Decimal and avoids binary floating point rounding.

Pitfall. json.dumps(float("nan")) returns NaN, which is not valid JSON and which many other languages refuse to parse. Pass allow_nan=False to get a ValueError instead.

TOML with tomllib

TOML is a configuration format with real types: strings, integers, floats, booleans, dates, arrays and tables. It is the format of pyproject.toml (Packaging and tooling). Since Python 3.11 the standard library can read it with tomllib. It cannot write TOML; the third party tomli-w package does that.

tomllib.load() insists on binary mode because the TOML specification fixes the encoding as UTF-8, so the parser decodes the bytes itself. A syntax error raises tomllib.TOMLDecodeError.

YAML with PyYAML

YAML is popular for configuration files such as CI pipelines and Kubernetes manifests. It is not in the standard library: install the third party PyYAML package with python -m pip install PyYAML (the import name is yaml).

Always load with yaml.safe_load(), which builds only plain Python objects: dicts, lists, strings, numbers, booleans, dates and None. The other loaders understand Python specific tags that build arbitrary objects: yaml.load(..., Loader=yaml.UnsafeLoader) will run code named in the file, and FullLoader has had code execution vulnerabilities. Write with yaml.safe_dump().

Pitfall. PyYAML follows YAML 1.1, where NO, yes, on and off are booleans, 3.10 is the float 3.1, and a leading zero makes a number octal (01234 became 668). Quote any value that must stay a string. safe_dump adds the quotes for you when writing, as it did for '42'.

XML with ElementTree

XML stores a tree of elements. Each element has a tag, a dict of attributes, optional text, and child elements. xml.etree.ElementTree (imported as ET by convention) reads a file with ET.parse(path), which returns an ElementTree; call .getroot() on it. ET.fromstring(text) parses a string and returns the root element directly.

find() returns the first match or None, findall() returns a list, and iter() walks the whole subtree. A plain tag such as "book" only matches direct children; iter("title") and ".//title" search at every depth. get() reads an attribute with an optional default.

find(), findall(), findtext() and iterfind() accept a small subset of XPath. On the library above:

book[@lang!='en'] skips b3, which has no lang at all. There are also [.='text'] and [.!='text'], which compare an element's complete text, including the text of its descendants. Anything outside this subset, such as contains(), raises SyntaxError: invalid predicate; use a loop in Python or the third party lxml library.

To change a document, edit the elements and write the tree back. ET.SubElement(parent, tag) is a module level function that creates a child and appends it; set(), remove() and the text attribute do the rest.

Pitfall. XML that comes from outside your control can be built to attack the parser. The best known attack is "billion laughs", where nested entity definitions expand a small file into gigabytes. Current Python releases bundle a version of the Expat parser that detects this kind of expansion, so ElementTree raises ParseError instead of filling the memory. The risk remains when Python uses an old system copy of Expat (older than 2.7.2, according to the documentation). pyexpat.EXPAT_VERSION shows which one you have. If you parse untrusted XML and cannot be sure, the third party defusedxml package adds a second layer of defence: defusedxml.ElementTree.fromstring() and parse() refuse those constructs outright.

Pickle: Python objects as bytes

pickle turns almost any Python object into bytes and back. It is convenient for caching your own intermediate results, and multiprocessing uses it to send data between processes. It is Python only, so it is a poor choice for data other programs will read.

Unlike JSON, the tuple and the set survive. The protocol is the version of the byte format: protocols 0 to 5 exist, and newer ones are faster and more compact. On Python 3.15 both the default and the highest protocol are 5. Pass protocol= only when an older Python must read the data.

The danger is that a pickle is a small program that pickle.loads() executes. Any class can tell pickle which function to call when the object is rebuilt, and nothing stops that function from deleting files or opening a network connection. This harmless demonstration makes it call print:

Pitfall. Never unpickle data from a source you do not trust: a download, an upload, a message queue, or a file someone could have replaced. Use JSON or another data only format for anything that crosses a trust boundary.

Columnar and binary formats: Parquet

CSV and JSON store data row by row as text. Parquet stores it column by column in a compressed binary file with a schema, so each column keeps its type, files are much smaller, and a reader can load only the columns it needs. pandas, Polars, DuckDB and Spark all read it. In Python the most widely used library for it is the third party pyarrow package (python -m pip install pyarrow); pandas' read_parquet() and to_parquet() use it by default when it is installed (Working with data).

Avro and MessagePack are listed for recognition only; install their libraries when a project needs them. The same goes for Excel workbooks: the third party openpyxl package reads and writes .xlsx files, and pandas uses it for read_excel().

Compressed files and archives

gzip.open() works like open() but compresses on the fly; use "wt"/"rt" with an encoding for text. zipfile.ZipFile reads and writes ZIP archives that hold many files.

The thousand lines (about 20 KB of text) shrink to well under 5 KB, because they are so repetitive.

Pitfall. An archive can contain names such as ../../evil.py that point outside the target folder. zipfile tries to neutralise them, and since 3.14 tarfile applies its safe "data" filter by default, but inspect archives from untrusted sources before extracting them, and never extract them into a folder that matters.

Check yourself

1. Python 3.15 uses UTF-8 by default. Why should you still write encoding="utf-8"?
2. What is the difference between the modes "w", "a" and "x" when the file already exists?
3. Why must a file used with the csv module be opened with newline=""?
4. You dump {"point": (1, 2), 7: "seven"} to JSON and load it back. What do you get?
5. Why does tomllib.load() need a file opened in "rb" mode, and how do you write a TOML file?
6. What does yaml.safe_load("country: NO") return, and why is safe_load the right function?
7. In ElementTree, what is the difference between root.findall("title") and root.iter("title")?
8. Why is pickle.loads() on downloaded data dangerous?

Suggested answers

1. So the code behaves the same on older Pythons and on any machine, where the default is the locale encoding (often cp1252 on Windows). It also documents your intent.
2. "w" empties the file and starts writing, "a" keeps the content and writes at the end, and "x" raises FileExistsError and leaves the file untouched.
3. The csv module handles line endings itself: it writes \r\n and parses quoted fields that contain line breaks. With translation on, Windows writes \r\r\n (blank rows), and embedded line breaks can be mangled on reading.
4. {"point": [1, 2], "7": "seven"}: the tuple becomes a list and the int key becomes a string.
5. TOML files are always UTF-8, so the parser reads bytes and decodes them itself. The standard library cannot write TOML; use a third party package such as tomli-w.
6. {'country': False}, because YAML 1.1 treats NO as a boolean; quote it to keep the string. safe_load builds only plain data types, while the full loaders can construct arbitrary objects and have had code execution vulnerabilities.
7. findall("title") searches only the direct children of root; iter("title") walks every level of the tree (like findall(".//title")).
8. Unpickling runs code chosen by whoever made the bytes, for example through __reduce__, so a malicious pickle can do anything your program can do.

Exercises

These exercises are optional practice. Pick the ones that help you.

The original lab files are not part of this course, so each exercise embeds a small sample. Write each solution so that it creates its sample file first and then reads it back, as the examples above do.

1. Crimes by type. The full Chicago crime dataset (data courtesy of the City of Chicago Data Portal) is a large CSV file with one row per reported incident. Using the embedded sample below, write count_by_type(path, year) that returns a list of (primary_type, count) pairs for incidents in that year, sorted by count (highest first) and then by name. Then write the result to crimes_<year>.csv with the header PrimaryType,Count. Dates look like 01/15/2012 08:30:00 AM. You can parse them with datetime.strptime and the format "%m/%d/%Y %I:%M:%S %p"; the format codes are explained in A tour of the standard library. For 2012 the first pair is ("BATTERY", 3).

2. Quiz answers (JSON). A quiz file nests questions by topic: {"quiz": {"maths": {"q1": {"question": ..., "options": [...], "answer": ...}}}}. Write answers_only(quiz) that keeps the same nesting but each question holds only question and answer. Read quiz.json, write quiz_answers.json with indent=2 and ensure_ascii=False, and print it. Use your own questions; the solution uses three.

3. Quiz answers (YAML). Do the same for a YAML version of the quiz with yaml.safe_load and yaml.safe_dump(..., sort_keys=False, allow_unicode=True). Check that a numeric answer such as '42' stays a string after the round trip.

4. Payments per currency. A payments.xml file holds <payment> elements, each with CARDPAN, EXPIREDATE (MM/YY), COUNTRY and PAYAMOUNT (such as EUR361.29). Write totals_for_active_cards(path, today) that returns a dict mapping each currency to the sum of its amounts, counting only cards that have not expired on today. A card is valid until the last day of its expiry month. Use Decimal for money. Use test card numbers only, as in the sample below. For this sample and 11 November 2026, the totals are EUR 375.00 and USD 518.44.

5. Restructure payments. Transform the same sample. Inside each PAYAMOUNT element, replace the text with two child elements, <AMOUNT> and <CURRENCY>. Add a groups attribute to CARDPAN that counts its digit groups ("4111 1111 1111 1111" has 4, "4242424242424242" has 1). Remove COUNTRY, and write the result to payments_new.xml. The first payment then holds <CARDPAN groups="4">4111 1111 1111 1111</CARDPAN>, <EXPIREDATE>07/27</EXPIREDATE>, and a <PAYAMOUNT> with <AMOUNT>361.29</AMOUNT> and <CURRENCY>EUR</CURRENCY> inside it.

6. Words starting with A. Read a text file and write every word that starts with a or A to a_words.txt, one per line, in order, without surrounding punctuation. For the text Anna arrived at eight. As always, an announcer apologised; Bob ate an apple. the file holds 11 words, starting with Anna and ending with apple.

Solutions

1. Crimes by type.

Sorting on (-count, name) gives "highest count first, then alphabetical" in one pass. On the full dataset the same code works unchanged because it streams the rows.

2. Quiz answers (JSON).

3. Quiz answers (YAML).

4. Payments per currency.

Comparing (year, month) tuples handles the "valid until the end of the month" rule without any day arithmetic. The 04/26 card is expired; the 11/26 card still counts in November 2026.

5. Restructure payments.

str.split() with no argument splits on any run of whitespace, so it counts groups correctly whatever the spacing. The other two cards get groups="1" and groups="3".

6. Words starting with A.

word[:1] is safe even for an empty string (a token made only of punctuation strips down to ""), where word[0] would raise IndexError.