File system & I/O (and reading online-judge input)

Two audiences need this material: real engineering work (read config, parse logs, dump JSON, walk directories) where the #1 bug is forgetting to close files or assuming the wrong encoding — and online judges, where reading stdin slowly with input() is the difference between Accepted and Time Limit Exceeded.

Why this matters

The lesson is linearized. We start at the lowest level (opening a file, the with context manager, and why it exists), move up to convenient APIs (pathlib), then to structured formats (csv, json), and finally to the interview-critical topic of reading stdin fast.

The one big idea

A file handle is an OS resource that must be released. with open(...) as f: guarantees release even if an exception fires. Everything else — encodings, lazy iteration, pathlib sugar — is about correctness and ergonomics on top of that guarantee.

The with context manager — why it is non-negotiable

An open file holds an OS file descriptor and a write buffer. If you do not close it, the buffer may not flush (data loss) and descriptors leak (you can run out). with calls f.close() automatically on block exit — even if the body raises. Never rely on the garbage collector to close files.

# WRITE mode "w": creates/truncates. The file is closed at block end.
with open(path, "w", encoding="utf-8") as f:     # O(n) to write n bytes
    f.write("line 1\n")
    f.write("line 2\n")
assert f.closed is True                          # `with` closed it for us

# READ mode "r" (default).
with open(path, "r", encoding="utf-8") as f:
    content = f.read()                           # whole file into one str
assert content == "line 1\nline 2\n"

Reading: read / readline / readlines / lazy iteration

Four ways to read, with very different memory profiles. The lazy form — iterating the handle directly — is the idiomatic, memory-safe way for large files: you process and discard line by line without ever holding the whole file in RAM.

with open(path, "w", encoding="utf-8") as f:
    f.writelines(f"row {i}\n" for i in range(5))

# Eager: all lines as a list (fine for small files).
with open(path, encoding="utf-8") as f:
    lines = f.readlines()
assert lines[0] == "row 0\n" and len(lines) == 5

# Lazy: iterate the handle directly — memory-safe.
# str.strip() removes the trailing newline each line carries.
with open(path, encoding="utf-8") as f:
    stripped = [line.strip() for line in f]      # O(1) memory per line
assert stripped == ["row 0", "row 1", "row 2", "row 3", "row 4"]

# readline reads exactly one line (returns "" at EOF).
with open(path, encoding="utf-8") as f:
    first = f.readline()
assert first == "row 0\n"

Writing & appending — mode matters

"w" truncates or creates then writes; "a" appends (creating if missing); "x" is exclusive create (fails if the file exists). Add "b" for binary or "+" for read+write. Reopening in "w" wipes the existing contents — proof that mode matters.

with open(path, "w", encoding="utf-8") as f:
    f.write("first\n")
with open(path, "a", encoding="utf-8") as f:     # append, keep existing
    f.write("second\n")
assert path.read_text(encoding="utf-8") == "first\nsecond\n"

# Reopening in "w" would have wiped "first" — proof that mode matters.
with open(path, "w", encoding="utf-8") as f:
    f.write("fresh\n")
assert path.read_text(encoding="utf-8") == "fresh\n"

Text vs binary mode & encodings

Text mode ("r"/"w") decodes bytes to str using an encoding. Always declare utf-8 — never rely on the platform default, which differs across machines. Binary mode ("rb"/"wb") reads and writes raw bytes with no decoding: use it for images, archives, or when you must control byte layout exactly.

text = "héllo — café ☕"                           # non-ASCII on purpose
with open(path, "w", encoding="utf-8") as f:
    f.write(text)
with open(path, "r", encoding="utf-8") as f:
    assert f.read() == text

# The same file in BINARY: bytes, and utf-8 uses >1 byte per fancy char.
raw = path.read_bytes()                          # bytes, no decoding
assert isinstance(raw, bytes)
assert len(raw) > len(text)                      # multibyte encoding proof
assert raw.decode("utf-8") == text               # decode bytes -> str

pathlib.Path — the modern path API

Prefer pathlib over os.path basically always. The / operator joins paths portably, .read_text() / .write_text() are one-liners, and .exists() / .glob() are obvious.

sub = base / "data"                              # `/` operator joins paths
sub.mkdir(exist_ok=True)                         # mkdir -p semantics

(sub / "a.txt").write_text("alpha", encoding="utf-8")   # write in one call
(sub / "b.txt").write_text("beta", encoding="utf-8")
(sub / "note.md").write_text("# hi", encoding="utf-8")

assert (sub / "a.txt").exists() is True
assert (sub / "a.txt").read_text(encoding="utf-8") == "alpha"

# glob: pattern-match files in a directory. Sort for a stable assertion.
txts = sorted(p.name for p in sub.glob("*.txt"))
assert txts == ["a.txt", "b.txt"]

# Useful Path attributes.
p = sub / "a.txt"
assert p.name == "a.txt" and p.suffix == ".txt" and p.stem == "a"
assert p.parent == sub

os / os.path basics

Still useful for environment variables, walking directory trees, and the process working directory. os.path joins and splits string paths; os.listdir lists names; os.walk recurses.

joined = os.path.join(str(base), "x", "y.txt")   # portable join (str API)
assert joined.endswith(os.sep + "x" + os.sep + "y.txt")
assert os.path.basename(joined) == "y.txt"
assert os.path.splitext(joined) == (os.path.join(str(base), "x", "y"), ".txt")
assert os.path.isdir(str(base)) is True

# os.listdir returns the names directly inside a directory.
names = set(os.listdir(str(base)))
assert "data" in names

csv — structured tabular text

Use the csv module, never str.split(","), because it handles quoting, embedded commas, and newlines inside fields correctly. Always pass newline="" when opening a CSV file — the module manages newlines itself. DictReader turns the header row into keys, which is far more robust than indexing by column position.

import csv

rows = [["name", "age"], ["alice", "30"], ["bob", "25"]]
with open(path, "w", newline="", encoding="utf-8") as f:
    csv.writer(f).writerows(rows)                # quotes/escapes for you

# reader: each row is a list of strings (everything is text in CSV).
with open(path, newline="", encoding="utf-8") as f:
    parsed = list(csv.reader(f))
assert parsed[0] == ["name", "age"] and parsed[1] == ["alice", "30"]

# DictReader: first row becomes keys; each row is a dict.
with open(path, newline="", encoding="utf-8") as f:
    records = list(csv.DictReader(f))
assert records[0]["name"] == "alice" and records[1]["age"] == "25"

json — the lingua franca of config and APIs

loads/dumps work on strings; load/dump work on file objects. JSON objects map to Python dicts, arrays to lists, null to None, and true/false to True/False.

import json

data = {"name": "alice", "scores": [90, 85], "active": True, "note": None}

# To/from FILE handles (dump/load).
with open(path, "w", encoding="utf-8") as f:
    json.dump(data, f, indent=2)                 # pretty-printed
with open(path, encoding="utf-8") as f:
    loaded = json.load(f)
assert loaded == data                            # full fidelity round-trip

# To/from STRINGS (dumps/loads). sort_keys gives deterministic output.
s = json.dumps(data, sort_keys=True)
assert json.loads(s) == data
assert s.index('"active"') < s.index('"name"')   # keys sorted in the text

tempfile — scratch space that cleans itself up

Use it for tests, demos, and intermediate files you must not leak. A TemporaryDirectory removes itself and everything inside on context exit — no cleanup code needed.

import tempfile

with tempfile.TemporaryDirectory() as d:
    p = Path(d) / "scratch.txt"
    p.write_text("temp", encoding="utf-8")
    assert p.exists() is True
    recorded = str(p)
assert not os.path.exists(recorded)              # gone after the block

Reading input in online judges — the section that wins time-limited problems

Competitive and OJ problems feed your program via stdin. How you read it affects both correctness and speed.

The helpers below take an explicit stream argument (defaulting to sys.stdin) so they are unit-testable with io.StringIO. In a real submission you would call them with no argument. The first is the go-to fast bulk read: slurp everything, split on whitespace, map to int.

import sys

def read_all_ints(stream=None):
    """FAST bulk read: slurp everything, split on whitespace, map to int.
    Handles ints separated by spaces and/or newlines uniformly. O(tokens)."""
    if stream is None:
        stream = sys.stdin
    return [int(tok) for tok in stream.read().split()]

For matrix, 2D-DP, or flood-fill problems, parse a grid: a header line R C followed by R rows of C integers each.

def read_grid(stream=None):
    """Parse a grid: first line `R C`, then R rows of C ints each. O(R*C)."""
    if stream is None:
        stream = sys.stdin
    first = stream.readline().split()            # "R C"
    r, c = int(first[0]), int(first[1])
    grid = []
    for _ in range(r):
        row = [int(x) for x in stream.readline().split()]
        assert len(row) == c                     # sanity: row width matches
        grid.append(row)
    return grid

For BFS/DFS problems the standard input format is a header N M (nodes, edges) followed by M lines of u v. Return the node count and an adjacency list.

def read_graph(stream=None):
    """Parse an undirected graph: `N M`, then M lines of `u v`.
    Returns (n, adjacency_list). O(N + M)."""
    if stream is None:
        stream = sys.stdin
    n, m = (int(x) for x in stream.readline().split())
    adj = {i: [] for i in range(1, n + 1)}
    for _ in range(m):
        u, v = (int(x) for x in stream.readline().split())
        adj[u].append(v)
        adj[v].append(u)
    return n, adj

Driving the parsers with io.StringIO simulates piped stdin in a test:

import io

assert read_all_ints(io.StringIO("1 2 3\n4 5\n6")) == [1, 2, 3, 4, 5, 6]

grid = read_grid(io.StringIO("2 3\n1 2 3\n4 5 6\n"))
assert grid == [[1, 2, 3], [4, 5, 6]]

n, adj = read_graph(io.StringIO("3 2\n1 2\n1 3\n"))
assert n == 3
assert adj == {1: [2, 3], 2: [1], 3: [1]}