From 1.7s to 75ms: How Small Changes Sped Up Our Deltalake Reads

Reading a 16 TB Delta Lake table with Polars, and what made it 23× faster.

We read the same Delta Lake table over and over: a training loop pulling batches to feed a model, and a FastAPI route serving one series per request. The data lives on S3, about 16 TB of it, spread across 56,827 files, roughly 200 million rows, partitioned by series. The query is dull. We filter to one series (a uid) and hand back 500 rows. The obvious way to write it:

df = (
    pl.scan_delta(delta_path)
    .filter(pl.col("uid") == uid)
    .head(500)
    .collect()
)

Each call took 1700 milliseconds. That’s fine if you run it once. But we don’t run it once: we run it in a loop, thousands of times, and 1700 milliseconds per pull turns a batch job or a request handler into something that crawls.

The reason is in the call itself. We pass scan_delta a path, and a path means “go figure out the current state of this table from scratch.” Every iteration rescans the Delta path: it re-reads the transaction log from S3 and re-enumerates all 56,827 files, before it touches a single row we actually asked for. We were rebuilding the entire scan from the path on every pass through the loop.

Rather than assume S3 is just slow, we measured where the 1700 milliseconds actually went.

Almost none of it was the query. Of the 1700 milliseconds, about 1100 milliseconds was Delta re-reading and replaying its transaction log from S3. About 260 milliseconds more was enumerating all 56,827 files to build the query plan. Around 300 milliseconds was reading the actual columns, most of which we didn’t even want. The query itself was about 75 milliseconds.

So 95% of the time was setup, and the setup was identical every time. The transaction log doesn’t change between two requests a second apart, and neither does the file list. We were deriving the same facts on every call, throwing them away, and deriving them again.

There were three redundant pieces, and we removed them one at a time.

The first is the log replay, the cost of passing a path. Load the table once and pass the loaded object around instead, and the log gets read a single time in the life of the process rather than once per query. That’s the 1100 milliseconds gone.

The second is the file enumeration. Even with the table loaded, building the scan walks all 56,827 files to decide what the plan looks like. But that walk produces the same answer every time too, so you do it once: build the scanner once, and per request only change the filter and collect.

from deltalake import DeltaTable
import polars as pl

# 1. Load the snapshot once: one transaction-log read
#    from S3, for the whole process.
dt = DeltaTable(delta_path, storage_options=storage_options)

# 2. Freeze the scan once: file enumeration happens here,
#    and the file list is baked into this LazyFrame.
#    Project only the columns you need while you're at it.
base_lf = pl.scan_delta(dt).select(
    "uid", "partition_key_int", "start_time"
)

# 3. Loop: per iteration only swap the filter and collect.
#    No re-read, no re-enumerate.
for uid in uids:
    df = (
        base_lf.filter(pl.col("uid") == uid)
        .head(500)
        .collect()  # ~75 ms each
    )
    ...

The third is reading columns you’re not going to return. This is projection pushdown: when you .select() only the columns you need, Polars pushes that projection down into the Parquet reader so the wide columns are never fetched from S3 at all. Our wide signal columns are thousands of times larger than the scalar ones, and for these queries we don’t want them, so the pushdown drops nearly all the payload bytes. On the raw scan_delta(path) it looked negligible; it shaved under a fifth off a number the log read already dominated, so it’s easy to dismiss. But the log read was masking it. Once the log read and the enumeration are gone, projection is the largest lever left: it takes the reused read from ~380 milliseconds down to ~75, about 5×.

Removing them in order, the cost drops each time. Measured against the live table, same query, warm process:

(Means over repeated warm reads against the live table; S3 latency varies run to run, so treat these as representative, not exact.)

For contrast, once you’re projecting the scalar columns, rebuilding the scan on every call sits steady at ~300 milliseconds while build-once holds at ~75: that ~225 milliseconds gap is the file-enumeration tax over 56,827 files, paid again and again.

A few things surprised us.

Asking for fewer rows doesn’t help. head(1) costs the same as head(500). The floor is per-file and per-column, not per-row, so once you’ve paid to open the right file the rows are nearly free.

Better partition pruning doesn’t help either, which was the counterintuitive one. Our first instinct was to prune harder. But we were already pruning perfectly, 1 file out of 56,827, and pruning happens after the log is replayed, so no amount of it can shrink the thing that was actually slow.

And the 75 milliseconds doesn’t depend on which query you run. Different series, different filter shapes, they all land around 75 milliseconds, because the expensive work is the setup and the setup is the same regardless. That’s what makes the whole thing safe to build a service on.

This points at how you actually serve it, and at the one condition the whole approach depends on: it only works if the process stays alive and reuses the snapshot across many requests. We query the Delta table directly from Polars inside a long-lived FastAPI service, and the trick is where the setup happens: once, at startup, instead of once per request.

Call this from a one-off script that loads the table, runs a single query, and exits, and none of it helps. That script pays the full ~1700 milliseconds every time, because there is no second request to amortize the setup over. The 75 milliseconds is what every request after the first one costs. The build-once pattern only earns its keep inside something that serves request after request off the same loaded state.

In FastAPI that “something” is the running server and its event loop. The snapshot and the built scan get created in the lifespan, stored on app.state, and shared by every request that follows:

@asynccontextmanager
async def lifespan(app: FastAPI):
    table = isiji.deltatable.Merged256()
    # loads the Delta snapshot once
    app.state.delta = DeltaSource(table.dtb)
    # reuse it, no second log read
    yield

app = FastAPI(lifespan=lifespan)

@app.get("/rows/{uid}")
async def get_rows(
    uid: str, request: Request, limit: int = 500
) -> Response:
    df = await request.app.state.delta.fetch(uid, limit)
    return Response(
        content=df.write_json(),
        media_type="application/json",
    )

DeltaSource is just the build-once idea wrapped in an object. It holds the loaded DeltaTable and the one scan built from it, and every request swaps in its filter:

class DeltaSource:
    def __init__(self, table: DeltaTable) -> None:
        self._table = table
        # loaded snapshot, kept for the app's life
        self._base_lf = self._build_lazyframe()
        # the scan, built once
        self._refreshed_at = time.monotonic()
        self._lock = asyncio.Lock()

    def _build_lazyframe(self) -> pl.LazyFrame:
        return pl.scan_delta(self._table).select(SCALAR_COLUMNS)

    async def fetch(self, uid: str, limit: int) -> pl.DataFrame:
        await self._refresh_if_stale()
        lf = self._base_lf
        # capture before collecting, survives a swap
        return await asyncio.to_thread(
            lambda: lf.filter(
                pl.col("uid") == uid
            ).head(limit).collect()
        )

There are a few details to get right. The S3 calls and the .collect() are blocking, so they run in asyncio.to_thread to stay off the event loop. We tested it with 200 concurrent requests and a refresh fired in the middle, and got 200 correct answers and no crashes. End to end, a real request now comes back in about 104 milliseconds: the 75 milliseconds of query plus FastAPI and JSON on top.

There’s one honest catch, and it’s the price of the whole approach. State you build once is frozen in time. It won’t see data written after you built it. So you trade a little freshness for the speed, and you pay it back on a timer. Delta makes the refresh cheap: update_incremental() reads only the new commits instead of replaying the whole log, and then you rebuild the scan so the new file list is actually picked up:

    def _refresh_blocking(self) -> pl.LazyFrame:
        self._table.update_incremental()
        # read only new commits, not the whole log
        return self._build_lazyframe()
        # rebuild so the new files are seen

    async def _refresh_if_stale(self) -> None:
        if time.monotonic() - self._refreshed_at < TTL_SECONDS:
            return
        async with self._lock:
            # only one request rebuilds
            if time.monotonic() - self._refreshed_at < TTL_SECONDS:
                return
            # double-check after waiting for the lock
            new_lf = await asyncio.to_thread(
                self._refresh_blocking
            )
            self._base_lf = new_lf
            # atomic rebind; readers see the
            # old one until here
            self._refreshed_at = time.monotonic()

Two things make that safe under load. Only one request rebuilds when the window expires, because the rest wait on the lock and then find the work already done. And the swap is a single assignment, so an in-flight query either sees the old scan or the new one, never a half-built one. The freshness window is a number you choose on purpose: ours is a day, because the source refreshes on a daily schedule and there’s no point paying for a refresh more often than the data changes.

That’s the whole change. Load the snapshot once, build the scan once, project only the columns you need, and per request swap only the filter. The magnitudes are specific to this table and scale with file count, but the pattern holds anywhere you read Delta from a long-lived process: 1700 milliseconds down to 75 milliseconds, with no faster hardware and no change to the data.