Track · 2:09 · Liner note
Incremental Load
An incremental load transfers only the records that are new or changed since the last run, instead of reloading the entire table each time.
Reloading a billion rows to pick up ten thousand new ones is a habit, not a strategy. An incremental load remembers where it stopped, with a timestamp or an ID watermark, and moves only the difference.
The hard part is honesty about the edges: late-arriving rows, updates to old records, deletes that leave no trace. Solve those and the pipeline gets cheaper and faster with every table you convert. Play this while you replace the full refresh and watch the runtime drop.