Track · 2:07 · Liner note
The Lookback Window
A lookback window re-reads a recent period of source data on each run, for example the last seven days, to catch late-arriving or corrected records that an incremental load would otherwise miss.
Data is rarely on time. Mobile events arrive days after they happened, payments are adjusted, and someone corrects a record last Thursday. A lookback window accepts that by re-reading a recent stretch on every run and overwriting what it finds.
Choose the width from evidence: measure how late your data actually arrives and cover the ninety-ninth percentile. Too short, and you lose corrections. Too long, and you pay to rescan weeks that never change. This is where incremental loading meets reality, and compromise is a design decision.