Data Pipeline

Think of a retailer that wants one daily sales report. Orders sit in an online store database, refunds in a payments tool and stock levels in a warehouse system. A pipeline runs on a schedule: it copies the new records out of each system (extract), fixes dates, currencies and duplicate customer IDs (transform), and writes the result into one table the finance team can query (load). When it works, nobody notices it. When it breaks, the dashboard quietly shows yesterday’s numbers.

ELT swaps the last two steps. The raw data is loaded first, often into a cloud data warehouse or data lake, and transformed there. That became practical once storage and in-warehouse computing got cheap. Pipelines can also run continuously on streams of events instead of in nightly batches.

AI raises the stakes. Training sets, evaluation sets and the documents behind retrieval-augmented generation (RAG) all arrive through pipelines, so a silent bug upstream becomes a wrong answer downstream. Recording which pipeline run produced which data is the job of data lineage.

On the show, guests describe what pipelines cost at scale. In episode 66 at 28:54, Starburst’s Jitender Aswani says centralizing data worked until sources multiplied, and then “the ETL tech debt starts to really accumulate.” His alternative is data federation, which queries data where it lives; that is his company’s approach, not a settled industry verdict. In episode 75 at 38:47, WisdomAI’s Kapil Chhabra describes a customer whose Google Analytics to BigQuery ETL and dashboards cost about $5 million a year, by his account. Episode 75 was sponsored by WisdomAI (sponsor thanks at 4:13).