Big data

A retailer that combines website clicks, store receipts and support tickets can collect billions of events a day. No single database server can hold and query that comfortably, so the data is split across many machines, processed in parallel and often streamed in continuously. Standards body NIST describes big data by four characteristics, volume, velocity, variety and variability, and ties the term to needing a scalable architecture rather than to any fixed size.

In episode 66 at 2:45, Jitender Aswani recalls the early big data era at Meta, when Hadoop and Hive dominated and Hive pipelines over data from roughly a billion daily users ran for hours, slowing product experiments. He tells that story as the backdrop to the Presto query engine, later Trino. At 28:54 he argues that the habit of moving all data into one central lake started in that era and creates its own governance and pipeline problems. That is his case for data federation, which queries data where it already lives.

Big data tools solve storage, processing and query cost. They don’t by themselves make a model better. For machine learning, relevant, accurate and correctly labeled data usually matters more than raw volume, and a larger pile of data can still be skewed or out of date.