Edge inference runs a model close to the data source, such as on a phone or meeting-room device. Google’s LiteRT documentation describes an on-device runtime and deployment across CPU, GPU and NPU backends (Google, LiteRT). The term tells you where inference happens. It does not tell you whether the wider application uploads recordings, stores results remotely or sends later requests to a cloud service.
For example, a camera could identify whether a person is in frame locally and send only a framing decision to another device. Write down that data path before calling the design private. Test the local inference time, peak memory and behavior when connectivity is unavailable. If part of the task goes to a remote service, measure that path separately as well.
Explaining the deployment split for Neat’s meeting-room systems, Tormod Ree says, “we do a combination of edge and cloud.” (episode 70, 5:22).
For builders, start from the task’s timing and data-handling requirements rather than treating local deployment as an automatic upgrade. Quantization covers reducing numerical precision, and small language model vs frontier model covers the capability decision. The multimodal AI hub follows systems that combine audio, video and text.
Sources
- Google, LiteRT — Primary source for the explanation and distinctions in this entry.
