Edge AI
Edge AI runs AI models on devices or nearby computers close to where data is produced. Local inference can reduce network delay and data transfer, and keep an application working without a cloud connection.
Also known as: edge inference, on-device inference
A camera that detects objects with a local model or a phone that recognizes speech on-device is doing edge inference. Processing data close to its source can improve response time and reduce the amount sent over a network. It can also help keep sensitive recordings local, depending on what the wider application stores and transmits. NVIDIA’s edge AI overview describes these benefits and how edge systems work with cloud services.
The model still has to fit the device’s memory, processing capacity, and power budget. Smaller models and quantization can help, with quality checked against the task. A hybrid system can handle urgent work locally and send heavier requests to the cloud; the right split depends on the application. In the meeting-room hardware episode below, Tormod Ree explains why time sensitivity and privacy shape that decision.
