Edge AI means running a trained AI model on or near the device that produces the data, such as a camera, sensor, robot or gateway, instead of sending the data to a remote data center for every decision. Models are usually trained in the cloud or a data center and then deployed to edge hardware, which gives lower latency, less network traffic and the ability to keep working when the connection drops (IBM).
We make HyperLake, the sovereign AI operating platform. Most edge AI questions do not need it, and this page says where it does and does not fit.
What is edge AI, and how does it differ from cloud AI?
In cloud AI, the device sends data to a data center, the model runs there, and the result comes back. In edge AI, inference (running the model to get a prediction) happens locally. IBM describes the two as complementary: edge AI handles time-sensitive work on the spot, while the cloud offers more storage and processing power for training advanced models (IBM).
| Question | Edge | Cloud |
|---|---|---|
| Where does inference run? | On the device, a gateway or a nearby server | In a data center |
| What happens if the network drops? | Local decisions continue | Requests fail or queue |
| What data leaves the site? | Only what you choose to send | Usually the raw input |
| Model size | Limited by device memory and power | Limited mainly by budget |
| Updating models | Must be rolled out to every device | Update once, centrally |
Most real systems are hybrid. The edge makes the fast local decision; the cloud or a central data platform handles training, fleet management and analysis across sites. For a longer treatment of that split, see Local AI vs Cloud AI.
What does an edge AI architecture look like?
An edge AI system has three tiers, plus a loop that connects them.
- Device tier. Sensors, cameras, machines or phones that generate data. Small models may run directly here, including on microcontrollers.
- Near-edge tier. A gateway or edge server at the site that aggregates several devices and runs larger models. IBM lists edge devices, edge gateways, edge servers, specialized processors and the models themselves as the key components (IBM).
- Central tier. A cloud or data center that stores selected data, trains and evaluates models, and manages the fleet.
The loop: devices run inference locally, send back summaries, exceptions or sampled data, the central tier retrains the model, and a new version is rolled out. IBM describes the same pattern: when the model hits a problem, data goes back for further training and an improved model replaces the edge version.
sensor -> device inference -> local action
|
v (summaries, exceptions, samples)
gateway / edge server -> central data platform
|
train + evaluate + approve
|
signed model package -> fleetA small, hypothetical routing policy shows how teams decide what stays local. The file format is illustrative, not any product's syntax:
# Hypothetical edge routing policy (illustrative only)
model: defect-detector
version: 14
run_at: device # inference stays local
on_low_confidence: send_sample_to_central
upload:
raw_video: never
detections: always
samples: when_confidence_below_0.6
rollout: canary_5_percent_then_fleetWhich hardware runs edge AI: processors and devices?
Edge AI hardware falls into a few groups. CPUs handle small models and control logic. GPUs and NPUs (neural processing units) accelerate neural networks. IBM names NPUs, GPUs and AI accelerators as the specialized processors used at the edge (IBM). Pick by the model you need to run, the power budget and the software support, not by headline TOPS (trillions of operations per second) alone, since TOPS figures are not directly comparable across vendors.
As of October 2026, from each vendor's own public pages:
- NVIDIA Jetson. A family of compact edge AI computers supported by the JetPack SDK, including Orin modules and the Jetson AGX Thor series, which NVIDIA describes as "Blackwell Jetson for high-end physical AI at the edge". NVIDIA lists the AGX Orin series at up to 275 TOPS and 15 to 60 W, and the Orin NX series at up to 157 TOPS and 10 to 25 W. Best for: robotics, autonomous machines and edge applications that need a compact GPU-based computer, with module size chosen by power and performance needs (NVIDIA).
- Hailo accelerators. Hailo lists the Hailo-8 at 26 TOPS and the entry-level Hailo-8L at up to 13 TOPS for edge inference, and the Hailo-10H for generative AI inference, in chip and M.2 module form (Hailo). Best for: adding a power-efficient accelerator to an existing board or PC with an M.2 slot.
- Raspberry Pi AI HAT+. An add-on board for the Raspberry Pi 5 with a Hailo NPU, offered at 13 TOPS (Hailo-8L) and 26 TOPS (Hailo-8); a newer AI HAT+ 2 uses a Hailo-10H and supports local LLMs and VLMs (Raspberry Pi). Best for: adding AI acceleration to a Raspberry Pi 5.
HyperLake does not run on edge devices or make edge hardware; for on-device compute, look at vendors such as NVIDIA or Hailo. HyperLake does not make edge hardware.
Which edge AI platforms and runtimes manage the software?
Hardware needs a runtime to execute models and a way to deploy and update software across many devices.
- LiteRT (built on TensorFlow Lite). Google describes it as a cross-platform inference runtime for deploying models on edge devices, covering Android, iOS and macOS, web, desktop and embedded or IoT targets, with CPU, GPU and NPU backends. Models from PyTorch, TensorFlow and JAX can be exported to the .tflite format (Google). Best for: shipping a model inside a mobile or embedded application.
- AWS IoT Greengrass. An open source edge runtime and cloud service for building, deploying and managing IoT applications on devices, including running machine learning predictions locally. Software is packaged as components and run as containers, Lambda functions or native processes (AWS). Best for: fleets already connected to AWS IoT services. AWS ended support for Greengrass Version 1 on October 7, 2026, so new projects should use Version 2.
- Azure IoT Edge. A device-focused runtime that runs containerized Linux workloads, made of modules, a runtime on each device and a cloud interface for remote management, as a feature of Azure IoT Hub (Microsoft). Best for: teams standardized on Azure who want cloud-managed containers at the edge. Check the support dates for your IoT Edge version; Microsoft lists 1.6 LTS as the supported release as of April 2026.
A minimal example of running a converted model with the LiteRT Python package (pip install ai-edge-litert):
import numpy as np
from ai_edge_litert.interpreter import Interpreter
interp = Interpreter(model_path="model.tflite")
interp.allocate_tensors()
inp = interp.get_input_details()[0]
out = interp.get_output_details()[0]
x = np.zeros(inp["shape"], dtype=inp["dtype"]) # replace with real sensor data
interp.set_tensor(inp["index"], x)
interp.invoke()
print(interp.get_tensor(out["index"]))Edge models are usually shrunk first. Quantization stores weights at lower precision to cut memory and compute; see Quantization Explained.
When should you use edge AI instead of cloud AI?
Choose edge when one or more of these is true:
- A decision must happen faster than a network round trip allows, such as a robot avoiding an obstacle.
- The site has unreliable or no connectivity.
- Raw data is too large or too sensitive to send out. IBM notes that keeping data local can strengthen privacy and make it easier to stay within jurisdictions.
- Many devices produce data continuously and only the exceptions matter.
Choose cloud when the model is too large for the device, when requests are occasional, or when you want one central place to update. Costs also differ: edge shifts spend to hardware and fleet operations, cloud to usage and bandwidth. Measure your own workload before deciding.
Edge brings its own problems. IBM notes that scaling edge AI across many locations raises data gravity, heterogeneity, scale and resource constraint issues. Different devices mean different runtimes, and every model update has to reach each one safely.
Where does HyperLake fit in edge AI?
Mostly it does not. If you are choosing a camera module, a board or an on-device runtime, use the vendors above.
HyperLake is relevant to the central tier. Per its public site, it is a platform for assembling, deploying and governing the data, models, applications and tools behind physical AI and enterprise agents in infrastructure the customer controls. Its components are open technologies: Kubernetes as the foundation, KServe for serving open models, Kafka for streams, Iceberg with object storage and Trino for lakehouse analytics, Qdrant or Milvus for vectors, and OAuth/OIDC, JWT identity and OPA for access policy. It runs on AWS, Azure, Google Cloud, OVH, private cloud and on premises.
Concretely, that means the place where edge devices send their data, where fine-tuning and evaluation happen, and where approved models are served from, under your own access and audit rules. The physical AI use case is described on the physical AI page and in the sovereign physical AI reference architecture on Kubernetes. Running HyperLake on the edge devices themselves is not something we claim here.
Frequently asked questions
What is edge AI in simple terms?
It is AI that makes its predictions on the device or a nearby machine rather than in a remote data center. The model is still usually trained in the cloud.
What is the difference between edge AI and cloud AI?
Edge AI runs inference locally, which cuts latency and lets it work offline. Cloud AI runs in data centers, with more compute and central updating. Most production systems use both.
What are examples of edge AI devices?
NVIDIA Jetson modules, Hailo accelerators, the Raspberry Pi AI HAT+, smartphones with NPUs, factory cameras, robots and IoT gateways.
Do I need a GPU for edge AI?
Not always. Small models run on CPUs or microcontrollers. Larger vision and language models usually need a GPU, NPU or accelerator.
Can large language models run at the edge?
Smaller, quantized models can. Hailo markets the Hailo-10H for generative AI inference, and Raspberry Pi says the AI HAT+ 2 supports local LLMs. Memory is the main limit.
Is edge AI more private than cloud AI?
It can be, because raw data can stay on site. IBM lists stronger privacy among its benefits. It does not by itself make a system compliant.
