Edge and On-Device ML

Not all models live in data centres. The keyboard on your phone predicts words. Your camera finds faces. A watch detects falls. Running models directly on devices, brings speed, privacy, and offline use. But it also brings tough limits on size, power, and updates. Let's see how operations change at the edge.

• Models on phones and devices
• Speed, privacy, offline
• Tough size and power limits

Why run on the device?

There are good reasons to run on the device. Latency, no network trip, so answers are instant. Privacy, personal data never leaves the phone. Availability, it works without a connection. And cost, no server bill per prediction. But there are limits. Models must be small, and efficient, because memory, and battery are scarce. Updating a model means updating thousands, or millions, of devices. And you see less of what happens, because you cannot watch every device the way you watch a server.

Getting a model onto a device

The path to a device has a few steps. Train the model, as usual. Compress it, with quantization, pruning, or distillation, which you learned earlier. Convert it to a mobile format, such as T F Lite, Core M L, or O N N X Runtime Mobile. Test it on real devices, because speed varies enormously. Ship it, inside the app, or as a downloadable update. And monitor what you can, with anonymous, privacy safe metrics, like speed, and crash rates, sent back to you.

1. Train and compresssmaller and faster
2. ConvertTF Lite, Core ML, ONNX Mobile
3. Test on real devicesspeed varies widely
4. Ship and monitoranonymous metrics

Federated learning

There is a clever idea for learning from devices without collecting their data, federated learning. Each device trains the model a little, on its own data, and sends back only the update, the change to the weights, never the raw data. A server combines the updates from many devices into an improved model, and sends it back out. It protects privacy, but it is complex, with challenges in communication, uneven data, and security. It is used by some keyboards, and health apps, where data is especially sensitive.

Summary