Mobile AI is different

I work on AI models that run on mobile devices. There are few people or companies working on this, even in my circle. When I talk with other engineers working on model or inference performance, I realize that we share some concerns, but not all of them. Here are a few important concerns for models that run on-device: Raw performance (single-request latency), of course. Paradoxically users are even less willing to wait on mobile form factors so you need lower execution times. Weights size. Unless you are a manufacturer or an OEM, end users will need to download your model to their phone. If it is too large adoption will plummet. Memory usage. RAM is expensive, so the amount of memory available in phones is going down, e.g. the Pixel 11 Pro has less RAM than the Pixel 10 Pro. Moreover, even if a phone has 8 GB of RAM, you cannot use more than e.g. 2 GB reliably. If you use too much, you will stress the system and end up getting killed. Heterogeneous hardware. Of course Android is…

Commenting on Bubbles has been removed. Why? If you want to talk about this post, write to the author, use the comments on their blog if it has them, or reply with a "RE:" post on your own blog. Suggestions? feedback@bubbles.town