On-device AI in 2026: what fits on a phone and what does not

"On-device AI" is doing a lot of marketing work at the moment, usually without anyone specifying what the device is actually doing. This post is the unglamorous version: what we can run on a phone today, what we cannot, and how that shaped the product.

The three constraints

Everything about local inference reduces to three limits, and they interact badly.

Memory. A model must fit in RAM alongside the operating system and everything else running. Phones ship with 6–12 GB total, and an app that claims too much gets killed. This is the binding constraint, and it is the reason phone models are small.

Compute. Modern chips include dedicated neural accelerators, which are genuinely fast, but far from a datacentre GPU. Throughput determines whether an answer feels instant or feels like waiting.

Power. Sustained inference generates heat, and a hot phone gets throttled. A feature that works beautifully in a demo can degrade badly when used for two minutes straight.

The uncomfortable consequence: you do not get to run a general-purpose model. You get to run a specific one.

What works well locally

The good news is that a narrow model can be excellent at a narrow job. These are the tasks where local inference is genuinely competitive:

Classification. Is this spam? Is it a scam? Is it work or personal? Small models are very good at this, and it is cheap enough to run on every incoming message.
Extraction. Pulling a date, an amount, a tracking number or an address out of text. Structured output from short input plays to a small model's strengths.
Semantic search. Embedding messages and comparing them. Embeddings are small, fast and can be computed incrementally as messages arrive.
Short summarisation. Condensing a bounded conversation. Quality holds up well when the input fits comfortably in context.

Every one of these shares a shape: short input, narrow output, clear success criterion.

What does not work well

Being straight about the limits is more useful than pretending they are not there.

Long-context reasoning. Summarising two years of a conversation is not a phone task today. We work around it by summarising incrementally rather than in one pass.
Open-ended world knowledge. A small model does not reliably know facts about the world. Textly deliberately never answers questions that are not grounded in your own messages, partly for privacy and mostly because it would be wrong.
Long-form generation. Suggested replies are short by design. That is a constraint dressed as a product decision, and we would rather admit it.
Anything on very old hardware. Below a certain chipset generation, the experience is bad enough that shipping it would be worse than not having it.

This is why our minimum requirements are higher than a plain SMS app's. We would rather exclude devices than ship an experience that makes local inference look bad. It is a real cost, and we are not happy about it.

How this shaped the product

Once you accept the constraints, the design follows from them.

Many small jobs, not one big one

Textly does not run "an AI". It runs several small specialised models: a classifier for spam and scams, an embedding model for search, an extractor for structured items, and a compact generative model for summaries and drafts. Each is sized for its job.

This is more work to build than calling one large model with different prompts. It is also the only way it fits.

Do the work when nothing is waiting

Classification and embedding happen as messages arrive, in the background, while you are not looking. By the time you open the app the work is done. Only summaries and drafts run on demand, because they cannot be precomputed.

Fail visibly, never silently

When the model is uncertain, Textly says so rather than guessing confidently. An extracted appointment with a low-confidence time is shown as needing confirmation. A summary that dropped something is easier to forgive when the source is one tap away, so every generated item links back to the message it came from.

Where this is heading

The trend lines are unusually clear. Phone neural accelerators are improving quickly, memory budgets are growing, and small models keep getting better at specific tasks, often faster than large ones improve at the same tasks.

The interesting consequence is not that phones will run frontier models. It is that the set of jobs where cloud inference is necessary keeps shrinking. For a messaging app in 2026, that set is already empty.