Where Should Your AI Run: In the Cloud or On the Device?

Where AI runs is not a feature toggle; it is a decision about the foundation of the product. When a chat, a search or a summarisation request arrives, either a distant server does the work or the device in the user's hand does. Both paths work — but which one you pick decides your cost structure, your privacy promise, the app's speed and who you depend on, all at once. That is why it is not something to defer with 'we'll sort it out later', but a choice made deliberately at the start.

Most teams start in the cloud, because that is the easy path: the most capable models are one call away and you do not have to build the infrastructure. It goes well for a while. Then cost accumulates on the monthly bill, a customer asks 'where does our data sit', another wants to use the app with no internet. At that point the sentence 'make it work on device too' lands on the table, and it is no longer a simple addition: which engine is the default, how the two coexist, how the decision ripples through every layer of the product. We are writing this as a team that started from the other end of that question — the device.

PocketAnima does exactly this: it runs the AI conversation on the user's phone, inside the device. Out of the box it needs no internet at all; no message ever passes through our servers. Below we will first put the 'where should it run' decision into a framework across five criteria, and then walk through how we lived each part of that framework while building PocketAnima — what we gained, what we gave up and what we deliberately left unfinished.

Five criteria decide it together, and none is sufficient alone: cost (the cloud's per-use charge, the device's storage and battery load), privacy (where data sits, who can reach it), speed (the round trip to a remote server versus running offline), model power (the gap between a large model in a data centre and a small one that fits on a phone) and control (how dependent you stay on a provider). The right answer depends on how these five are weighted for your product.

The trap here is to elevate one criterion and forget the rest. 'Privacy matters, so put everything on the device' sounds clean; but the model that fits on a device is weaker, and if you did not account for that, the decision backfires. 'Power matters, so always cloud' likewise ignores the bill and the dependency. A good decision puts all five on the same table, decides knowingly how much to trade on each, and keeps that decision consistent over the life of the product.

Cloud cost grows with usage; device cost is largely one-time. In the cloud everything looks cheap on day one: a few requests, a small bill. But every conversation, every query is a line of expense, and as the number of users grows that line grows with it. A growth that should make you happy comes back at month-end as an unexpected bill. In the cloud, scale grows the expense alongside the revenue.

On the device the equation is different. The upfront cost is higher — preparing the model to fit on a phone, managing downloads and updates, making sure it runs across different devices is not easy. But once that work is done, the user's device carries out the computation; your server does not meter a charge for every conversation. A thousand users and a hundred thousand users cost you the same in running terms. In PocketAnima this was a deliberate choice: however much the chatting, we wanted every message to cost us nothing to run.

Privacy is not a technical setting but a promise; and that promise is kept or broken by where the data physically sits. In the cloud, data lives on a server far from the user's city; who reaches it, how long it is kept and whether it is truly deleted when required now depends on your policy and your provider's. When personal or sensitive corporate data is involved, this is not a technical detail but a legal responsibility: which country's law, which network, which audit trail.

A model that runs on the device changes this question at the root: data stays on the phone, goes to no server, so the answer to 'how do you store it' becomes 'we never hold it at all'. This is where PocketAnima's default privacy comes from. But let us be honest: on-device does not eliminate privacy risk, it relocates it. The data is now the device's responsibility, not the network's; if the phone is lost or seized, the protecting layer becomes the device's lock. So on-device is a strong answer for organisations that will not let data leave — healthcare, law, the public sector, anyone who wants full control over their own data — but it is not magic.

However fast the cloud is, it carries an unavoidable round-trip delay; the device removes that delay. A request travelling to a remote server and the answer coming back adds the network's own share, independent of how fast the processor is. Small delays go unnoticed one by one, but stacked across a conversation the interface starts to feel heavy. On the device, because the work finishes in place, there is no round trip; the answer comes at the same speed whatever the network is doing.

The real difference shows up when there is no network at all. On a plane, in a tunnel, in a dead-zone, a cloud-based AI quietly stops; the on-device app opens and works. Trying PocketAnima, we saw this concretely: being able to put the phone in airplane mode and keep chatting turns 'works offline' from a promise into a demonstrable fact. This is an advantage the cloud cannot give by its very structure — but its price is hidden in the next criterion: for speed, you give something up on the model.

A model that fits on a device gives up capability: the model that fits on a phone is less powerful than the large one in a data centre. This is not a design flaw, it is a physical reality. A model's power is largely tied to its size, and a phone's memory is limited; shrinking a large model to fit there means leaving some capability behind. On complex, multi-step reasoning the gap widens; on short and direct tasks the small model is usually more than enough.

We think it matters to say this plainly, because some products that market on-device AI ignore this cost. Building PocketAnima, our stance was not to hide it but to ask: if the small model does your job, stay on the device; if you need more power, switch to the cloud from the same app. On-device models are also improving fast; work that once only large models could handle can today be done by far smaller ones. But 'enough for most tasks' does not mean 'enough for every task', and hiding that distinction from the user would be wrong.

Vendor dependency is a hidden cost, because once you tie yourself to a cloud provider, the model's behaviour, price and continuity are no longer fully in your hands. If the provider updates the model, your product's answers change; if it raises prices, your budget grows without you doing anything; if it closes an endpoint, you rewrite your pipeline. Because every provider's interface is its own, the decision to 'move to another' is usually more expensive than it looks.

Our answer to this was to tie the decision not to a single provider but to an abstraction. In PocketAnima the engine sits behind one interface: the local model on the device and a cloud provider the user connects with their own key both speak the same interface. If the provider changes, only that adapter changes, and the rest of the app stays put. This is in fact a general principle of our product family: when the source changes, only the adapter changes and the core is untouched. Making on-device the default takes that independence to its far end — because in the default state there is no one you depend on.

We built PocketAnima to run on the device because the product's promise was privacy from the start: we wanted the conversation to live on the user's device and leave no trace with us. We saw this not as a feature but as a sentence about what the product is — 'your conversations are entirely on your device; they never reach our servers'. For that sentence to be true, the whole architecture had to be built around it, not bolted on later as an option.

This choice also had a technical face. Had we run it from the cloud, we would have had to meter how much each user consumes, bill it and manage keys; a model running on the device removes that burden entirely. In return we accepted a different engineering burden: running the model on the phone, downloading and managing the model the user picks, making it reliable within limited memory and battery. On-device AI, past a certain point, stops being 'a language model' and becomes an infrastructure problem — downloading, updating, storage, working across different devices.

We discussed the biggest limit — model power — above; the second limit is device diversity. The phone world is not uniform: processor, memory and graphics capability vary from device to device. So you design an on-device app to run properly even on the weakest reasonable device, then progressively unlock the extra capacity of stronger ones. Accelerating the model on the phone's graphics processor or a dedicated AI chip (NPU) is one of those steps; since it is not equally possible on every device, the right order is to build a solid CPU baseline first and advance acceleration on the roadmap.

The third limit is storage and distribution. Models are not small files; if a user wants to keep several, the device's memory starts to speak up. So we made downloads resumable, told the user in advance whether a model will fit, and left deleting an old one to the user's control. The fourth is not speed but maturity: in the cloud you can use a new model the day it appears, whereas on the device you wait for a verified version of that model that runs on a phone. None of these disproves the on-device idea; they are simply the things a team that takes it seriously has to design for from the start.

The best answer is often 'both', because the real decision is not between cloud and device but over who gets to decide. In PocketAnima we set it up like this: the default is the fully offline, on-device model; if the user wants, they add their own cloud key and switch to a more powerful cloud. If no key is ever added, the app works in full — the cloud is an option, not a requirement. This way we do not resolve the 'privacy or power' dilemma on the user's behalf; we give them the informed right to choose.

What makes this possible is the engine abstraction we mentioned at the start: the interface, the features and the flow stay the same, and the engine underneath changes. The user can say 'I'm working with a sensitive document today, let me stay on the device'; or 'I want the strongest answer for this task, let me switch to the cloud'. In both cases the same app runs with a different brain. The most honest answer a product can give is often not to force a single path, but to make the path visible and reversible.

We solved these heavy tasks with the same principle: moving the work to the device and designing for the limited resource. Take document chat — the user uploads a text, a code file or a PDF, asks questions, and sees which source the answer came from. Doing this on the device is harder than in the cloud, because you have to break the document into meaningful pieces and find the right one. We built the solution as a hybrid: each piece is prepared for both keyword search and meaning-based similarity; when a question arrives the two are combined and the most relevant pieces are handed to the model. All on the device, with no internet, and showing the source of the answer.

Voice is solved with the same approach. In PocketAnima you can speak alongside typing and listen to the answer aloud; both turning speech into text and text into speech run on the device, not in the cloud. That means microphone data is not sent to a server and the audio recording never leaves the device; the privacy promise is not broken here either. On-device voice works within the limits of the library — pronunciation may not always be studio-grade, but the function is solid and, most importantly, offline. The general lesson here is this: once you choose on-device, you design every new feature with the question 'how do we do this on a limited resource, with no internet'.

The decision for your own product becomes clear by answering five questions in order. First, data: is the user data personal or sensitive corporate data, or something with no harm in leaving the device? If personal data dominates, the scale tips toward the device. Second, scale: as the user count grows, can the cloud bill be carried, or do you need to fix the running cost? Third, speed and offline: is working without internet a requirement, or a nicety?

Fourth, power: does your task genuinely require a large model's reasoning, or is the small model's answer enough? If it requires it, keep at least the path to the cloud open. Fifth, independence: is staying tied to a single provider a risk for you? For most products the answer to these five is not 'pure cloud' or 'pure device' but a hybrid that unites both behind an abstraction. What matters is making the decision not by intuition over a coffee break but by putting these five axes on the table — and staying consistent once you have.

There is no single right answer to where AI should run; the right answer changes with your product's balance of cost, privacy, speed, power and control. But there is one wrong thing: never making the decision, or making it and then staying inconsistent. The privacy promise you give the customer must match the data you keep on the server; the 'works offline' you say in marketing must match what actually happens. Architecture matters precisely because it secures that match.

Building PocketAnima, we started directly from the device instead of starting in the cloud and moving later; that won us offline operation, privacy and control, and in return brought openly accepting the model's limits and building the hybrid system. We write this not because marketing prompts it but because we actually lived it. If your product is at the threshold of this decision too — if your use cases are not yet clear or you are caught between cloud and device — we would be glad to share our experience. At Vosetu, alongside building PocketAnima, we also help other teams set up their AI architecture on the right foundation.