An AI That Runs Inside the Phone: Building PocketAnima

We built PocketAnima for three reasons: privacy, learning and proof. As a team that adds AI to client products, we wanted to know how these capabilities actually behave in production code not from a slide but from a product we built with our own hands. There is a big difference between describing a capability and carrying it in a product that runs every day; the second teaches details the first can never see.

Privacy was at the centre of the product. We made the decision from the start: let the AI conversation run on the user's phone, inside the device; let no message pass through our servers. This was not an option added later but a sentence about what the product is — and for that sentence to be true, the whole architecture had to be built around it. Below we walk step by step through the technical decisions we made building this, where we struggled, and what we carried into client projects.

A language model fits on a phone by being compressed and chosen to match the device's memory. A large model's weights do not go into a phone as they are; so the model is reduced to a smaller form, giving up some of its accuracy. PocketAnima runs models on the phone's processor with an open format (GGUF) and a mature runtime library (llama.cpp). The user downloads the model; from then on the conversation runs on the device without ever going to the network.

The honest equation here is this: the model that fits on a phone is less capable than the large one in a data centre. This is not a flaw but a physical reality — memory is limited, the model shrinks, some capability is left behind. But for most everyday conversations this model is more than enough, and in return it gives speed, privacy and offline operation. Rather than hide this cost, we chose to give the user a choice: if the small model does the job, stay on the device; if you need more power, switch to the cloud.

In an on-device app the user chooses the model, and that alone is an infrastructure problem. Models are not small files; a phone's memory is not infinite either. So we offered an in-app model list: it shows in advance whether each model fits the device, resumes a download where it left off, activates the downloaded model and leaves deleting an old one to the user's control. Making the 'download, run, delete if needed' flow smooth is, in an on-device app, as important as the chat itself.

The lesson we learned building this flow was that it is better to confront the user honestly with their device's limits. Downloading a model and only then finding it does not fit the device is a bad experience; so we put a marker before the download — 'this fits your device comfortably' or 'this model may be large for your device'. Protecting the user from a surprise meant showing them a warning. In an enterprise product the same principle holds: telling the user the system's limit in advance is better than apologising afterwards.

In PocketAnima the engine sits behind a single interface: the local model on the device and a cloud provider the user connects with their own key both speak the same interface. The default is the fully offline device model; if the user wants, they add their own cloud key and switch to a more powerful model. If no key is ever added, the app works in full — the cloud is not a requirement but the user's informed choice.

We saw the value of this abstraction again and again while developing. Adding a new provider came down to writing a single adapter without touching the rest of the app; the engine changed but the interface, the chat history and the features stayed the same. This is in fact a general principle of our product family — when the source changes, only the adapter changes and the core stays untouched — and it works especially well in AI, because this field keeps changing beneath us. We set up the same pattern in client projects: we do not embed an AI provider into the core, we add it as an adapter.

In PocketAnima you can upload a document and ask questions over it; the answer also shows which source it came from. Running this on the device is harder than in the cloud, because you have to solve two separate problems: breaking the document into meaningful pieces and finding, among thousands of pieces, the one that answers the question. We solved the first with chunkers for text, code and PDF, and the second with a retrieval layer — all on the device, with the uploaded document going to no server.

The lesson we learned in retrieval was that a single method is not enough. Keyword search alone misses a piece that has no word overlap but is semantically relevant; semantic search alone sometimes stays too general. So we built a hybrid retrieval that combines the two: computing both keyword and meaning-based similarity and blending the results. We also built in showing the source of the answer from the start, because the user must be able to ask 'where did you find this'. In an enterprise RAG project we carry the same two lessons: retrieval quality matters more than the model, and the source must be visible.

In PocketAnima you can speak alongside typing and listen to the answer aloud; both turning speech into text and text into speech run on the device. Bringing this on-device means microphone data is not sent to a server and the audio recording never leaves the device. This is the often-ignored privacy dimension of a voice assistant: your voice is at least as personal as your text, and the on-device approach keeps it inside too.

The cost of solving voice on the device is living with the library's limits. Voice models running on the device may not be as polished as the large ones running on a server; pronunciation does not always come out studio-grade. But the function is solid and, most importantly, offline. The design decision here was to position voice as a convenience, not a show: asking while your hands are busy, listening to the answer instead of reading it. Voice did not replace chat; it added an alternative input-output path to it.

In PocketAnima you can chat with built-in characters or one you create yourself; each character has a system prompt and a tone. The interesting thing is this: the same model behaves differently with a different character. When you tell one character 'answer short and technical' and another 'be warm and explanatory', the model underneath is the same but the output changes markedly. This is a practical example of how powerful it is to steer a model's behaviour without changing the model itself.

This is a lever often skipped when adding AI. Most products look for a bigger or different model to change behaviour; yet often it is enough to give the same model a well-written system prompt. In PocketAnima we put this in the user's hands — they set the character. In an enterprise project we design that prompt ourselves: we decide what the assistant will do, what it will not, and which tone it uses, with a system prompt, without changing the model.

We dealt with device diversity by building a base that runs properly even on the weakest reasonable device, and progressively unlocking the capacity of stronger ones. The phone world is not uniform: processor, memory and graphics capability vary from device to device. So we first built the app on a solid CPU baseline — so it runs reliably on every device — and treated acceleration as a layer on top of that. Accelerating the model on the phone's graphics processor or a dedicated AI chip is a topic we advance on the roadmap, since it is not equally possible on every device.

The lesson here was that in an on-device product you must design for the 'most common case', not the 'best case'. If a feature works only on the strongest phones, it does not work for most users. So we built every new feature with the question 'how do we do this on a limited resource, with no internet'; if acceleration is there it is a bonus, if not the baseline is still solid. This discipline holds in enterprise software too: aim at the condition most common in the field, not the most impressive demo.

Building a rich app in the right order is better than trying to do everything at once. So we first built a reliable on-device chat base, then added document chat, voice and characters on top of it, layer by layer. PocketAnima is a product that keeps growing; the same core reaching more platforms, more hardware acceleration and new capabilities being added are natural parts of the roadmap. Seeing a product as 'maturing' rather than 'finished' lets us build each layer properly.

The principle that set the order was this: do not move to the next layer until one is solid enough to place another on top of it. Adding voice was meaningless before the chat base was settled; document chat would have been fragile before running the model was reliable. This is an approach we defend in enterprise projects too — aiming for a rich release but building it in the right order, a solid base first and then layers. Rushing to turn everything on at once leaves, in the end, a product where none of it is reliable.

The biggest thing we learned was that on-device AI is not a 'language model' matter but an engineering matter. Running the model was perhaps the easiest part; the real work was downloading, updating, storage, working across different devices, retrieval quality and an honest interface. This holds when adding AI to any product: the model is one component, the product is everything around it.

We carry this experience directly into client projects. The on-device, privacy-first approach is a strong answer for sectors where data must not leave — healthcare, law, the public sector, organisations that want full control over their own data — and we proved it not on a slide but in a working product. Adapter architecture, hybrid retrieval, human-approved decision support and an honest interface are all patterns tried in PocketAnima and then carried into enterprise work. A team that uses its own product learns its lessons in a few hours; a team that does not, only when a customer complains.

PocketAnima taught us that on-device AI is not a marketing line but an engineering discipline from end to end. From fitting the model on the phone to managing downloads, from solving documents and voice on the device to dealing with device diversity, every part was designed with the question 'how do we do this on a limited resource, with no internet, without data leaving'. What we gained was offline operation, privacy and control; what we paid was openly accepting the model's limits and building each layer in the right order.

We use these lessons both in growing PocketAnima and in client projects. If you are considering an on-device or privacy-first AI application, an in-house assistant or a solution that works with your own data, we would be glad to share our experience. At Vosetu we do not just describe AI; we run it in a product, inside the device.