Open research

How small can an app's AI be on a 4 GB phone, with no network?

We test a small open model, taught only one app's tasks, on an entry-level 4 GB phone. The method, the measurements and their limits are public, so anyone can check them and build on them.

Method

A per-app pipeline, tested on a real phone.

  1. Map the app

    Map the actions the app can take and the data behind them.

  2. Build a sandbox

    A copy of the app with fake data, so the model can practise every action safely.

  3. Freeze the exam first

    An exam from real traffic, frozen before training and kept apart from training data.

  4. Teach a small model

    A small open model learns from a larger teacher, with checks.

  5. Add code guards

    In our first build, bills, tax and stock are computed by code from the app's own data, not by the model; the AI drafts and the owner approves.

  6. Pass real-device gates

    Quantise the model and pass gates on a real low-end device.

The model is a small open model taught only the app's tasks. It starts from Qwen3-0.6B, an open base model under the Apache-2.0 licence, and is retrained only on one app's tasks with examples from a larger 8B teacher, then checked against the frozen exam. It runs with llama.cpp (MIT licence) on the phone's own CPU.

Results so far · first build

What we have measured, with the layer of each test.

Our first build is the offline edition of the Nesayo pharmacy software. FTL means a real phone in Firebase Test Lab; EVAL means a frozen exam scored on a server. Each row says what it does not show.

Model and speed bench

llama-bench on a Galaxy A05s. The bench did not control the network.

0.6BParameters FTL

Size only, not quality.

Q6_KPrecision / runtime FTL

6-bit Q6_K, llama.cpp, 4 CPU cores. No GPU, no NPU.

14.8 tok/sWriting speed FTL

11.0 tok/s at 768 tokens of context.

25 tok/sReading speed FTL

Same bench.

Bench conditions: Galaxy A05s SM-A057G, Snapdragon 680 'bengal', 4 GB, Android 15; llama.cpp 620d222 (armv8a); 4 threads; f16 KV, flash-attn 0, repack 0; llama-bench -r 3 (mean of 3); thermal_status 0, battery 31 °C at start.

Whole-app run, a separate run

Galaxy A05s, 4 GB, Snapdragon 680. A test build (31 Oct) of the whole app passed a real-phone run with airplane mode on. No later merged build has passed a whole-app run yet.

4.25 sStock answer, offline FTL

The first turn of that run took 13.4 s (right after a print preview); still open.

5.76 sBill from a sentence FTL

One run.

673 MBModel memory FTL

The model in memory during the run, not the whole app.

18.8 sFirst launch FTL

Cold start of one run.

Exam

Scored on a server, not on a phone.

+13 pts0.6B vs its 8B teacher EVAL

95% CI +4.0 to +24.4. An earlier version of our model (v7), not the current one (v18), scored on a server.

322 questionsExam size: 40 shops EVAL

16 of the 40 shops are QA-script traffic, not shop owners.

Full rows, run conditions and sources: benchmarks and evidence.

Working in the open

Open today, and what we plan to publish next.

Plans are not releases. Each one will appear here with its licence when it is out.

Public

Phone benchmark dataset

neykosa/phone-llamacpp-bench on Hugging Face (CC BY 4.0): benchmark rows with their standard deviations, run metadata, and a recipe for measuring a public Qwen3-0.6B file on your own phone. Your numbers describe the public model, not ours: our measured weights are private.

Open on Hugging Face →
Public

Benchmark write-up

The full run conditions, including the settings that did not ship.

Read the benchmarks →
Public

Evidence page

Every number on our site with its test layer and what it does not show.

Read the evidence →
Planned

Real-phone test gates

A write-up of what a build must pass on a low-end phone before anyone uses it.

Planned

Open eval kit

The exam format and scorer, with synthetic data, so others can freeze an exam for their own app.

Planned

Synthetic-data weights

Model weights trained only on synthetic data. Weights trained on real customer chats stay private, because that data belongs to the people who wrote it.

Also: Can a small language model run on a budget Android phone with 4 GB RAM? on dev.to.

Limits

What these results do not show.

  • Every phone result on this page (FTL) comes from one phone model, the Galaxy A05s. The exam results (EVAL) were scored on a server, not on a phone.
  • Energy and battery use have not been measured.
  • The offline edition is not yet in users' hands: these are lab results.
  • Results describe one app. Another app needs its own exam and its own measurements on its users' devices.
Take part

Check it, extend it, or use it.

The dataset card has a recipe for timing a public Qwen3-0.6B file on your own Android phone over adb, and a script that prints your row ready to paste. Send the row, or tell us where the recipe breaks, at hello@neykosa.com. We also welcome reviews of the method from people who research small models or on-device AI.

Building software whose data cannot go to a cloud AI? Our design-partner offer builds the small model for your app's tasks and tests it on your users' phones.