On-device AI / ML

The build behind this entry's react-native-rag / local-RAG option rows, done by hand so every constraint is visible: a WhatsApp export chunked, embedded and searched entirely on a $100 Android phone in airplane mode. The stack is one engine and one store — @react-native-ai/llama (llama.cpp) runs BOTH the embedder and the generator, and react-native-nitro-sqlite statically links sqlite-vec so KNN is a vec0 virtual table queried with `WHERE embedding MATCH ? AND k = ?`, no second ML runtime and nothing fetched at runtime. What makes it durable is the failure list. MEMORY IS THE CEILING, not accuracy: ~900MB usable RAM holds Qwen2.5-0.5B-Instruct Q4 (469MB) beside the 119MB embedder, and Qwen2.5-1.5B (1.07GB) hard-crashed the process with an uncatchable native OOM that JS cannot trap. The context window is paid for in KV-cache RAM, so it is capped at 4,096 tokens and a 21,000-token export is REJECTED outright ('Context is full'), not truncated — which is the whole reason retrieval exists here. Query and documents must be embedded by the same model with the same normalization, or search returns noise (the author calls this the number-one RAG pitfall). The multilingual embedder (paraphrase-multilingual-MiniLM-L12-v2, 384 dims, ~119MB at Q4) is worth its size the moment chats aren't pure English — all-MiniLM-L6 collapsed non-English messages to nearly identical vectors. Chunk boundaries are part of the data model: strip the speaker or timestamp and the model hallucinates them back. And a small model needs an explicit 'say you don't know' escape plus replayed recent turns, or follow-up questions retrieve the wrong thing entirely.

A reasoned cost/latency/privacy case for on-device TTS (Kokoro via react-native-executorch), with the economics and the tradeoffs (voice/language coverage, a custom C++ phonemizer). A durable framing for the on-device-vs-cloud decision.

Survey evidence that on-device AI is production-proven, not experimental: Apple ships multimodal-embedding photo search + hybrid Siri; Meta runs ExecuTorch in WhatsApp (network prediction), Instagram (SqueezeSAM cutouts) and Messenger (transcription); Google ships scam detection + Gemini Nano; Snapchat runs 10MB AR models. Ends in RN ExecuTorch pointers (Gemma/Qwen, ~30MB quantized Whisper Tiny STT) — vendor self-promotion, they author the lib. The durable framing: 'fine-tuning is what makes a small model good enough.'

The end-to-end build this entry's option rows only imply: a fully on-device recognition pipeline — detection → tracking → alignment → embedding → matching → liveness — on VisionCamera 5 frame processors, with ONNX Runtime (CPU/XNNPACK) for inference and Reanimated shared values bridging worklet and JS. Two things make it durable knowledge rather than a tutorial. First, the architectural reason it can work at all: the frame processor gets each YUV frame synchronously on a dedicated native thread as a worklet, with no bridge serialization and no pixel copy into JS. Second, real numbers on a Galaxy S21 Ultra — YuNet detection (~230KB model) 3–6ms/frame at 128×128, SFace embedding (37MB, 19MB int8) ~29ms, ~39ms end-to-end per detection cycle, gallery matching in microseconds for 512 entries — plus the budget trick of running detection every 10th frame and letting tracking bridge the gaps. Uses ONNX Runtime rather than this entry's ExecuTorch default, which is itself a useful data point.