Most of the AI industry is racing toward bigger models. This week, a small team called Cactus Compute went the other direction — and Hacker News noticed. Their new release, Needle, is a 26-million-parameter model that handles tool calling (the skill that lets an AI agent actually do things, not just talk) and runs entirely on a phone, smartwatch, or other low-power device. No cloud, no API key, no per-call fee. The Show HN post landed on the Hacker News front page with hundreds of upvotes and a lively technical discussion — a reminder that "smaller and smarter" is becoming as newsworthy as "bigger and better" in AI right now.
Needle is distilled from Google's Gemini 3.1 Flash Lite, but it isn't a shrunk-down general chatbot. It's purpose-built for one job: tool calling — matching a user's request to the right function, pulling out the right arguments, and returning clean JSON so an app or agent can execute it. Cactus Compute's insight was that this task is closer to retrieval-and-assembly than open-ended reasoning, so the model doesn't need the huge feed-forward layers that make LLMs expensive to run. Their architecture — a "Simple Attention Network" with the MLP layers stripped out entirely — gets the job done with a fraction of the parameters.
The numbers back it up. Quantized to INT4, Needle's footprint is about 14 MB. It runs at roughly 6,000 tokens/second on prefill and 1,200 tokens/second on decode on consumer hardware — fast enough to feel instant on a phone. In head-to-head tests on single-shot function calling, Needle reportedly outperforms other small models like FunctionGemma-270M, Qwen-0.6B, and Granite-350M, all of which are 10x or more its size.
The practical implication is bigger than the benchmark numbers: it's now realistic to ship an AI agent that lives entirely on a user's device, with no round trip to a cloud API. For a smart-home app, a fitness wearable, an offline-first tool, or anything where latency, privacy, or connectivity is a concern, that's a meaningful shift. It also removes the recurring cost problem — once the model is on the device, there's no per-call bill, because there's no API to call.
Needle is released under the MIT license, which is about as permissive as it gets: commercial use, modification, fine-tuning, and redistribution are all fair game. The weights are on Hugging Face, the full training and data-generation pipeline is open on GitHub, and Cactus Compute even shipped a local playground where developers can fine-tune the model on their own tool schemas through a web UI — no registration or API keys required to get started.
It's also a useful data point on where training costs are heading. Cactus Compute says pretraining ran for 27 hours on 16 TPU v6e chips across 200 billion tokens, while the tool-calling fine-tune — using a 2-billion-token synthetic dataset generated by Gemini across 15 categories like timers, messaging, navigation, and smart-home control — took just 45 minutes. That's a training bill within reach of a small team, not just a frontier lab.
Getting hands-on with Needle doesn't require much setup:
This isn't a replacement for a general-purpose assistant model — it's a specialist. If you're building or advising on AI agents for mobile apps, wearables, IoT, or any offline-capable product, Needle is worth a look as a lightweight tool-calling layer that doesn't require negotiating API rate limits or per-token pricing. For AI agencies and automation builders evaluating where on-device inference is heading, it's also a useful signal: the gap between "runs in a datacenter" and "runs in your pocket" for agentic AI is closing faster than most roadmaps assumed a year ago.
RAG assistants on your documents, AI agents that act in your tools, and LLM features inside your product.
Our cutting-edge features simplify collaboration and creativity, making your workflow intuitive and efficient. Transform your vision into reality effortlessly with Hadidiz Flow.
