A 5.9 GB AI model could make private assistants feel ordinary

A model that once required a data center could soon look less like a remote service and more like an application. It could sit beside a photo editor, a notes app or a coding environment, responding without an account, a subscription or a round trip to a cloud server.

TechCrunch reported that PrismML has released Bonsai 2 27B, a compressed reasoning model that the company says retains almost all of the performance of its larger reference model. The claim goes to the heart of the local AI movement: if a model can become small enough to run on consumer hardware without losing its most useful abilities, private and offline AI could move from an enthusiast project into a standard product feature.

The important question is not simply whether the model fits. It is whether the intelligence that fits in the device remains dependable when users take it outside a benchmark.

A large model in a small package

In PrismML’s official announcement, the company describes Bonsai 2 27B as its most capable model yet. The model is based on Qwen3.8 27B and uses ternary weights, a format intended to represent model parameters with far less memory than conventional approaches.

PrismML says the result fits in 5.9 GB and reduces memory requirements by more than nine times. It also reports that Bonsai 2 retains 98.2% of the reference model’s aggregate benchmark performance. Those figures are striking because memory, rather than raw processing power alone, is one of the main barriers to running sophisticated language models on personal devices.

The practical effect could be significant. A laptop with limited graphics memory might run a capable assistant locally instead of sending every prompt to a paid API. A phone could summarize a private conversation or search through personal files without uploading them. A small business could deploy an internal coding or research assistant without building a large cloud budget around usage fees.

Ternary compression is the central technical idea. Instead of storing weights with the higher precision typically associated with larger models, Bonsai 2 uses a format described in its release documentation as 1.72 bits per weight. That reduction is what allows a model with 27 billion parameters to occupy a space closer to the size of a large software package than a conventional data center model.

But storage size is only the first part of the local AI experience. A model must also load, generate responses at a useful speed, manage long prompts and work reliably with the software stack available on a consumer device.

The software matters as much as the model

The official Bonsai 2 model card documents two files, including a 5.95 GB PTQ1_0 version and a larger 7.21 GB PQ2_0 version. It also lays out the benchmark methodology, throughput results and the requirement to use PrismML’s fork of llama.cpp.

That requirement is an important qualification for anyone treating the 5.9 GB figure as a simple download and play experience. A model can be small on disk while still requiring a particular runtime, backend or memory configuration. Compatibility becomes part of the product.

PrismML’s official Bonsai demo repository supplies local runtime and setup scripts for Mac, Linux and Windows, with support for CPUs and GPUs. The repository also documents vision features, tool calling, context length and backend limitations. Those details point toward a broader ambition than a text chatbot. PrismML is presenting Bonsai 2 as a model that can participate in workflows, interpret some visual information and interact with tools.

That is where the model could become a consumer feature rather than a technical curiosity. A local assistant that only answers short questions might be useful. One that can inspect a document, reason across a project folder and trigger an approved action could change how people organize work.

Yet every additional capability raises the cost of mistakes. A slightly weaker answer in casual conversation may not matter. A missed detail in a software patch, a faulty research summary or an incorrect tool call can quickly undermine trust.

The 98% claim is contested

PrismML’s headline result is not universally reproduced. ByteShape’s independent comparison scored Bonsai 2 at 91.4%, or 91.7% of its BF16 reference, depending on the comparison presented, well below PrismML’s reported 98.2%. ByteShape also described Bonsai 2 as the fastest tested point on its GPU plots, while noting that the model requires a custom llama.cpp build.

The disagreement is not a minor footnote. A model that preserves 98.2% of a reference model’s benchmark performance suggests unusually gentle compression. A result around 91.5% still represents a potentially useful model, especially if it runs much faster or on much cheaper hardware, but it tells a different story about the tradeoff.

The two figures may reflect differences in evaluation design rather than a simple dispute over arithmetic. PrismML’s number comes from its own reported aggregate benchmark performance. ByteShape used an independent comparison and its own testing conditions. The results therefore do not necessarily measure the same workloads in the same way.

Still, the lower score offers the strongest argument against treating compression as a free lunch. Aggregate benchmark parity can conceal uneven performance. A model might preserve broad knowledge while losing precision in a narrow coding task. It might answer ordinary questions well but become less reliable when a prompt demands many steps, a long context or exact tool use.

That distinction matters because local AI is likely to be judged by moments of failure rather than by average scores. Users will remember the assistant that confidently changed the wrong file, misunderstood a spreadsheet or invented a citation. The promise of privacy and low cost may not compensate for an unpredictable workflow.

Running it is possible, but not always simple

A report from the operator of halfpennymac.com, published in a Reddit discussion about Bonsai 2 on an 8 GB M2 Mac mini, said the model reached about 7.6 tokens per second on that machine. The same report said the result required non-default memory, micro-batch and single-slot settings, while stock runtimes produced allocation or compatibility problems.

That experience captures the difference between technical feasibility and consumer readiness. Seven or eight tokens per second can be perfectly usable for many exchanges. It may make a local assistant feel responsive enough for drafting, summarizing or brainstorming. But if a user must adjust hidden settings, select a special runtime and troubleshoot memory allocation before the model works, the experience remains closer to a workshop project than a polished phone feature.

The local AI ecosystem has often advanced through exactly this kind of experimentation. Enthusiasts accept custom builds and unusual settings because they want control over their models. Mainstream users generally expect an installation process that hides those decisions.

Device makers will have to decide which side of that divide they want to serve. Smaller models could let manufacturers advertise private assistants, reduce server costs and keep sensitive data on the device. They could also make product behavior more consistent in places with poor connectivity. But supporting multiple backends, model formats and hardware limits adds engineering complexity, especially when the model’s best performance depends on a particular software fork.

Compression could reshape the business of AI

If Bonsai 2 or models like it become reliable enough, the economic implications could be as important as the technical ones. Cloud inference turns every serious user into a recurring operating cost. Local inference shifts more of that cost into the device, where the hardware has already been purchased.

That does not mean cloud AI would disappear. Large servers will remain useful for the hardest reasoning tasks, massive context windows and workloads that need centralized data or constant updates. A more likely future is a layered one. The phone handles private notes, quick editing and routine automation locally. The cloud takes over when the request exceeds the device’s memory, speed or capabilities.

Compression also changes who controls the assistant. A model stored on a personal computer can be inspected, modified or replaced without waiting for a provider to change an online endpoint. Users gain more privacy and potentially more resilience. Companies lose some control over distribution, usage measurement and recurring revenue.

The unresolved benchmark gap will influence how quickly that future arrives. The ByteShape Team said in a separate Reddit comparison that its common-methodology evaluation produced roughly 91.5% on a composite benchmark, while emphasizing that the result was independent of PrismML’s reported figures. That result does not erase the value of Bonsai 2’s size or speed, but it reinforces the need to test compressed models on the work people actually do.

The most important test may happen after the benchmark window closes. It will involve a developer asking for a safe code change, a student organizing research, or a traveler using an assistant without a connection. In those moments, Bonsai 2’s real achievement will not be that a 27 billion parameter model fits into 5.9 GB. It will be whether users stop noticing where the model lives, and start trusting what it does.