Launch HN: Tokenless (YC S26) – Automatic model switching to save money

So this only switches models if the cache is cold, because otherwise the economics of switching don't work. But most agentic work involves long strings of successive tool calls that benefit from a hot cache. Hot cache calls reduce input cost by 90%. This can basically only deliver cost savings in turns where the AI delivers a result to the user, the user waits at least 5 minutes (or the length of the cache), and then responds. But user->AI calls are very much the rare case now, the more agentic the workload. Most of them will be tool->result->tool without the user involved. And token burn is highest with these long running agentic chains, but that's precisely where routing doesn't work because of the KV cache. How do you deal with that?

I ran some tests last quarter with a similar multi-model routing setup focused on code completion tasks. Using an open-source Claude alternative combined with Claude-instruct for fallback, I saw about a 45% cost reduction compared to consistently hitting the Claude Code API alone, with latency penalties under 300ms for the fallback requests. The key was tuning the confidence threshold so that around 70% of calls got handled by the cheaper local models without sacrificing accuracy much. One interesting data point was that for smaller completions (~50 tokens or less), open-source models like OpenHands outperformed cloud-hosted Claude alternatives in terms of 95th percentile latency—likely because no network involved. But for bigger, multi-file refactoring queries, fallback to frontier models was unavoidable to hit the 90+% pass rate. The dynamic routing balanced those two extremes to keep API spend down while still scaling. That said, the caching layer remains a bottleneck. Changes in the input context or tool states quickly cold the cache, forcing more queries to the expensive model. For workflows with lots of sequential tool calls (e.g., automated code lint + formatting + test generation), the raw switch cost was around 15-20% overhead compared to fixed-model baselines until cache warm-up happened. This matches the experiences others have reported.

Super interesting approach. It's probably novel. I can say this because I've been working on something similar (while building a code version of http://pellmell.ai). I'm skeptical though. In order to pick which model is on the right trajectory, you actually need intelligence. But real intelligence would make your system painfully slow and more expensive. I suspect you're using a classifier of some sort, but I also suspect what it's really measuring is confidence. Most likely, this is a fantastic approach for the kind of problem where there's uncertainty but only one correct solution. But this is going to be really bad for cases where there are many potential solutions, some of which look good but are in fact bad. You only show one benchmark, and I'm wondering if it happens to be nicely shaped for this kind of router. Have you run it on DeepSWE?

The failure mode I would want addressed before putting this in front of an agent fleet is silent quality regression. When a turn gets routed to a cheaper model and the agent still completes the task, the trace reads as a success, and nobody attributes the worse output to the routing decision until it has compounded downstream. In the enterprise pipelines I build the spend win rarely came from per-turn model choice anyway, it came from cutting how many turns hit an LLM at all, with models pinned per workflow step so a regression was attributable to one change. That pinning is what made the system auditable, which is what a buyer's platform team actually signs off on. Do you run a shadow sample where routed turns are also executed on the frontier model, so a customer can see the measured quality delta instead of trusting the router's own confidence estimate?

Interesting approach. The multi-model progress monitoring idea is clever, most routing I've seen is either static rules or a cheap classifier that picks once upfront. Querying in parallel and deciding mid-turn feels different. One thing I'm curious about: how do you handle the latency hit from spinning up multiple models on the harder turns? Does the user-facing latency still feel competitive with just going straight to Claude, or is there a noticeable pause while the router decides? Also, any plans to expose the routing decisions (or at least the model chosen per turn) so people can debug when it picks poorly? That seems useful for the feedback loop you're asking for.