Q-Router: A Neurosymbolic Routing Engine for the Multi-LLM Inference Stack

Bringing intelligence to a frontier stack at a fraction of the cost.

From Thesis to Product

In our previous blog post, “AI Routing: Why Flow of Intelligence is the Hidden Key to Enterprise AI Economics”, we proposed that for enterprises utilizing multiple AI models, the true advantage lies in effectively governing the flow between them rather than merely chasing the latest frontier model. We established that routing policies serve as the foundation of modern AI unit economics. Building on that thesis, this post introduces the next step: a tangible product backed by concrete benchmarks and peer-reviewed research.

We developed Q-Router, a neurosymbolic routing engine that integrates between your application and your AI models, intelligently determining the optimal model for every query. We also subjected our most advanced routing algorithm, PROTEUS (Polymorphic Router for Operational Target Enforcement with Unified SLA), to rigorous academic peer review. The resulting research was accepted at ACM EuroSys 2026, a top-tier venue for computer systems research. Today, Q-Router is production-ready, available as a Python package and HTTP server with a comprehensive SDK for seamless integration into existing AI pipelines.

Figure 1 – Market Comparison

Why Q-Router Stands Out

Current market solutions for routing tend to be fragmented, focusing on isolated capabilities like load balancing or basic latency metrics rather than a comprehensive production solution. None are inherently poor, but they simply do not offer enough coverage. Our analysis (Figure 1) bears this out: when evaluating eleven routing products against ten critical production capabilities, Q-Router was the only solution to reach full coverage, while the competition stagnated between zero and 1.5. The market data suggests not just a lack of a clear leader, but a lack of complete solutions.

Most routers force an architectural compromise between two extremes:

1. Rules-only systems: Fast and transparent, but brittle; they fail whenever they encounter unexpected inputs.
2. Model-only systems: Adaptable and effective at generalizing, but opaque; teams often cannot explain or override their decisions.

Both architectures hand you the same two operational headaches. Your model pool remains locked to the router’s initial configuration, that was trained on a fixed list of models, so the day you want to add Claude 4.7 or swap out GPT-4o for whatever ships next month, you’re staring down a full retraining cycle and the training data doesn’t exist, custom routers require thousands of labeled (query, best-model) pairs, they remain inaccessible to most teams.

Q-Router rejects these compromises. We built it on the belief that production routing requires both hard rules and learned intelligence, a pool that adapts to your specific models, and the ability to bootstrap training data without an existing labeled set. This led us to develop three key capabilities that our competitors lack.

Part 1 : The Neurosymbolic Engine

Q-Router is built on the belief that choosing between rule-based logic and machine learning is a false dilemma. Production routing requires both: hard rules for predictable queries, and learned models for cases where data-driven intuition is required. To achieve this, we designed Q-Router as a neurosymbolic system, where a symbolic layer and a neural layer operate in partnership, each mitigating the other’s limitations.

Figure 2 – Overview Neurosymbolic System and Q-Router

Layer 1: The Symbolic Heuristic Scorer

Every incoming query is first processed by a symbolic scorer based on 14 hand-crafted dimensions, including token count, code presence, technical vocabulary density, and output format requirements, and so on. Using weights derived from a curated test suite, the system assigns each query to one of three tiers: SIMPLE, MEDIUM, or COMPLEX.

This layer provides advantages that neural-only routers cannot match:

• Instant deployment: Works immediately with zero training.
• Total interpretability: Every decision is traceable to specific signals within the query.
• Catastrophe prevention: A Bloom’s-Taxonomy verb gate forces “recall” queries (e.g., “define X”, “list Y”, “name Z”) to the SIMPLE tier regardless of what the neural layer thinks. This prevents common factual queries like “What is the Pythagorean theorem?” from being routed to an expensive frontier model just because it contains the word “theorem.”
• Sub-millisecond speed: No GPU, no API calls, no model loading.

Layer 2: The Neural ML Router

For specialized domains like legal, biomedical, multilingual, or code-heavy, rules alone often miss the necessary nuance. Q-Router complements the symbolic layer with a neural network. A 384-dimension sentence embedding feeds an MLP that calculates confidence scores per model. The neural layer learns from (query, model quality score) pairs to identify which models excel at specific query types, capturing fine-grained patterns that are impossible to encode manually.

Ultimately, these layers work together rather than competing. The symbolic layer handles clear-cut cases definitively, while the neural layer refines routing for the “grey zone.” This hybrid design ensures that Q-Router is robust enough for production, not just a research prototype.

Part 2 : Pluggable Model Pools

Most routers train against a fixed list of model identifiers, making them rigid. If you swap a model, for example, replacing GPT-4o with Claude Opus 4.7, the router loses its ability to function, forcing you to either retrain it from scratch or accept an outdated policy.

Q-Router eliminates this constraint by decoupling models from capability tiers. You simply register your models into abstract tiers (small, medium, large), and the router reasons in “tier-space” rather than by model name. This allows for seamless model swaps and the integration of self-hosted or experimental models without downtime, router migration projects, or retraining. The router remains a stable, long-term asset, while the model pool is managed via a simple configuration file.

Part 3 : Self-Bootstrapping Training Data

The primary barrier to custom routing has always been data. Surpassing heuristic baselines typically requires thousands of labeled examples of “on this query, model X performed better than model Y“, a manual and costly process that leads most teams to settle for prebuilt solutions.

Q-Router overcomes this with Inspection Mode, an automated pipeline that generates its own training data. You provide real traffic queries, and Inspection Mode runs them across your model pool, using an LLM-as-judge (e.g., GPT-4o or Claude) to score the responses and produce a labeled JSON dataset ready for training.

This creates a powerful, iterative loop: start with a heuristic router, log production traffic, use Inspection Mode to generate labels, train a custom router, and hot-swap it into production. As your traffic evolves and new models emerge, you can regularly refine your router to stay optimized.

By closing this loop, Q-Router offers an adaptive, self-improving infrastructure that standard static routers cannot match. Together with its neurosymbolic engine and pool extensibility, these features enable Q-Router to provide capabilities unmatched in the current market.

Beyond the Engine: The Operational Surface That Makes Q-Router Deployable

Routing intelligence is the engine. But what makes a router actually deployable is the operational surface around it and this is where Q-Router pulls away from the field.

Here are the capabilities most teams discover they’re missing only after they’re in production:

1. TTFT / ITL / E2E latency metrics: Time-to-first-token, inter-token latency, and end-to-end latency are the three numbers that determine whether your app feels fast. Almost no router in the market reports all three. Q-Router does, on every request.
2. Cost-saving reports: Most routers don’t tell you how much you actually saved. Q-Router logs every decision and produces a 3-scenario savings range (conservative / expected / optimistic) against your specified baseline model.
3. Routing confidence on every decision: When the router picks model X, you can see exactly how confident it was relative to the alternatives. This makes debugging traceable and makes X-Ray Mode possible.
4. Auto Mode and Transparent (X-Ray) Mode: Auto Mode picks for you. X-Ray Mode reveals the top-k candidates with confidence scores so a power user or a downstream agent can override the router’s decision. Both behaviors come from the same server, toggled per request.
5. Multi-model response comparison: Inspection Mode runs the same query across multiple models in parallel, has an LLM-judge compare responses, and produces training-grade labels. No other router on the list does this natively.
6. Token-level signals and preference cache: Log-probs, top-k entropy, perplexity, and cached preference signals. The full quality observability stack, ship by default.
7. Runtime SLA enforcement: This is the capability that none of the alternatives provide and the one our research directly enables. More on this next.

PROTEUS: Accuracy SLAs as Runtime Input

The standout feature of Q-Router and our primary contribution to the AI systems community is PROTEUS, a routing algorithm specifically designed to solve the challenge of runtime accuracy SLAs. It’s a reinforcement-learning router that accepts accuracy targets as runtime input and consistently meets them.

Unlike traditional routers that rely on indirect, offline tuning of variables like budget caps or confidence thresholds, where the connection to actual performance is often non-monotonic and dataset-dependent, PROTEUS treats accuracy targets as a direct runtime input.

When your business requirements dictate specific performance tiers (e.g., 92% accuracy for enterprise clients vs. 87% for free-tier traffic), PROTEUS allows you to simply input those targets. By leveraging Lagrangian dual control, the router translates these targets into precise routing decisions in real-time. This enables a single trained model to effectively cover the entire accuracy spectrum (τ = 0.85 to τ = 0.95) without the need for additional retraining.

Figure 3: Pareto Frontier View

With a target-response correlation of 0.97–0.98. When you ask PROTEUS for 90% accuracy, you get 90% accuracy. When you ask for 92%, you get 92%. The router doesn’t just try to hit your SLA rather it delivers it, predictably, across the full operating range. This is what makes it usable as an actual contract.

One router. Three SLAs. Every contract met.

Most LLM routers treat SLAs as a parameter to be tuned. PROTEUS treats them as an input to be served.

Consider a SaaS application with tiered service levels: Economy (88% accuracy), Standard (92%), and Premium (95%). Conventional routing typically requires three separate, siloed deployments, each with its own threshold tuning and maintenance overhead. Any contract modification necessitates a full retraining cycle.

PROTEUS eliminates this complexity. A single trained model accepts an SLA target (τ) per query, allowing you to pass τ=0.95 for Premium users and τ=0.85 for Economy users simultaneously. The system adapts its routing policy in real-time, directing simple queries to cost-effective models that still meet the target, while reserving frontier models for complex reasoning tasks.

Figure 4: Overview SLA Compliance

The numbers from the SPROUT benchmark tell the story:

• Economy (τ ∈ 0.85–0.88) delivers 92.3% accuracy at $0.30 per 1,000 queries, 7.3 points above the contracted floor.
• Standard (τ ∈ 0.88–0.92) delivers 93.8% accuracy at $0.62 per 1,000 queries, 5.8 points above floor.
• Premium (τ ∈ 0.92–0.95) delivers 96.0% accuracy at $1.69 per 1,000 queries, 4.0 points above floor.

Across all three tiers, PROTEUS ensures 100% floor compliance. By maintaining a 5.6× cost gradient between Economy and Premium tiers, it enables flexible pricing that optimizes margins without compromising quality. Crucially, it eliminates adaptation latency; whether a customer upgrades mid-session or operational load spikes require temporary adjustments, PROTEUS adapts instantly, no retraining, parameter recomputation, or model swapping required.

This real-time responsiveness separates PROTEUS from theoretical research, turning it into a production-grade system. While traditional serving infrastructure treats latency SLAs as firm, enforceable contracts, PROTEUS extends that reliability to LLM accuracy, finally making accuracy guarantees a reality.

Why We Believe This Matters

Our initial post established a thesis: intelligent routing is the primary lever for optimizing enterprise AI margins. With Q-Router, we have transitioned that concept into a concrete, production-ready solution. Through PROTEUS, we demonstrate that AI infrastructure can finally achieve the rigorous, provable SLA enforcement long standard in networking and database systems. We subjected PROTEUS to the ACM EuroSys peer-review process not as a formality, but to ensure that our work met the high standards required for critical production infrastructure, with the acceptance serving as independent validation of its efficacy. Just as latency is treated as a binding contract, we believe accuracy should be as well.

Q-Router represents the complete ecosystem supporting this capability encompassing the symbolic-neural engine, versatile operational modes, and an extensive developer toolkit. It reflects our vision for the future of enterprise AI, where routing is no longer an afterthought but a foundational component of robust, cost-effective infrastructure.

Where to Go Next

Q-Router and PROTEUS mark only the beginning. Our lab is already actively pursuing two major research frontiers:

Multimodal Routing

While single-modality LLM routing is effectively solved, the industry is shifting toward vision-language models, document understanding, and multimodal reasoning. These tasks introduce significantly greater complexity, as query signals now include both text and images, and the cost-quality gap across multimodal models is far wider than in text-only models. Because existing benchmarks like RouterBench and SPROUT fail to address this, we’ve developed a purpose-built in-house benchmark for multimodal routing. Preliminary results indicate that our neurosymbolic architecture extends effectively into this domain, and a formal research paper is currently in development.

Agentic Routing

We are particularly enthusiastic about agentic routing, a frontier the broader industry has yet to fully recognize as a routing challenge. Agentic workflows extend beyond simple, single-shot LLM interactions into complex decision chains: determining which model is best suited for context assembly, tool selection, memory retrieval, or sub-agent allocation, and orchestrating routing across a diverse team of specialized agents. Currently, these intricate decisions are often relegated to rigid, hard-coded defaults. We are actively expanding Q-Router’s capabilities to address this agentic stack, with research findings to be published in the coming year.

Routing as a margin lever was our initial thesis, and Q-Router is the platform that delivers on that promise today. The next phase of our evolution lies in mastering these multimodal and agentic frontiers.

References:

• Read the EuroSys paper: PROTEUS: SLA-Aware Routing via Lagrangian RL for Multi-LLM Serving Systems
• Catch up on the thesis: AI Routing: Why Flow of Intelligence is the Hidden Key to Enterprise AI Economics
• Talk to us: If you’re running multi-LLM workloads in production and want to explore what routing could save you, reach out to us at philabs@quantiphi.com.