Model-agnostic by design: how Sova routes across models

When we started building Sova, the first architectural question was whether to tie the pipeline to one model provider or build routing in from the start. We chose routing, and it has shaped how the product works at every layer since.

Why we did not pick one model

The obvious path for an AI-native startup in 2023 was to pick the best available model and build everything around it. The risks were not obvious then but they are clearer now: model providers update capabilities with little warning, and specific task performance can shift between versions in ways that are hard to detect without targeted evaluation. Pricing changes. Context window limits change. Rate limits change at the worst possible moment, which is always during a large ingestion job.

More importantly, no single model performs best at every task in a customer intelligence pipeline. Extraction, ranking, synthesis, and classification have different performance profiles. A model that excels at long-context extraction may produce weaker results on structured output with tight token constraints. Building monogamously means accepting that model's worst-case performance for every task type in the pipeline, not just the tasks it is built for.

The routing problem

Model routing is not a new idea, but most implementations treat it as a fallback mechanism: try model A, and if it fails, fall back to model B. That is error handling, not routing. Real routing means selecting the right model before a task runs, based on what that task actually requires.

The extraction layer

Sova ingests unstructured text from support tickets, call transcripts, NPS responses, cancellation notes, and review sites. The extraction step asks a focused question: what problems, requests, and signals are present in this text, and what exact language did the customer use to describe them? This is a long-context, instruction-following task that benefits from models with high output fidelity for structured JSON. It is not a reasoning task. Sending it to a reasoning model adds latency and cost without improving the output, and reasoning models often produce weaker structured JSON than a smaller instruction-tuned model given the same prompt.

The ranking layer

The ranking step aggregates extracted signals across hundreds or thousands of inputs and asks a harder question: how should these be ordered, with what confidence, and why? This requires understanding the relative weight of different signal types, the segment-specific context that affects importance, and the relationship between frequency and impact. Ranking benefits from more capable models because the reasoning is genuinely complex. Using the same model for extraction and ranking wastes cost on the extraction step and produces weaker results on the ranking step. Routing lets each layer use the model suited to its actual demands.

What we route by

When a task enters the pipeline, the router selects a model based on the following criteria, evaluated together rather than in isolation:

• Task type: extraction, ranking, synthesis, or classification each have a default model assignment with override logic for edge cases
• Context length relative to the input being processed, which determines whether a model's window is sufficient and at what cost
• Cost per token given the expected signal volume for this customer's plan tier and current usage
• Latency requirements for the pipeline stage, since some steps are synchronous and others tolerate batching
• Model quality scores from our internal evaluation set for this task class, updated each time we run evals against a new model version

The tradeoffs

Routing adds real complexity. You are no longer debugging one model; you are debugging four or five, each with different prompt formats, different output schemas, and different failure modes. We handle this by treating every model output as untrusted at the boundary: every extraction result is validated against a schema before it enters the ranking step, and every ranking output is validated before it surfaces to the product. Failures are caught at the boundary and handled explicitly rather than propagating silently.

The second tradeoff is consistency. When two similar requests go to different models because the router made different selections, the outputs may differ in small ways. We treat this as acceptable. Sova produces ranked decisions across large signal sets, not deterministic answers to individual questions. At the aggregate level, the variance is low enough that it does not affect decision quality. What matters is that the ranking reflects the actual distribution of customer signal, and that holds regardless of which model handled which extraction.

What this means in practice

When a new frontier model ships with measurably better extraction quality on long documents, we can route a subset of production traffic to it on the extraction step the same day, run our eval benchmark, and increase allocation as confidence grows. The product improves without a release cycle and without any customer-visible change.

When a model provider has an outage, traffic reroutes to the next-ranked model for that task type. Customers see no interruption. When a customer's data residency requirements mean they need to run an open-weight model on their own infrastructure, the routing layer accommodates that without rebuilding the pipeline around a new assumption.

The broader point

Model-agnostic is not a claim about neutrality. We have clear preferences: some models perform substantially better at specific task types, and we route toward them. What the architecture means is that the product is not a wrapper around one model's current capabilities. It is a system that treats models as components, which makes it more resilient to provider changes, more improvable as the model ecosystem evolves, and more adaptable to the constraints that enterprise customers actually bring. That is a better foundation for a product that will need to keep working as the underlying technology changes.