Orca’s query agent made a 900-model schema usable by refusing to send all of it to the LLM

Orca Security’s first natural-language query prototype did what many enterprise teams try first: it placed the full schema in one large-model prompt. The result was usable enough to validate the idea, but each comparable workload consumed roughly 2 million input tokens, took about 60 seconds and still struggled with queries spanning multiple models. Its production replacement is instructive because the largest gain came from narrowing the problem before generation, not merely choosing a larger model. AWS and Orca Security describe the deployed system; the same first-party text is available in the AWS Machine Learning Blog RSS feed.

Nine gates before a query is accepted

Discovery AI translates plain-English security questions into Orca’s JSON-based Sonar filters. Sonar covers more than 900 cloud and application models, relationships among them and 18 filtering operators. The production pipeline first adds customer context, then sends recognizable identifiers—such as IP addresses, CVEs, ARNs and file hashes—through a deterministic fast path. A guardrail blocks prompt injection, off-topic requests and unsupported inputs before generation begins. AWS/Orca

For questions that need generation, the system retrieves 20 semantically similar, previously validated examples using Titan embeddings and reranks them to five with Cohere Rerank 3.5. A model selector then reduces the 900-plus model universe to the schemas relevant to the request. Only those detailed schemas and their connecting relationships reach the generator, which runs as a tool-using agent. AWS/Orca

The resulting filter must pass static checks and validation against the Sonar API. Failed checks return the offending model, field or operator to the generator for a bounded retry; the system returns an invalid-filter response if it cannot reach a valid result. Orca uses Claude Haiku 4.5 for model selection and generation, Nova 2 Lite for guardrails and validation, Titan for embeddings and Cohere for reranking. AWS/Orca

The practical result—and the evidence gap

Orca reports that the new pipeline lowered input use from about 2 million to 140,000 tokens per comparable workload, a 93% reduction. End-to-end latency fell from about 60 seconds to 15 seconds, while overall query cost dropped 90%. The system has handled thousands of production queries per day since March 2026. AWS/Orca

The post says accuracy improved on complex multi-model questions, but it does not publish an absolute accuracy score, the evaluation-set size or a failure-rate breakdown. Its evaluation harness uses curated production-like slices and an LLM judge for correctness, completeness and semantic equivalence, with domain experts feeding corrections back into the example bank. That supports continuous regression testing, but it does not let outsiders compare Discovery AI with another system. AWS/Orca

The transferable lesson is narrower and stronger: when a structured-query agent faces hundreds of models, context reduction is an architectural stage, not a prompt tweak. Deterministic routing, example retrieval, schema selection and executable validation can reduce the amount the generator must see—and leave it fewer ways to invent a valid-looking but nonexistent field.