Before buying “agentic BI,” make the demo prove who approves each query

Colrows published a five-level autonomy ladder for analytics agents on August 30, arguing that most production tools remain at level 2 or level 3: either a person approves the drafted query, or the system executes while a person samples the output. The vendor reserves level 4 for systems that run unattended and escalate exceptions, and level 5 for systems that detect and repair their own errors. Colrows explicitly says the ratings are its own assessment, not lab measurements, and discloses that it sells a competing product.

That disclosure matters. This is not an independent benchmark, and buyers should not treat the scorecard as one. The useful part is the operating test beneath it: who plans, who executes, and who approves? Those questions are harder to blur with an “agentic” label.

The approval step is part of the product

Colrows distinguishes a copilot that drafts a query for human approval from an agent that decomposes a question, runs several steps and escalates exceptions. Its sharpest warning is that relabeling a level-2 tool as a level-4 agent can cause teams to remove the very approval step that made the workflow tolerable. The software may not have changed; the control did.

That suggests a procurement rule: document the approval boundary before testing answer quality. If every generated query needs review, measure reviewer load and queue time. If only exceptions need review, define what counts as an exception and prove that the escalation appears. “Human in the loop” is too vague unless the loop has a named trigger.

Five questions for one demo

The vendor proposes five questions that can be asked in a single evaluation session:

1. Ask why a metric moved, rather than only asking for its value. Does the system plan a driver analysis or return one number?
2. Repeat the same question in a fresh session. Does it reproduce the SQL?
3. Ask to see every query the workflow executed, not just the final one.
4. Remove permission to one table in the chain. Is the plan prevented before compilation, or refused only when it reaches the warehouse?
5. Ask how many billable calls the answer consumed.

These are acceptance tests, not proof that Colrows’ product rankings are correct. They turn broad autonomy claims into artifacts a buyer can retain: plans, SQL, permission failures, escalation records and usage counts.

The billing question becomes more important as agents add steps. Colrows notes that a workflow decomposing one business question into many calls can multiply a per-query meter. The governance question is similarly specific: the post argues that regulated buyers should distinguish controls applied before a plan is built from permissions enforced when generated work reaches the warehouse.

The practical takeaway is modest but useful. Do not ask a vendor whether its BI product is agentic. Ask which steps run without approval, what evidence each step leaves, what event stops the chain, and what the whole chain costs. If those answers are not visible in the demo, the autonomy claim is not ready for production procurement.