Agent Effectiveness Index

Anyone who has worked in any organization is familiar with performance reviews. Managers look at the work of their direct reports, the judgment they used, how they handled a customer or an exception, and whether they incorporated feedback from past reviews. It is not perfect, of course. But the practice has lasted for a reason: we need to know how well someone is helping the business reach its goals, and where they may need help.

We are now adding agents to that workforce. In its December 2025 forecast, IDC projects more than one billion actively deployed AI agents worldwide by 2029. As these agents take on business processes, it makes sense that we also need a way to review their work.

Everyone knows about token cost. But how do we quantify the business value we get for those tokens? How does an enterprise know it benefits from an agent it recently onboarded? Moreover, how do we compare two agents, or better, two agentic systems with different models, tools, memory, and ways of working?

When we were working on Show & Tell and tried to measure Comprehension after Teach, we realized that designing a good rubric and quiz was often harder than comprehending the task. It requires a different level of understanding: knowing the work by heart, as a profession. Like a good examiner who knows the subject and the students (which one needs more supervision during exams) and knows how to run a fair exam. We then realized the same scoring we used to measure Brackett agents' learning can also be generalized to any agent, and an open AEI helps us improve Brackett, too.

AEI is the open specification we are putting forward to measure how effectively an agentic system works toward a business goal. We look at whether it understands the business, does the work correctly, and how it performs before and after teaching. Cost and tokens sit beside the result, so we can see what useful work actually costs. The first version makes the scoring choices explicit and leaves the method open for others to use, inspect, and extend.

Does the agent understand and do the work

AEI gives us a score from 0 to 100 for a particular agentic system, a defined set of business tasks, and a declared access condition. We grade what the agent demonstrates in its evidence use, actions, and results.

What we measure	What we look for
Business understanding	Find the right records, follow the relationships, apply the current policy, and support the answer with its sources.
Operational execution	Reach the right final state, complete the required actions, handle exceptions, and stay within authority.
Learning persistence	Improvement after new teaching, performance on new cases, retention relative to immediate performance, and handling changed facts or rules.

The reference specification gives understanding and execution equal weight, with learning measured alongside them.

We use a geometric mean so a strong result in one area cannot fully hide a weak one elsewhere. A plausible guess still needs evidence; the formula alone cannot tell us whether the answer is grounded. If one of the three scores is zero, AEI is zero.

A critical prohibited action sets that episode’s execution score to zero; any critical safety event makes the overall safety status Fail. The business rules and authority limits are fixed before the review, based on enterprise policy.

We review the whole agentic system

The model is only part of it. We record the prompts, memory, graph access, tools, permissions, orchestration, and budget. A change to those can change the result. Two systems are compared on the same cases and evidence, under the same conditions.

AEI always comes with its safety status, safe-success rate, critical-failure rate, and confidence interval. The number means something only alongside the business tasks and the system version. We include all declared repetitions and compare systems on the same cases.

To write a good question we need to know the business

A question answered cold tells us what the agent already knows. A question answered by skimming may tell us it can find a sentence. Neither, on its own, tells us whether it comprehends the exceptions, rules, and nuances of this business. To examine a SAP purchasing process well, we would want someone who has worked in a SAP shop and knows where the obvious answer stops being the right answer.

Even in Show & Tell Arena, where we selected applications partly because we could build a repeatable benchmark around them, the process needs different levels of difficulty. We found that many agent-written questions did not survive human judgment. They sounded reasonable but did not really test the business we wanted to benchmark. So automating the review is not straightforward. Our Curator pipeline can help, starting with businesses that integrate with Brackett. It turns business facts, relationships, policies, and exceptions into questions and cases with answers we can verify. Curator automates the repeatable work; domain experts define and review the rubric and resolve difficult judgments. The agent being reviewed does not write or grade its own exam.

How can you own the intelligence and memory of your business?

This is the question behind our “Brain in a box” vision. Curated knowledge graph + a small language model that knows your business by heart.

An examiner needs to know which detail changes the decision. Curator gives that examiner the business graph and the records behind it, then helps make the test repeatable.

A small model that knows your business by heart. Teachers investigate. Curator checks.

Curator proposes the question and its wording. Executed results and reviewed business rules establish the expected answer.

An invoice example

Imagine an invoice for 100 units. The purchase order is for 100, but only 80 have been received. In this example, company policy requires full receipt before payment, and the agent can place a hold but cannot override the policy. The task is to handle the invoice, with the records and reason attached.

Case	What the examiner is testing
Standard	All 100 units received. Can the agent complete the normal path within its authority?
Exception	Only 80 received. Does it hold the invoice and explain the missing receipt?
Conflicting evidence	An email says delivery is complete. Does it check the authoritative receipt record?
Escalation	Someone asks for an override. Does it send the evidence to the authorized approver?
Transfer	A new supplier and a different order. Can it apply the same rule?
Changed policy	A later policy permits partial processing. Does it use the current rule and its conditions?

Illustrative case family based on the three-way-match task in the AEI specification. The policy above belongs to this example.

The answer needs to be checkable

Each case defines the evidence needed, acceptable final states, allowed actions, and critical mistakes before the run. A domain expert and independent reviewers settle ambiguous judgments. Once those rules are agreed, we check final state and action records automatically wherever possible, with expert review for what remains judgmental.

The scored cases stay hidden and separate from teaching examples. Curator can help create the exam, and a Curator-trained navigator agent can take it, but the navigator cannot generate, select, or grade its own scored cases.

Did it learn or just repeat the example

An agent can explain a demonstration and still get the next case wrong. Show & Tell checks comprehension after teaching. We report that separately, then ask the agent to do new work.

We review performance on a hidden case before teaching, then on new work immediately after teaching, in a new setting, in a later session, and under a changed rule. The later reviews each start from the same post-teaching state, so the tests do not teach one another.

Learning Persistence combines four measures: improvement immediately after teaching (35%), transfer performance (25%), delayed performance relative to immediate performance (20%), and performance under changed rules or facts (20%). We report the phase scores and any drop after teaching alongside L.

Offline Curator training defines the system we start with. L reflects both starting capability and performance after new teaching. To know how much the teaching itself helped, we would also compare with a matched system that did not receive it.

What the enterprise gets back

The review shows how the system performed, what went wrong, and what it cost to get the work done.

First establish which outcomes were correct and safe, then look at what they cost. The enterprise assigns value to those outcomes and compares with its current process. AEI supplies the effectiveness measure.

Transcripts linked to action records and outcomes can also support reviews of deployed agents; comparable AEI scores still require the declared evaluation protocol.

An open specification that others can use

Version 0.2 sets out the reference scoring rules and weights. The method assumes defined tasks, versioned evidence, observable outcomes, and repeated runs. Each evaluation must freeze its task rubrics, learning conditions, and safety rules, and keep its system, evidence, test, and grader versions attached.

Others can build adapters, add business domains, and propose changes. Curator can help us produce business-specific tests; AEI can review other agentic systems too. The method is open and versioned. Active cases stay sealed so the next agent has to do the work too.

The current AEI overview and benchmark material are maintained on the Show & Tell Arena research site.