If you want to know whether an AI system is good at a finance task, you need a way to score its answers. When the answer is a number, that is straightforward: the reconciliation ties or it does not. Much of the work in finance operations that matters is not like that.
Should this item be investigated?
Is this explanation of a variance adequate?
Does this summary of a contract's payment terms leave out anything that matters?
The answer is a judgement, and a judgement needs a different kind of yardstick.
That yardstick is a rubric: a written definition of what a good answer contains, specific enough that two careful people scoring the same answer would mostly agree. In model training and evaluation it is often called a reward signal, because it is what tells the system, or the people assessing it, which answers to prefer. The idea is older than AI. Every good review checklist is one. What AI changes is how much rides on writing it well, because a model trained against it will learn to chase what the rubric rewards, including its mistakes.
Why "sounds right" is the wrong target
The natural first draft of a rubric scores what a reviewer notices first: is the answer clear, confident and well structured? Those are real qualities, but in finance work they are also the qualities of a persuasive wrong answer. A rubric that rewards fluency will produce a system that is very good at sounding right. The first discipline is to score substance before style, and to write down what substance means for this task.
What does a usable rubric contain?
Separate criteria, each checkable on its own. "Is the answer good?" cannot be scored consistently. "Does it identify the unmatched items?", "Does it state the assumption it relied on?" and "Does it say what it could not determine?" can. Each criterion should be something a reviewer can point to in the answer, or point to its absence.
Weights that reflect consequence, not effort. Some criteria matter more than others. Getting an unmatched item wrong matters more than a clumsy sentence. Writing the weights down forces the argument about what matters to happen once, in the open, instead of in every reviewer's head.
Failures that cannot be averaged away. We have made the case for this before; the practical point is how to write it. Name the errors that make an answer unusable however good the rest is, such as reversing the direction of a variance or presenting a guess as a fact, and mark them as critical: present them, and the answer fails. This is the part that is easiest to leave out.
A worked example of a good answer. A short reference answer, written by someone who understands the task, anchors the criteria. It also exposes vague criteria quickly, because a criterion that the reference answer cannot clearly satisfy is a criterion nobody can score.
What this has to do with governance
A rubric is verify made explicit. It turns "a person checked it" into "a person checked it against this, and here is what they found". It is also how you find out whether a system is ready, which is where most AI projects actually fail: not in the demo, but when nobody has written down what correct means.
Where to start
Take one judgement-heavy task in your finance workflow and try to write down, in a handful of plain criteria, what a good answer must contain and what would make it fail outright. If that is hard, the difficulty is the finding: the task is being judged on instinct, and neither a model nor a new team member can be measured against an instinct.
If you are building evaluation for finance or GRC workflows and want a second view on the rubric: [email protected]
AI projects fail at evaluation, not at the demo: why evaluation is where projects succeed or stall.
The 15-point gate we run before any AI touches a real person: a written standard applied before launch.
What an auditor will ask about your AI system: "what did a human actually check" as an evidence question.
TL;DR: When an AI system's answer is a judgement rather than a number, you need a rubric to score it. A usable rubric scores substance before style, splits "good" into separate checkable criteria, weights them by consequence, marks some failures as critical so they cannot be averaged away, and includes a worked example. Writing one is also the quickest way to learn whether anyone has defined what correct means.
Not legal, security or financial advice. Governed performance.