When the model should ask for help

01 / DEFINE THE DECISION

A score is not a decision.

Imagine an inspection model that assigns each part a defect score. High scores look suspicious. Low scores look acceptable. Somewhere between the two, a team has to draw a line. The line is a decision about consequences, even if it arrives in the software as a default value of 0.5.

A missed defect might lead to a return or a failed assembly. Rejecting a good part wastes material and capacity. Sending it to a person takes time. There is no single threshold that tells us how those costs should be balanced without knowing what happens after inspection.

The example uses 240 synthetic parts, including 30 defects. You can change the threshold, reserve an interval for review and limit the number of reviews available for the batch. Nothing here is measured factory performance. The point is to make the operating policy visible.

02 / DEFINE THE DECISION

Start with what each mistake costs.

For a fixed test set, count missed defects and good parts rejected at each threshold. Then connect those counts to the relevant consequences. An overall accuracy number hides that distinction. A threshold that avoids most rejection can still pass the defects the inspection was meant to catch.

Here a rejected good part costs 40 illustrative units, a review costs five and a held part incurs ten units of handling cost for the batch. The missed-defect cost is adjustable. These are scenario inputs, not estimates from an actual operation. Hold costs also leave out any unresolved downstream risk.

If the score were a well-calibrated defect probability p, and the only choices were accept or reject, accepting would have expected cost p times the cost of a miss. Rejecting would cost one minus p times the cost of scrapping a good part. Those assumptions lead to the threshold below. The demo’s score is not calibrated, so it evaluates its actual synthetic outcomes instead of applying that formula as a probability rule.

Reject when p > scrap cost / (miss cost + scrap cost)

03 / DEFINE THE DECISION

Give the uncertain cases somewhere to go.

A review band creates a third route. Scores below the band are accepted automatically; scores above it are rejected. Cases inside it are offered to a reviewer. That can reduce automatic mistakes, but it creates a queue that needs an owner, capacity and a response time.

Widen the band and watch the review demand. Once capacity is exhausted, this model holds the remaining cases. It does not quietly accept them. That behavior matters. A policy that appears safe only because it leaves a growing pile unresolved is not ready to operate at that arrival rate.

People make mistakes too. The reviewer-accuracy setting is an explicit simplifying assumption: the same independent correctness rate for every reviewed case, including the difficult ones. Real review should be evaluated on the cases it actually receives. In practice, those are often a more demanding set than a random sample.

04 / DEFINE THE DECISION

Confidence needs a reference point.

A displayed score of 0.9 is not automatically a nine-in-ten chance of a defect. Calibration asks whether reported probabilities agree with observed frequencies across suitable groups of predictions. Good ranking and good calibration are different properties. A model can order parts usefully while giving probabilities that are too confident.

Guo and colleagues demonstrated calibration problems in modern neural networks and studied post-processing methods such as temperature scaling. That is a useful reason to check probability claims, not a reason to assume one adjustment will fix every model. Calibration needs independent data that represents the setting in which the scores will be used.

If a policy relies on probability, evaluate calibration as well as discrimination. Look across relevant defect types, instruments and operating conditions, and keep enough examples to see uncertainty in the estimates. A small bin with no observed misses does not establish that the miss rate is zero.

05 / DEFINE THE DECISION

The world can move while the threshold stays still.

Switch on changed conditions in the model. It moves good-part scores upward and defect scores downward, producing more overlap while keeping the same 30 defects in the batch. This is a deliberately simple stress test. It is not a simulation of a specific camera, material or failure mechanism.

An unfamiliar input does not have to receive a cautious score. A model can be confidently wrong outside the conditions it was evaluated on. A review band based only on the score will not catch every such case. Monitoring input changes, checking labels and maintaining separate rules for known operating limits can all matter.

The NIST AI RMF Playbook treats monitoring, human roles and responses to changing performance as ongoing work. For an inspection system, I would name who can pause automatic decisions, what evidence triggers that pause and how held work is cleared. “A person can override it” is not enough if nobody owns the queue.

06 / DEFINE THE DECISION

Decide what earns the right to run automatically.

Start with a defined decision and a representative held-out set. Agree on the consequences of each outcome, the review capacity and what happens when the system cannot complete a decision. Compare policies on the same cases, then check them in the operating conditions that matter.

The outputs below include expected reviewer mistakes, so some totals are fractional. They also show held work separately. That keeps a cheap-looking partial result from being mistaken for a completed inspection batch. Reveal the reference labels to see which regions contain the errors.

A useful model does not need to make every call. It needs a clear boundary around the calls it can support, a workable handoff for the others and evidence that those boundaries still make sense. That is part of the engineering, and it stays part of the leadership work after launch.