Human-in-the-Loop AI Workflow Design

Adding a person to an AI workflow is usually the first safety idea, and often the right one.

The harder question is where that person sits. An approval placed after the consequential action is a notification. An approval placed on every step becomes a queue that people learn to clear without reading. An approval that cannot block anything is a ritual.

A human-in-the-loop workflow is a workflow that stops at a defined point and cannot continue until a named person decides. The design work is choosing that point, the evidence the person sees, and what happens when they say no, or say nothing.

Here is how I would place those gates.

Approve the action, not the answer

A model output is not yet a consequence.

Extraction, classification, and drafting can run freely as long as their results are validated before anything relies on them. A wrong draft that nobody sends costs a little time. A wrong record that three teams build on costs much more.

The approval belongs at the handoff where an output becomes an action that changes something outside the workflow: sending a message, committing money, granting access, publishing a page, or changing a record other people rely on. Reviewing every intermediate answer spends attention where it changes little.

A useful test for each step:

Can the effect be undone cheaply and completely?
Does it leave the system boundary and reach a customer, a supplier, a regulator, or a public page?
Does it commit money, access, or a professional judgment?
Is the input outside the cases the workflow was evaluated on?

If every answer is no, the step usually needs validation and logging, not a person. If any answer is yes, it is a candidate for a gate.

In the loop, on the loop, or after the fact

The phrase covers three different designs, and mixing them up causes most of the confusion.

In the loop: the workflow halts and waits. Nothing proceeds without a decision. This fits irreversible or externally visible actions at modest volume.
On the loop: the workflow proceeds while a person monitors it and can stop or reverse an action. This fits recoverable, repeatable work where waiting on every item would defeat the purpose.
After the fact: the workflow runs unattended and a person reviews a sample or the audit trail later. This fits low-consequence work with cheap corrections.

One workflow can use all three. Intake can run unattended, classification can be monitored, and only the step that sends a result outside the team waits for approval.

Choosing per step is the point. A single global rule, either “a human checks everything” or “the agent is trusted now”, is usually wrong in both directions at once.

Decide what the reviewer must see

A gate is only as good as the decision it enables.

If the reviewer sees a draft and an Approve button, they are judging the prose. They are not judging whether the action is valid, allowed, or supported by the source.

The review view should carry:

The proposed action in plain words, including where it goes and what it changes.
The source evidence behind it, with the exact passage, page, or field the model relied on.
What the deterministic checks already confirmed, and what they could not confirm.
The uncertainty the model reported, and any conflict between sources.
The available decisions: approve, reject, correct, or send back for more evidence.

In document-heavy work, the source evidence is the part that saves time. A reviewer who can jump to the page and figure an output came from checks a claim quickly. A reviewer who has to reopen a 40-page file rechecks everything, or quietly stops checking.

That is why AnalyzTech presents its results in a review interface where engineers inspect the extracted room polygons on the plan itself before triggering AI-assisted checks. The system supports their verification. It does not replace their professional judgment.

See the AnalyzTech plan-review case study. A production floorplan and construction-document verification workflow with human-reviewed outputs.

Approval is a state, not a message

A notification that asks someone to approve is not yet a gate.

The workflow has to hold a state that means “waiting for a decision”, and the consequential action has to be impossible from any other state. Otherwise the approval is advice that the system is free to ignore.

That state needs rules of its own:

Who may decide, and whether that person may also be the one who configured the workflow.
What happens on rejection: stop, return to an earlier step, or route to an owner.
What happens on silence: a timeout that escalates or expires, never one that approves by default.
What happens on a duplicate decision: one approval produces one action, even if the button is pressed twice.
What is recorded: the evidence shown, the decision, the person, and the time.

These are the cases worth testing before a pilot, because they are the ones a happy-path demo never exercises.

In the synthetic construction example on this site, approval denial, timeout, duplicate events, and fallback each had an expected outcome defined before the run, and a bypassed approval was an automatic no-go condition.

See approval denial, timeout, and duplicate cases in an executed synthetic workflow. The company and documents are fictional; the example demonstrates the method, not a client outcome or production deployment.

Keep the gate from becoming a rubber stamp

The common failure is not a missing gate. It is a gate that approves everything.

When most requests are routine, reviewers learn that approving is always right. The gate still appears in the diagram and in the audit log, but it no longer changes outcomes.

A few signals tell you whether the review is real:

How often reviewers reject or correct. A rate near zero for months means the gate is in the wrong place or nobody is reading.
How long a decision takes compared with how much evidence it covers.
Whether known-bad cases, added on purpose to the review queue, get caught.
How many requests are waiting, and for how long.

The fix is usually to remove gates rather than add them. Move routine, reversible steps to validation and sampling, and keep human attention for the few actions where a person can see something the checks cannot.

Start with one gate you can defend

Pick the single action in the workflow that would be hardest to explain if it went wrong.

Then write down four things: the state the workflow must be in before that action, the evidence the reviewer sees, the rule for rejection and for silence, and the person who owns the decision.

If you cannot fill in those four lines, the workflow is not ready for more automation. If you can, the gate becomes a design decision instead of a reflex.

Which action in your workflow would you least want a model to take on its own?

Read how the deterministic layer decides what a model output may do. Validation, state, and permissions handle most steps, so the human approval gate can stay small.