Human-in-the-Loop Design: Deciding Which Decisions an Agent May Not Make Alone

“Start small” has become unhelpful advice. Some teams interpret it as a chatbot visible to everyone but connected to nothing. Others call a strategic transformation small because the first phase has only six departments. Neither creates safe evidence.

A genuinely small first agent has a narrow operating boundary. It serves a defined group, handles one repeatable task, reads approved information, proposes an outcome and leaves the consequential decision to a named person. The organisation learns about real language, messy data, integration and review without placing customers, money or records at uncontrolled risk.

For a Saudi enterprise, this is a practical way to align innovation with SDAIA’s emphasis on responsibility, privacy, security and human oversight across the AI lifecycle. Governance should be proportionate to the decision, not proportional to the excitement around the technology.

Human oversight is an operating design, not a checkbox

Teams often begin with a model comparison. That is several decisions too early. The first questions are: whose work is changing, what exact outcome counts, which information is authoritative, what action is permitted, and who owns a wrong result?

The model is one component. A production agent also needs identity, retrieval, tool permissions, validation, approval, trace capture, monitoring and an incident route. A powerful model inside a vague workflow is harder to control than a modest model inside a clear one.

Write one sentence in this form: “For [named user group], the agent will [defined task] using [approved evidence], may [bounded actions], and must [escalate or obtain approval] when [conditions].” If the sentence requires “and” four times, the scope is probably several deployments.

An example: “For the procurement operations team, the agent will check supplier onboarding packs against the current checklist, cite missing or inconsistent documents, and draft a follow-up; it may read the approved repository but cannot approve a supplier or send a message without review.” This is useful, measurable and reversible.

Classify decisions by consequence and reversibility

The best first task is frequent enough to generate evidence, painful enough that improvement matters and bounded enough that reviewers can recognise correctness.

Score candidates on six dimensions:

Dimension	Favourable first deployment	Unfavourable first deployment
Frequency	Dozens or hundreds of cases monthly	Rare exceptional decision
Inputs	Available, permissioned, recognisable	Scattered, undocumented, highly sensitive
Outcome	Reviewers can define correct completion	Success is subjective or delayed for months
Reversibility	Draft or recommendation can be rejected	Irreversible external or financial action
Variation	Repeated pattern with manageable exceptions	Every case follows a different process
Ownership	One process owner can decide	Several functions dispute authority

Good candidates include document completeness checks, internal policy retrieval with citations, service-ticket classification, meeting-action extraction, tender requirement comparison and draft responses. Poor first choices include hiring decisions, credit approval, safety-critical control, autonomous payment, disciplinary action and unrestricted customer advice.

Avoid selecting a task only because data is easy. A generic summariser may demonstrate technology but teach little about an operational outcome. Equally, do not select the most visible customer journey merely to attract sponsorship. Early visibility amplifies every gap before the team has learned to operate the system.

Decide what the agent may recommend, prepare or execute

It does not mean no governance. It means decisions can be delegated within an agreed low-risk boundary.

The minimum accountable group is small. A business owner defines the task and accepts the outcome. A technical owner controls configuration, access and operation. A domain reviewer determines correctness. A privacy or security contact confirms the data and access design where relevant. One person may hold more than one role in a smaller organisation, but accountability should remain explicit.

Record a one-page deployment charter: purpose, users, data, approved sources, tools, prohibited actions, human-approval point, success measures, stop conditions, retention and owners. Obtain the approvals required by existing information-security, privacy and change processes. Do not invent an AI committee where normal delegated authority already covers a low-risk internal tool.

Escalate when the design crosses a real threshold: sensitive or regulated data, external users, material decisions, employee monitoring, consequential profiling, autonomous writes, safety impact, new cross-border processing or a use not covered by existing policy. A steering body should resolve those issues, not review every prompt revision.

Put every autonomous action inside a hard boundary

Boundaries must be enforced in the system. A policy saying “the agent should not send email” is weaker than simply withholding the send-email tool.

Start with named users authenticated through existing identity. Give the agent access only to the document collection required by the task. Prefer read-only APIs. Use allow-listed tools with typed fields and server-side permission checks. Separate test and production credentials. Do not place secrets in prompts or knowledge documents.

Limit actions by workflow state. The agent can analyse and draft; a user must approve; the application validates fields; then an existing system executes under the user’s authority. Display exactly what will happen before approval. Record who approved it and the result.

Set conversation and data boundaries too. A procurement agent should not answer HR questions merely because the underlying model can. Reject or route out-of-scope requests. Define maximum document size, supported formats and handling for password-protected or corrupted files. Detect instructions embedded in retrieved documents that attempt to change the agent’s role.

Make the evidence presented to reviewers governable

Many first agents fail because they are connected to a folder rather than a source of truth. Inventory the documents needed for the task. For each, identify owner, effective date, review date, jurisdiction or business unit and superseded versions.

Retrieval should filter by user entitlement and document status. The response should cite the specific source and section. If sources conflict, the agent should surface the conflict and escalate rather than invent precedence. If evidence is absent, “not found in the approved source” is often the correct answer.

Create a small terminology list for Saudi entities, Arabic and English names, acronyms, units, dates and local process terms. Test Arabic, English and mixed-language input if users operate that way. A bilingual interface is not proven by translating ten polished prompts.

Document ingestion needs change control. When a policy is replaced, the old version should no longer be retrieved for current questions but may need retention for historical cases. Run relevant evaluation cases whenever the authoritative collection changes.

Design an approval point that can genuinely stop the agent

Human-in-the-loop is useful only when the human can detect the mistake and has time and authority to intervene. A button below a long, plausible draft is not meaningful oversight.

Show the reviewer the input, extracted facts, evidence, proposed action and uncertainty or exception flags. Highlight changed fields. Require confirmation for important identifiers and values. Make rejection and correction easy. Do not reward reviewers only for throughput; that encourages automatic approval.

Define which outcomes require specialist review. Routine completeness checks may go to an operations user. A contractual interpretation goes to legal or commercial authority. The agent should route based on the task contract rather than making the consequential judgement itself.

Capture corrections in structured form: wrong intent, missing evidence, incorrect fact, poor draft, wrong route or tool problem. Free-text comments alone are difficult to aggregate. Use those corrections to improve the evaluation set.

Establish thresholds before launch

Measure the current process for at least a representative sample. Record incoming volume, handling time, queue age, rework, error categories, escalation rate and completion outcome. Without a baseline, a pilot can celebrate fast model responses while the total process becomes slower due to review and correction.

Define a small balanced scorecard:

• correct-resolution rate, including correct escalation;
• critical-failure count;
• human correction rate and correction type;
• end-to-end cycle time;
• reviewer time per case;
• automation yield after quality gates;
• cost per correct resolution;
• user effort or satisfaction where appropriate.

Agree thresholds before the team sees pilot results. Also define stop conditions: unauthorised disclosure, an unapproved external action, systematic use of stale evidence, loss of trace capture, or performance below the agreed floor.

Test automation bias, missing context and exception handling

Start with representative historic cases after de-identification, then add boundary and adversarial cases. Include ordinary examples, incomplete requests, conflicting sources, wrong user permissions, duplicates, unusual file formats, mixed Arabic-English input, tool unavailability and requests outside scope.

Each case needs the expected outcome and prohibited outcome. For document checking, list required findings and false findings that would be harmful. For retrieval, identify the authoritative passage. For a draft, use a rubric for required claims, unsupported claims, tone and next action.

Keep a hidden holdout. Run the critical set on every configuration change. Version the model, prompt, knowledge index, tools and schemas together. A pass belongs to that complete configuration, not to the model brand.

An automated evaluator can help with structured fields and triage, but qualified people should review ambiguous and high-severity cases. Double-review a sample to calibrate the rubric. If reviewers disagree, clarify the underlying process before blaming the agent.

A 30-day human-in-the-loop validation

Days 1–5: contract and baseline. Select one task, name owners, write the charter, map data and permissions, establish the baseline and define measures. Collect an initial evaluation set.

Days 6–10: build the boundary. Configure approved retrieval, read-only tools, identity, trace capture and the review interface. Implement out-of-scope routing and deterministic field validation.

Days 11–15: offline evaluation. Run representative, boundary and critical cases. Diagnose whether failures originate in intake, retrieval, decision, tool or output. Fix the narrowest layer and repeat the full relevant suite.

Days 16–20: shadow operation. Process live cases without showing results or changing the workflow. Compare proposed outcomes with real resolutions. Check input coverage, latency, source quality and reviewer agreement.

Days 21–25: limited internal use. Release to a small trained cohort. Keep every external action behind approval. Hold daily review of failures and stop conditions. Do not expand the task during this period.

Days 26–30: evidence review. Compare with baseline, calculate cost per correct resolution, review incidents and open limitations, and decide to continue, repair, expand or stop. Expansion means one controlled change, such as a larger cohort or another document type—not simultaneous new tools, users and autonomy.

Monitor reviewer behaviour after day 30

Assign an operating cadence. Daily alerts cover critical events, integration failure and queues. Weekly review covers sampled outcomes, corrections, repeated failure clusters and source changes. Every release runs regression tests and receives owner sign-off. Monthly review considers value, drift and whether the boundary remains appropriate.

Keep a release manifest and an easy rollback path. Model providers, prompts, tools and documents change. If a change produces unexpected behaviour, the team must know what was running and restore the last accepted configuration.

Monitor distribution, not only averages. New departments may use different terminology. Ramadan schedules, seasonal demand or regulatory updates can change task shape. An aggregate score can remain stable while one important slice deteriorates.

Turn incidents into assets. Preserve a de-identified reproduction, root cause, repair and regression case. Close an incident only when the fix passes offline and production evidence shows the behaviour controlled.

Queue, latency and staffing choices

Use the smallest configuration that meets the task contract, but calculate total cost. A cheaper model can become expensive if it creates corrections or long prompts. A larger model may not solve missing documents or invalid tool schemas.

Cache stable retrieval where safe, constrain output structure, keep context relevant and route simple deterministic checks outside the model. Use a model for ambiguity and language, normal code for validation and permissions, and people for consequential judgement. This division is more reliable than asking one agent to improvise the whole process.

Avoid a platform architecture that makes the first deployment irreversible. Keep source connectors, model endpoints and tool interfaces sufficiently modular to test alternatives. Export traces and evaluations in usable formats. Ownership of operating evidence matters more than a polished demonstration.

Typical human-oversight failure modes

The scope expands during build. Every stakeholder adds a use case. Freeze the task contract for 30 days and queue additions.

The agent receives broad permissions for convenience. Start read-only and expose one allow-listed capability at a time.

The knowledge base has no owner. Retrieval quality cannot compensate for obsolete policy. Assign document stewardship.

Reviewers rubber-stamp. Show evidence and changed fields, sample approved work and measure corrections downstream.

Success is a demo. Measure accepted outcomes over real volume against a baseline.

Governance arrives after launch. A one-page charter, risk boundary, trace and stop rule should exist before the first live case.

The pilot never ends. Set a decision date and evidence thresholds. Continue, redesign, scale or stop.

The rule for relaxing or tightening oversight

Expand only after the existing boundary is stable and the organisation can explain its failures. Change one axis at a time: more users, another task, an additional data source or greater action authority. Each change alters risk and requires relevant evaluation.

Greater autonomy should follow evidence, not impatience. Drafting before sending, recommending before approving and read-only before write access are useful stages. Some workflows should remain human-approved permanently because the consequence, not the model capability, determines the correct control.

The first deployment succeeds when it leaves reusable capability: task contracts, evaluation cases, source governance, permission patterns, reviewer rubrics, traces, release gates and an incident loop. Even a decision not to scale can be valuable if it is based on credible evidence.

Where a system helps

Human oversight becomes governable when purpose, evidence, permissions, reviewer authority and decisions live in one operating record. CohortaOS for AI agents supports bounded agents, versioned runs, enforceable human decision points and evaluation queues, helping Saudi organisations turn oversight policy into a repeatable control.

FAQ

Does a first AI agent need executive committee approval?

Follow existing delegated authority. A low-risk internal, read-only, human-approved workflow may be approved by the responsible business, technology, privacy and security owners. Material decisions, sensitive data or autonomous actions require higher review.

What is the safest first use case?

A frequent internal task with approved inputs, objectively reviewable output and reversible action—such as checking document completeness or drafting a cited policy response—is usually safer than a customer-facing decision.

How long should the first deployment run?

Thirty days can provide an initial operating decision when volume is sufficient. Low-volume work may need longer. Define a target number of representative cases as well as a calendar period.

Should the first agent be allowed to update systems?

Usually not at first. Begin read-only, then introduce draft actions and explicit approval. Add bounded writes only after evaluation, validation, permissions, idempotency and rollback are proven.

How many use cases should be included?

One task family. Closely related variations are acceptable if they share evidence, tools, owner and success criteria. Separate processes should be separate deployments.

What evidence justifies scaling?

Stable correct-resolution and critical-failure results, lower cycle time or effort, acceptable cost, known limitations, effective review and the ability to reproduce, monitor and roll back every release.

Sources

• SDAIA AI Ethics Principles
• NIST AI Risk Management Framework 1.0
• NIST AI RMF Generative Artificial Intelligence Profile