AI Agents in UAE Government-Facing Workflows: Where Human Approval Stays Mandatory

An AI assistant can give a plausible answer and still send a Gulf regulatory case down the wrong path. The user names a ministry informally, mixes Arabic and English, supplies a Hijri date, omits the legal entity and assumes the system knows whether the worker, company or transaction is mainland, free-zone or another jurisdiction. The model understands the words but lacks the case state required for a valid answer.

That distinction matters because language fluency is not regulatory competence. It says nothing about jurisdiction, effective date, authority, evidence, action rights or the cost of a mistake. Gulf organisations need a workflow built around authoritative rules and case data, not confidence inherited from a general model.

Government-facing automation begins with the case state

Modern Standard Arabic is indispensable in UAE government correspondence, policies and formal knowledge bases. It is not the only language condition in a service centre, company workflow or portal submission. Users mix Arabic and English, transliterate names differently and use authority names informally. A single case may combine formal Arabic evidence, English commercial records and identifiers entered through a bilingual interface.

Variation appears at several layers. Vocabulary differs: a familiar operational word in Riyadh may be expressed another way in Jeddah or Dammam. Morphology attaches pronouns and particles in forms that tokenisers and retrieval systems may handle unevenly. Speech recognition introduces another error surface. Transliteration produces several Latin spellings for the same Arabic name. Code-switching puts an English SKU, medical term or system status inside Arabic syntax. Numerals may be Arabic-Indic or Western. Dates may be Gregorian, Hijri or ambiguous. Right-to-left rendering can obscure identifiers and punctuation even when the model reasoning is correct.

This is not an argument for a separate model for every city. It is an argument for a test distribution that resembles the population and process the system will serve. If ninety per cent of expected traffic is typed internal requests in mixed Arabic and English, a suite dominated by formal multiple-choice questions is measuring the wrong thing.

Why a fluent answer is not an authorised submission

Benchmarks such as ArabicMMLU are useful. They provide repeatable questions across subjects and expose broad differences between models. Cultural datasets can reveal that a system trained primarily on global English material lacks local knowledge. They are good screening instruments and useful regression indicators.

They do not establish readiness for a Saudi business process. First, many benchmark items are multiple choice. Production systems must determine intent, retrieve the correct authority, reconcile conflicting evidence, call a tool with valid parameters and communicate the result. Choosing one of four answers omits most of that chain.

Second, an aggregate hides where the errors are. Two models can receive the same score while one fails colloquial input and the other fails numerical reasoning. The first may be unacceptable for customer care; the second may be unacceptable for finance. Averages also hide low-frequency, high-severity failures such as revealing another customer’s record.

Third, translated tests can carry English assumptions and unnatural phrasing. They may evaluate whether a model handles translationese rather than ordinary Saudi communication. A grammatically perfect sentence can be a poor representation of what users type under time pressure.

Fourth, published datasets may have entered training corpora. A high score can reflect familiarity rather than transferable capability. Even without contamination, teams tune prompts against a visible test until it becomes an unofficial training set.

Finally, knowledge benchmarks are snapshots. Enterprise truth changes: policies are revised, prices move, authorities publish new guidance and internal approvals change. A model’s stored answer is less important than whether the system finds the current approved source and cites it faithfully.

The correct conclusion is not to ignore benchmarks. Use them to shortlist models and detect gross regressions. Do not convert them into a procurement acceptance test.

Define the boundary between preparation and declaration

Start with a task contract. “Answer Arabic questions” is too broad. “Classify incoming Arabic and bilingual HR requests, retrieve the current policy section, draft a response, and route exceptions to an officer without changing an employee record” can be evaluated.

For each task, define five things:

1. Eligible inputs. Channels, languages, dialects, document types and user groups the agent is expected to serve.
2. Permitted evidence. The approved systems and documents it may rely on, including effective dates and precedence rules.
3. Allowed actions. Which tools are read-only, which require approval and which are prohibited.
4. Success criteria. The facts, citation, format, action and routing that make the outcome correct.
5. Failure disposition. When the system must ask a question, abstain, escalate or stop.

This converts accuracy from “the answer sounded right” into observable decisions. A response may be linguistically elegant but operationally wrong because it cited an expired policy. Another may be factually correct but unsafe because it executed a change without confirmation.

Build a UAE government-workflow evaluation set

Create examples from real work, after removing or masking personal and confidential information. Sample across channels and outcomes rather than collecting only memorable failures. Include easy, ordinary requests because an agent that handles adversarial edge cases but adds friction to routine work has not succeeded.

A practical matrix crosses several dimensions:

Dimension	Examples to include	Failure being exposed
Language form	MSA, Najdi and Hijazi expressions, mixed Arabic-English	Intent loss and unnatural responses
Input quality	Typos, shorthand, voice transcripts, missing punctuation	Brittle parsing and false confidence
Business context	HR, finance, procurement, service and operations	Domain ambiguity and wrong authority
Local representation	UAE names, addresses, dates, emirates and AED amounts	Entity, jurisdiction, date and numeric errors
Evidence state	Current, expired, conflicting and absent documents	Stale answers and invented citations
Action risk	Read, draft, recommend, create and update	Excess permission and wrong tool use
User state	Entitled, unauthorised, unclear identity	Data leakage and access-control failure
Conversation	Follow-up, correction, topic switch, long context	Memory contamination and instruction loss

Do not label every regional sentence as a definitive dialect specimen. Language boundaries are not tidy, and annotators will disagree. Store the observed form, channel and intended meaning. Use dialect categories for coverage analysis, not as a claim that every utterance belongs exclusively to one region.

Each case needs an input, relevant context, expected outcome, prohibited outcome, severity and scoring rubric. For retrieval tasks, record the authoritative passage and its version. For tool tasks, record expected arguments and side effects. For open-ended answers, identify required claims and unacceptable claims rather than demanding one exact sentence.

Reserve a hidden set that prompt authors and vendors do not see. Refresh it with production examples. Otherwise the evaluation gradually rewards familiarity with the test rather than performance on the work.

Validate identity, jurisdiction, evidence and effective date

An enterprise agent is a sequence: interpret, retrieve, reason, decide, call, verify and respond. Score each stage so the repair matches the defect.

Intent and entity accuracy asks whether the system understood the request, names, dates, quantities, account identifiers and negation. Arabic clitics, spelling variants and code-switching deserve explicit slices.

Retrieval quality measures whether the required source appeared in the retrieved context and whether irrelevant text displaced it. If the source never reached the model, prompt rewriting is unlikely to solve the problem.

Groundedness checks that claims follow from approved evidence and that citations support the sentence attached to them. A link to the correct policy is not enough if the answer contradicts it.

Tool correctness checks tool choice, argument values, order, retries and side effects. Separate a correct decision from a successful execution: an API failure is different from model error.

Policy compliance tests permissions, privacy, prohibited content, disclosure and escalation. Safety scores should be treated as gates for severe categories, not averaged away by hundreds of harmless answers.

Communication quality covers comprehensibility, register, terminology and whether the Arabic feels appropriate to the intended audience. It should not reward ornament. A concise, clear service response is better than formal prose that obscures the next step.

Operational performance includes latency, completion rate and cost per resolved case. A model that is slightly more accurate but routinely times out may create a worse service.

Report metrics by slice and severity. Overall task success is helpful, but always show Arabic form, channel, task family and risk tier. For critical failures, publish counts and examples rather than a percentage rounded into comfort.

Keep final declarations under named human authority

“Native speaker review” is not a method. Select reviewers who understand both the language and the process. A fluent reviewer may judge style but miss that the response used the wrong finance rule. A subject expert may accept awkward wording that confuses customers. High-risk cases may need both.

Give reviewers an anchored rubric with examples of pass, partial and fail. Ask separate questions: Was the intent understood? Is every material claim supported? Is the action allowed? Is the language clear and appropriate? Would this outcome require correction? Do not ask for a single intuitive score.

Double-label a representative sample and measure disagreement. Adjudicate disagreements, then improve the rubric. Disagreement is useful evidence: it may reveal an ambiguous policy or task contract, not a weak reviewer. Keep the adjudication reason so future teams do not reopen the same debate.

Avoid evaluating vendors on examples they helped label. Maintain a common blind set and run all candidates with equivalent tool access, retrieval content and system instructions. Record model and prompt versions, temperature, tools, knowledge index and evaluation code. Without configuration lineage, a score cannot be reproduced.

Separate portal credentials from agent permissions

Local relevance matters, but culture should not become a vague proxy for safety. A system can use appropriate Saudi expressions and still reveal personal data. It can be secure and still answer in an alien or condescending register. Evaluate these separately.

The SDAIA AI Ethics Principles provide a useful governance vocabulary around fairness, privacy and security, humanity, social and environmental benefits, reliability and safety, transparency and explainability, and accountability. Convert principles into testable controls. “Respect privacy” becomes cases where an unauthorised user requests an employee file. “Fairness” becomes outcome comparisons across relevant groups and dialect forms. “Transparency” becomes a clear indication of AI involvement, evidence and escalation route where the context requires it.

Red-team in Arabic and mixed language. Attack prompts copied from English miss local obfuscation, indirect requests and script switching. Test instruction injection in retrieved Arabic documents, hidden text in PDFs, requests to disclose system prompts, impersonation, malicious tool parameters and gradual manipulation across turns. The correct response depends on the workflow: refuse, limit, ask for verification, route to a human or execute only the safe portion.

Move from drafting to controlled submission carefully

Offline tests are necessary but cannot reproduce every interaction, integration delay or policy change. Deploy in stages.

Begin in shadow mode, where the agent processes real inputs but cannot affect the user or downstream system. Compare its proposed outcome with the actual resolution. Then use an internal cohort with read-only tools. Next allow drafts that a human approves. Only consider bounded autonomous actions after task success and critical-failure thresholds hold over sufficient volume.

Define release gates before seeing results. Examples include no unauthorised disclosure in the critical set, a minimum end-to-end success rate for each high-volume task, grounded citation above a threshold, and a maximum p95 latency. Set rollback triggers too: a spike in escalations, a source-index failure, a policy change not yet ingested, or a critical incident.

Sample production runs continuously. Oversample low-confidence, high-risk, corrected, escalated and newly introduced cases, but keep a random sample or the programme will not estimate ordinary performance. Send reviewed failures back into a quarantine set, diagnose their stage, fix the system and promote stable examples into the regression suite.

Monitor the input distribution. A shift from typed MSA to voice-based dialect, a new product line or a new customer segment can invalidate the original evaluation even if the model version has not changed. Evaluation is a loop because the work moves.

A 30-day controlled workflow programme

In week one, choose two concrete tasks and appoint a business owner, Arabic language lead, domain reviewer and technical owner. Write the task contracts and risk tiers. Inventory the source documents and tools.

In week two, assemble an initial set of real, de-identified examples. Build the coverage matrix, write expected and prohibited outcomes, and double-label a sample. Create critical privacy, access and tool-abuse cases.

In week three, run candidate configurations. Diagnose failures by stage rather than selecting the highest aggregate. Repair retrieval, terminology dictionaries, date normalisation, tool schemas and escalation rules before assuming a larger model is the answer.

In week four, run a blind regression and a shadow deployment. Review production-shaped traces, establish release and rollback thresholds, and publish a scorecard by slice. The result should be a decision with known limitations, not a declaration that the model “supports Arabic.”

Evidence, logging and exception questions to answer

Ask a supplier which Arabic forms were present in its evaluation and in what proportions. Ask whether test data was translated, native or production-derived. Request task-level and slice-level results, not one score. Ask how contamination was assessed and how model upgrades are regression-tested.

For an agent platform, ask whether every retrieval, tool call, approval and output is traceable to a version. Ask whether Arabic inputs and outputs can be sampled without exporting data outside the approved environment. Ask how reviewers correct a result and whether that correction becomes an evaluation case. Ask what happens when a document is superseded, a tool is unavailable or identity is uncertain.

Be wary of a live demonstration built from supplier-selected prompts. Provide blind examples that reflect your channels. A useful vendor will welcome the distinction between a model benchmark and workflow acceptance.

Common UAE government-workflow mistakes

Testing only formal Arabic. It produces attractive results and a contact-centre surprise. Match the actual channel mix.

Treating style as accuracy. Fluent Arabic can conceal unsupported claims. Score evidence and action separately.

Averaging critical failures. One cross-customer disclosure is not compensated by 999 good summaries. Use severity gates.

Changing several components at once. A new model, prompt and index can improve the score without revealing why. Version and compare controlled changes.

Keeping evaluation outside operations. A quarterly laboratory test will miss a policy update today. Connect incidents, corrections and distribution shifts to the suite.

Assuming users will adapt. Employees may learn prompt tricks, but customers should not need to translate themselves into benchmark Arabic. Design for their ordinary language.

Where a system helps

The hard part is preserving the relationship between a government-facing task, jurisdiction, case evidence, rule version, agent configuration, tool trace, authorised human approval and portal outcome. CohortaOS for AI agents provides a controlled run ledger, versioned rules, approvals and exception queues so UAE workflows can be managed as accountable operations rather than unattended browser automation.

FAQ

Is Modern Standard Arabic testing sufficient for a Saudi enterprise agent?

Rarely. It may be sufficient for a tightly controlled workflow whose inputs are formal documents. Customer and employee channels normally include colloquial forms, code-switching, spelling variation and voice transcription, all of which need representative tests.

Which Arabic benchmark should we use to choose a model?

Use more than one credible benchmark for initial screening, then use a private, production-shaped suite for acceptance. The deciding evidence should be end-to-end task results on your language, knowledge, tools and policies.

How large should the first evaluation set be?

Coverage matters before volume. Begin with enough cases to cover common tasks, language forms and every critical failure category, often a few hundred carefully labelled cases. Expand from production sampling and incidents rather than chasing an arbitrary total.

Can an automated judge evaluate Arabic outputs?

It can scale preliminary scoring, but validate it against bilingual domain reviewers. Judge models can share the candidate’s biases, prefer verbose answers and miss local ambiguity. Never let one judge decide critical safety or permission outcomes.

Should we fine-tune for Saudi dialect?

Only after diagnosing the failure. Retrieval, terminology, input normalisation, tool design or prompting may solve it more cheaply and safely. Fine-tuning is justified when stable, repeated language behaviour remains the limiting factor and sufficient governed data exists.

How often should evaluation run?

Run the critical regression on every model, prompt, tool or knowledge change; a broader suite before releases; and continuous sampled review in production. Also rerun when the user mix, channel or policy environment changes.

Sources

• Qatar MCIT, Artificial Intelligence Committee and Ethical AI Guidance
• NIST AI RMF Generative AI Profile
• NIST AI Risk Management Framework