Qatar's Personal Data Privacy Protection Law and AI: Obligations for Deployers

Operational control map

Use this map when translating the guide into system configuration or procedure. Replace every placeholder and add jurisdiction-specific rows before approval.

Control area	Minimum requirement	Owner	Evidence
Scope and applicability	Confirm entity, jurisdiction, activity and effective date.	[ASSIGN]	[EVIDENCE LINK]
Authoritative requirement	Link the current source from PDPPL, NCSA.	[ASSIGN]	[EVIDENCE LINK]
Master data	Define fields, identifiers and ownership.	[ASSIGN]	[EVIDENCE LINK]
Workflow control	Record submission, approval, rejection and correction states.	[ASSIGN]	[EVIDENCE LINK]
Evidence	Retain source documents, acknowledgements and versions.	[ASSIGN]	[EVIDENCE LINK]
Exception handling	Assign escalation, resolution target and authority.	[ASSIGN]	[EVIDENCE LINK]
Periodic review	Set an owner and regulatory review date.	[ASSIGN]	[EVIDENCE LINK]

Treat this as a maintained record. Store source publication, internal approval and next review dates, and preserve prior versions whenever a rule or workflow changes.

“Hosted in the UAE” answers one architecture question: where specified data and compute are located under the service design. Sovereign AI adds other questions about legal jurisdiction, operational administration, privileged access, encryption keys, software supply chain, support, continuity and the customer's ability to verify controls. Core42's offerings make stronger local options available; procurement still has to test the exact workload and service boundary.

Most teams evaluate an AI agent twice: during the vendor demonstration and just before launch. After launch they watch uptime, token cost and complaints. The system can remain available, cheap and quietly wrong for months.

Production accuracy is difficult because an agent is not a single prediction. It interprets a request, retrieves information, chooses tools, supplies arguments, observes results, may retry, and produces an answer. The same final sentence can conceal a failed permission check or an unintended write. A useful evaluation programme therefore measures complete work, retains the trace and connects every observed failure to a controlled improvement.

Separate residency from sovereignty

Accuracy is straightforward when one input has one label. Agent work often has several valid paths and many invalid side effects. Define an outcome contract for each production task.

For example, an accounts-payable agent may receive an invoice, match it to a purchase order, identify discrepancies and draft a posting. Success is not “the summary is correct.” It requires the correct legal entity, supplier, tax fields, currency, PO lines, tolerance rule, duplicate check, evidence links and approval route, with no posting before approval.

Every contract should identify eligibility, required evidence, permitted tools, required fields, prohibited outcomes, completion evidence and escalation conditions. Tasks outside the contract are not failures if the agent correctly refuses or routes them. Conversely, a confident answer to an ineligible request is a failure even if its facts happen to be right.

Use a small metric hierarchy:

• End-to-end task success: all required conditions passed and no prohibited condition occurred.
• Correct resolution rate: the task was completed or correctly escalated.
• Critical failure rate: privacy, security, financial, legal or irreversible failures per eligible run.
• Human correction rate: completed outputs that a reviewer materially changed.
• Automation yield: eligible runs completed without human intervention.
• Cost and latency per correct resolution: operational efficiency after quality, not before it.

Keep task success binary where possible. Diagnostic scores can be graded, but a partially correct bank account or partially compliant access decision is not a successful outcome.

Why a local region is necessary but insufficient

An offline suite provides repeatability. It is where teams test known edge cases, compare versions and block obvious regressions. Benchmarks such as AgentBench and GAIA helped make multi-step tool use and assistant work more visible. They remain general tests in controlled environments.

Production adds conditions the suite did not model: incomplete master data, slow APIs, permission differences, concurrent updates, ambiguous employee language, policy revisions and users who correct themselves midway. It also changes the distribution. After users discover what the agent can do, they ask different questions. Automation alters the process it measures.

Offline and online evaluation are complementary. Offline evaluation answers, “Can this version pass the cases we know?” Production evaluation asks, “What work is it actually receiving, what outcomes is it producing, and what failures have we not imagined?” The loop joins those questions.

Map data, model, administrator and support access

If a team stores only prompts and final responses, it cannot diagnose an agent. A production trace needs enough lineage to reconstruct the decision without exposing unnecessary sensitive data.

Record a run identifier; task and risk class; user and entitlement context; model, system prompt and orchestration versions; retrieved document identifiers and versions; tool requests, arguments, responses and side effects; approvals; final output; latency and cost; and user or reviewer corrections. Redact secrets and minimise personal data. Apply role-based access and retention periods to traces.

Separate model reasoning from auditable evidence. Hidden chain-of-thought is not required and should not become the control record. The useful record is what entered the system, what evidence it retrieved, which actions it requested, what tools returned, which rules fired and who approved the outcome.

Log failures from dependencies as first-class outcomes. “Agent failed” is too coarse. Distinguish retrieval miss, stale source, malformed argument, permission denial, tool timeout, contradictory data, model decision error and rendering defect. Otherwise every improvement discussion collapses into changing the prompt.

Verify where telemetry, backups and derived data go

Reviewing only complaints underestimates errors because many users cannot recognise a plausible wrong answer. Reviewing only escalations overestimates difficulty because successful routine cases disappear. Use several streams.

A random sample estimates ordinary performance. A risk-weighted sample covers high-impact tasks. A triggered sample captures low confidence, retries, refusals, unusual tool sequences, long latency, user corrections and policy exceptions. A change sample focuses on new models, prompts, tools, document collections and user cohorts. An incident sample preserves everything around suspected harm.

Define sampling rates by task and risk. A low-risk internal summary may receive a small random sample. A payment instruction or access decision may require deterministic validation plus human approval on every run. Do not claim one “production accuracy” number across both.

UAE enterprise architecture adds sovereignty-relevant variation: mainland and free-zone entities, regulated sectors, Arabic and English data, cloud regions, administrator locations, network routes and local master-data conventions. Assess each workload and service configuration rather than assuming one residency claim covers the full portfolio.

Contract key control and privileged administration

Human review is often treated as temporary labour until the model becomes good enough. In a mature system it is a controlled measurement function.

Create rubrics directly from the task contract. Reviewers should answer observable questions: Was the correct source used? Were required fields extracted? Was the tool permitted? Did arguments match the source? Was approval obtained? Was the final communication faithful? Was escalation required?

Assign domain reviewers, not generic annotators, where rules matter. Train them with anchor cases. Double-review a sample and adjudicate disagreements. Record why the final label was chosen. If qualified reviewers disagree frequently, the policy or rubric may be ambiguous; do not hide that with majority voting.

Measure reviewer quality too: agreement, overturned decisions, queue time and reasons for uncertainty. Rotate experts through calibration sessions. Protect reviewers from seeing the candidate version when blind comparison is possible.

For large volumes, automated judges can prioritise cases and score constrained dimensions. Validate each judge against human decisions, by task and failure class. A judge that correlates well overall may miss the exact critical failures that matter. It must never be the sole arbiter of unauthorised disclosure, discrimination, payment or other severe outcomes.

Separate platform, model and application responsibilities

An end-to-end failure needs a root-cause label. A useful taxonomy includes:

Stage	Typical defect	Appropriate repair
Intake	Wrong intent or entity	Input schema, clarification, language handling
Retrieval	Correct source absent	Indexing, metadata, filters, chunking
Evidence	Outdated or conflicting source	Governance, precedence, effective dates
Decision	Evidence present but conclusion wrong	Instructions, examples, model or deterministic rule
Tool selection	Wrong or excess tool	Capability boundaries and routing
Arguments	Missing, swapped or invented values	Typed schema, validation, confirmation
Execution	API, identity or concurrency failure	Integration reliability and idempotency
Oversight	Approval skipped or ineffective	Enforced workflow and separation of duties
Response	Action right, explanation wrong	Grounded response template

This prevents model upgrades becoming the universal remedy. If retrieval never returned the approved circular, a larger model may hallucinate more persuasively. If a tool accepted an invalid account code, schema validation is a safer fix than prompt wording.

Turn sovereignty claims into testable controls

The skipped step is usually here. A reviewer corrects a run, operations closes the ticket, and the evidence disappears. Instead, every material failure enters a controlled queue.

Triage it by severity and recurrence. Preserve a de-identified input, relevant context, expected result, prohibited result, configuration and root cause. Reproduce the failure offline. Add it to a quarantine set first. Implement the narrowest repair, then run nearby cases to detect overfitting. Once stable, promote it into the regression suite.

Cluster repeated failures. Twenty individual corrections may reflect one missing policy version or one misunderstood field. Track recurrence after repair. Close the issue only when the regression passes and production evidence confirms the failure rate has returned below threshold.

Not every unusual case belongs in the permanent suite. Retain cases that represent a risk, common pattern, requirement or previously observed regression. Archive duplicates while preserving counts. A test collection that grows without curation becomes slow, contradictory and easy to ignore.

Version the complete service configuration

Model version alone does not define an agent. Prompts, tool descriptions, schemas, routing, retrieval index, source documents, safety policies and UI confirmations all affect outcomes.

Create a release manifest for every production configuration. Link each evaluated run and production trace to it. If a managed model provider changes a model behind an alias, detect and record the change where the provider exposes it; otherwise treat unexplained distribution shifts as release risk.

Run evaluation in layers. A fast critical suite runs on every meaningful change. Task suites run before deployment. A comprehensive suite runs on scheduled releases. Shadow traffic and canary cohorts test the combined system under actual conditions.

Compare candidate and incumbent on identical cases. Report wins, losses and critical regressions, not only average improvement. A version that improves drafting by five points while introducing one unauthorised action should not pass.

Establish approval gates, portability and rollback

Write thresholds before examining a favoured candidate. Gates can include zero critical permission failures in the critical set, minimum task success by task family, maximum unsupported-claim rate, maximum correction rate, latency bounds and cost per correct resolution.

Use risk-based gates. Lower-risk drafting can tolerate reviewable imperfections. External communications, payroll, payments or entitlement changes require stronger controls. Some tasks should remain recommendation-only regardless of aggregate accuracy.

Deploy progressively: shadow, internal users, a small canary, broader cohorts, then increased autonomy. Compare canary and control on matched tasks. Freeze unrelated changes during diagnosis. Keep the previous manifest deployable and define who can roll back, on what trigger and how quickly.

Rollback triggers should include critical incidents, sudden failure-rate changes, unavailable authoritative data, unusual tool denial rates and monitoring blind spots. If trace capture stops, the safe state may be reduced autonomy because the organisation can no longer demonstrate control.

Detect region, subprocessor and service drift

Model drift is only one kind. Input drift occurs when users, languages, channels or task mix change. Knowledge drift occurs when authoritative documents change. Tool drift occurs when an API or schema changes. Policy drift occurs when the approved business rule changes. Outcome drift occurs when downstream consequences change even though immediate agent metrics appear stable.

Monitor task volumes, language and channel mix, retrieval source distribution, refusal and escalation rates, tool sequence frequencies, argument validation failures, corrections and business outcomes. Compare with the evaluation set. A major production slice with little test coverage creates a coverage alert.

Tie document lifecycle to evaluation. When a policy is superseded, identify affected tasks, refresh expected evidence and run their suite before the new version becomes authoritative. Stale-answer tests should remain to verify that the agent does not retrieve the old rule.

Connect sovereignty requirements to workload value

An agent can achieve a high technical score and fail commercially. Measure the process baseline before launch: handling time, queue age, rework, error cost, completion rate and customer or employee effort.

Then calculate value on correct resolutions. Cost per attempted run flatters a system that creates downstream repair. Include model and platform cost, review time, integration operations, incident handling and rework. Track cycle time from request to accepted outcome, not merely model response latency.

Watch displacement. A fast agent may move work from one team to another by increasing exception quality problems. Follow the transaction through reconciliation, complaint or closure. The business owner should sign the success definition and review results with engineering and risk.

Sovereignty governance that does not become theatre

The NIST AI RMF organises activity around Govern, Map, Measure and Manage. Applied practically, governance names the owner and tolerances; mapping defines the task, people and impact; measurement runs the evaluation; management changes, limits or retires the system.

Keep a concise operating record for each agent: purpose, owner, eligible users, data, tools, risk tier, evaluation summary, open limitations, approval mode, release manifest and incident route. A committee does not need to inspect routine runs. It should approve risk appetite, review trends and decide exceptions that exceed delegated authority.

Publish a weekly operational scorecard for owners and a periodic risk report for governance. Include denominators and confidence intervals where appropriate. Do not turn ten reviewed cases into a decimal-point accuracy claim. Show examples of important failures and corrective status.

A workable supplier-assurance cadence

Daily operations should review critical alerts, tool anomalies and blocked queues. Weekly review should adjudicate samples, cluster failures and compare task metrics. Each release should run the critical and relevant task suites with signed gates. Monthly review should examine drift, value, reviewer performance and open limitations. Quarterly review should revisit task scope, risk classification and whether autonomy should expand, contract or stop.

The cadence must match volume. A low-volume, high-impact process may require case review rather than statistical inference. A high-volume support agent can support stable estimates but still needs complete review of severe events.

UAE sovereign-AI procurement checklist

1. Inventory production agents and split them into evaluable tasks.
2. Name a business owner and technical owner for every task.
3. Define eligible, successful, prohibited and escalated outcomes.
4. Create risk tiers and critical-failure categories.
5. Instrument versioned evidence, tools, approvals and results.
6. Establish random, triggered, risk and change samples.
7. Write anchored reviewer rubrics and calibration cases.
8. Build a private offline set plus a hidden holdout.
9. Define release gates and rollback triggers in advance.
10. Route production failures into quarantine and regression.
11. Monitor input, knowledge, tool, policy and outcome drift.
12. Report cost and cycle time per correct resolution.

Common sovereignty-assessment traps

Thumbs-up rate: voluntary feedback is sparse and biased toward strong reactions. Use it as a signal, not an accuracy estimate.

Answer similarity: a different but valid resolution may look dissimilar, while a copied wrong answer looks close. Evaluate requirements and effects.

One aggregate score: averages conceal the task and severity that determine risk. Publish slices.

No denominator: “three incidents” means little without eligible volume, exposure and detection method.

Review after launch only: without a baseline, manifest and release gate, production review cannot attribute changes.

Optimising automation rate: forcing fewer escalations can raise unsafe completion. Correct resolution is the goal; automation yield is conditional.

Where a system helps

The evaluation loop fails when traces live in engineering logs, corrections in tickets, tests in notebooks and approvals in email. CohortaOS for AI agents connects a versioned run to its evidence, tool actions, reviewer decision, failure record and regression case, giving Indian and global teams a defensible path from production observation to controlled release.

FAQ

What is a good accuracy target for an AI agent?

There is no universal target. Set thresholds by task and consequence. Low-risk drafting and high-risk financial actions need different gates, and severe permission or privacy failures may require zero tolerance in the critical suite.

How do we measure accuracy when several answers are valid?

Score required facts, approved evidence, permitted actions, prohibited outcomes and successful task completion. Use an anchored human rubric for communication quality instead of exact-text matching.

How much production traffic should humans review?

Base it on volume, risk and the precision required. Combine a statistically useful random sample with complete or elevated review of high-risk, changed, corrected and anomalous runs.

Are user complaints enough to identify failures?

No. Users may not detect plausible errors, may silently redo the work or may not report. Complaints are one triggered stream alongside random review and deterministic controls.

When should an agent be rolled back?

Roll back or reduce autonomy when a critical failure occurs, metrics cross predefined limits, authoritative evidence is unavailable, or observability is impaired. Decide triggers and authority before deployment.

Can we use another model as the evaluator?

Yes, for scale and prioritisation after validating it against qualified humans. Keep human adjudication for uncertain and high-severity outcomes, and monitor judge performance by failure class.

Sources

• NIST AI Risk Management Framework 1.0
• NIST AI RMF Generative AI Profile
• Core42, Sovereign Public Cloud
• Core42, AI Cloud Platform