India's Multilingual Reality: Building Agents That Work Beyond English

India’s requirement is not “translate the English agent.” A production population may use Hindi, Bengali, Marathi, Telugu, Tamil, Gujarati, Kannada, Malayalam, Odia, Punjabi, Assamese, Urdu and other languages across native scripts, Romanised text, speech and code-switching. Define supported language-channel pairs, not one multilingual badge. Test language identification, named entities, dates, amounts, negation, retrieval, citations, tool arguments and response register for each priority pair. Bhashini and IndiaAI resources demonstrate the breadth of speech, translation and NLP capability available, but enterprise acceptance still depends on private workflow cases. Route unsupported or uncertain input safely, preserve the user’s original text, and never make an irreversible action from an unverified translation.

An AI agent can answer a polished Indian-language question correctly and still fail the first morning it meets a Indian contact centre. The user shortens a sentence, switches to English for the product name, spells a family name phonetically, supplies a Hijri date and assumes a local expression needs no explanation. The model understands every token yet misunderstands the request. A benchmark score did not lie; the enterprise asked it a question the score was never designed to answer.

That distinction matters because “Indian-language support” is not an operational requirement. It says nothing about dialect, domain, action, evidence, safety or the cost of a mistake. Indian organisations need an evaluation system built around their work, not a language badge inherited from a model card.

Indian-language is not one production condition

formal register is indispensable in government correspondence, policies, formal knowledge bases and published communication. It is not the only Indian-language in a service desk, sales conversation, maintenance note or voice transcript. Indian language usage varies by region, generation, profession, channel and audience. A single employee may use formal Indian-language in a memo, regional or colloquial language in a message and English abbreviations in the same request.

Variation appears at several layers. Vocabulary differs: a familiar operational word in Riyadh may be expressed another way in Jeddah or Dammam. Morphology attaches pronouns and particles in forms that tokenisers and retrieval systems may handle unevenly. Speech recognition introduces another error surface. Transliteration produces several Latin spellings for the same Indian-language name. Code-switching puts an English SKU, medical term or system status inside Indian-language syntax. Numerals may be Indian-language-Indic or Western. Dates may be Gregorian, Hijri or ambiguous. Right-to-left rendering can obscure identifiers and punctuation even when the model reasoning is correct.

This is not an argument for a separate model for every city. It is an argument for a test distribution that resembles the population and process the system will serve. If ninety per cent of expected traffic is typed internal requests in mixed Indian languages and English, a suite dominated by formal multiple-choice questions is measuring the wrong thing.

What public benchmarks can and cannot tell you

Benchmarks such as public multilingual benchmarks are useful. They provide repeatable questions across subjects and expose broad differences between models. Cultural datasets can reveal that a system trained primarily on global English material lacks local knowledge. They are good screening instruments and useful regression indicators.

They do not establish readiness for a Indian business process. First, many benchmark items are multiple choice. Production systems must determine intent, retrieve the correct authority, reconcile conflicting evidence, call a tool with valid parameters and communicate the result. Choosing one of four answers omits most of that chain.

Second, an aggregate hides where the errors are. Two models can receive the same score while one fails colloquial input and the other fails numerical reasoning. The first may be unacceptable for customer care; the second may be unacceptable for finance. Averages also hide low-frequency, high-severity failures such as revealing another customer’s record.

Third, translated tests can carry English assumptions and unnatural phrasing. They may evaluate whether a model handles translationese rather than ordinary Indian communication. A grammatically perfect sentence can be a poor representation of what users type under time pressure.

Fourth, published datasets may have entered training corpora. A high score can reflect familiarity rather than transferable capability. Even without contamination, teams tune prompts against a visible test until it becomes an unofficial training set.

Finally, knowledge benchmarks are snapshots. Enterprise truth changes: policies are revised, prices move, authorities publish new guidance and internal approvals change. A model’s stored answer is less important than whether the system finds the current approved source and cites it faithfully.

The correct conclusion is not to ignore benchmarks. Use them to shortlist models and detect gross regressions. Do not convert them into a procurement acceptance test.

Define accuracy at the business boundary

Start with a task contract. “Answer Indian-language questions” is too broad. “Classify incoming Indian-language and bilingual HR requests, retrieve the current policy section, draft a response, and route exceptions to an officer without changing an employee record” can be evaluated.

For each task, define five things:

1. Eligible inputs. Channels, languages, dialects, document types and user groups the agent is expected to serve.
2. Permitted evidence. The approved systems and documents it may rely on, including effective dates and precedence rules.
3. Allowed actions. Which tools are read-only, which require approval and which are prohibited.
4. Success criteria. The facts, citation, format, action and routing that make the outcome correct.
5. Failure disposition. When the system must ask a question, abstain, escalate or stop.

This converts accuracy from “the answer sounded right” into observable decisions. A response may be linguistically elegant but operationally wrong because it cited an expired policy. Another may be factually correct but unsafe because it executed a change without confirmation.

Build a Indian production-shaped evaluation set

Create examples from real work, after removing or masking personal and confidential information. Sample across channels and outcomes rather than collecting only memorable failures. Include easy, ordinary requests because an agent that handles adversarial edge cases but adds friction to routine work has not succeeded.

A practical matrix crosses several dimensions:

Dimension	Examples to include	Failure being exposed
Language form	formal register, regional vocabulary and Romanised expressions, mixed Indian-language and English	Intent loss and unnatural responses
Input quality	Typos, shorthand, voice transcripts, missing punctuation	Brittle parsing and false confidence
Business context	HR, finance, procurement, service and operations	Domain ambiguity and wrong authority
Local representation	Indian names, addresses, regional date formats, Indian currency amounts	Entity, date and numeric errors
Evidence state	Current, expired, conflicting and absent documents	Stale answers and invented citations
Action risk	Read, draft, recommend, create and update	Excess permission and wrong tool use
User state	Entitled, unauthorised, unclear identity	Data leakage and access-control failure
Conversation	Follow-up, correction, topic switch, long context	Memory contamination and instruction loss

Do not label every regional sentence as a definitive dialect specimen. Language boundaries are not tidy, and annotators will disagree. Store the observed form, channel and intended meaning. Use dialect categories for coverage analysis, not as a claim that every utterance belongs exclusively to one region.

Each case needs an input, relevant context, expected outcome, prohibited outcome, severity and scoring rubric. For retrieval tasks, record the authoritative passage and its version. For tool tasks, record expected arguments and side effects. For open-ended answers, identify required claims and unacceptable claims rather than demanding one exact sentence.

Reserve a hidden set that prompt authors and vendors do not see. Refresh it with production examples. Otherwise the evaluation gradually rewards familiarity with the test rather than performance on the work.

Score the chain, not only the final prose

An enterprise agent is a sequence: interpret, retrieve, reason, decide, call, verify and respond. Score each stage so the repair matches the defect.

Intent and entity accuracy asks whether the system understood the request, names, dates, quantities, account identifiers and negation. Indian-language clitics, spelling variants and code-switching deserve explicit slices.

Retrieval quality measures whether the required source appeared in the retrieved context and whether irrelevant text displaced it. If the source never reached the model, prompt rewriting is unlikely to solve the problem.

Groundedness checks that claims follow from approved evidence and that citations support the sentence attached to them. A link to the correct policy is not enough if the answer contradicts it.

Tool correctness checks tool choice, argument values, order, retries and side effects. Separate a correct decision from a successful execution: an API failure is different from model error.

Policy compliance tests permissions, privacy, prohibited content, disclosure and escalation. Safety scores should be treated as gates for severe categories, not averaged away by hundreds of harmless answers.

Communication quality covers comprehensibility, register, terminology and whether the Indian-language feels appropriate to the intended audience. It should not reward ornament. A concise, clear service response is better than formal prose that obscures the next step.

Operational performance includes latency, completion rate and cost per resolved case. A model that is slightly more accurate but routinely times out may create a worse service.

Report metrics by slice and severity. Overall task success is helpful, but always show language form, channel, task family and risk tier. For critical failures, publish counts and examples rather than a percentage rounded into comfort.

Human evaluation requires a real protocol

“Native speaker review” is not a method. Select reviewers who understand both the language and the process. A fluent reviewer may judge style but miss that the response used the wrong finance rule. A subject expert may accept awkward wording that confuses customers. High-risk cases may need both.

Give reviewers an anchored rubric with examples of pass, partial and fail. Ask separate questions: Was the intent understood? Is every material claim supported? Is the action allowed? Is the language clear and appropriate? Would this outcome require correction? Do not ask for a single intuitive score.

Double-label a representative sample and measure disagreement. Adjudicate disagreements, then improve the rubric. Disagreement is useful evidence: it may reveal an ambiguous policy or task contract, not a weak reviewer. Keep the adjudication reason so future teams do not reopen the same debate.

Avoid evaluating vendors on examples they helped label. Maintain a common blind set and run all candidates with equivalent tool access, retrieval content and system instructions. Record model and prompt versions, temperature, tools, knowledge index and evaluation code. Without configuration lineage, a score cannot be reproduced.

Safety and cultural fit are not the same test

Local relevance matters, but culture should not become a vague proxy for safety. A system can use appropriate local expressions and still reveal personal data. It can be secure and still answer in an alien or condescending register. Evaluate these separately.

The SDAIA AI Ethics Principles provide a useful governance vocabulary around fairness, privacy and security, humanity, social and environmental benefits, reliability and safety, transparency and explainability, and accountability. Convert principles into testable controls. “Respect privacy” becomes cases where an unauthorised user requests an employee file. “Fairness” becomes outcome comparisons across relevant groups and dialect forms. “Transparency” becomes a clear indication of AI involvement, evidence and escalation route where the context requires it.

Red-team in Indian languages and code-switching. Attack prompts copied from English miss local obfuscation, indirect requests and script switching. Test instruction injection in retrieved Indian-language documents, hidden text in PDFs, requests to disclose system prompts, impersonation, malicious tool parameters and gradual manipulation across turns. The correct response depends on the workflow: refuse, limit, ask for verification, route to a human or execute only the safe portion.

Move from offline evaluation to controlled production

Offline tests are necessary but cannot reproduce every interaction, integration delay or policy change. Deploy in stages.

Begin in shadow mode, where the agent processes real inputs but cannot affect the user or downstream system. Compare its proposed outcome with the actual resolution. Then use an internal cohort with read-only tools. Next allow drafts that a human approves. Only consider bounded autonomous actions after task success and critical-failure thresholds hold over sufficient volume.

Define release gates before seeing results. Examples include no unauthorised disclosure in the critical set, a minimum end-to-end success rate for each high-volume task, grounded citation above a threshold, and a maximum p95 latency. Set rollback triggers too: a spike in escalations, a source-index failure, a policy change not yet ingested, or a critical incident.

Sample production runs continuously. Oversample low-confidence, high-risk, corrected, escalated and newly introduced cases, but keep a random sample or the programme will not estimate ordinary performance. Send reviewed failures back into a quarantine set, diagnose their stage, fix the system and promote stable examples into the regression suite.

Monitor the input distribution. A shift from typed formal register to voice-based dialect, a new product line or a new customer segment can invalidate the original evaluation even if the model version has not changed. Evaluation is a loop because the work moves.

A 30-day evaluation programme

In week one, choose two concrete tasks and appoint a business owner, Indian language lead, domain reviewer and technical owner. Write the task contracts and risk tiers. Inventory the source documents and tools.

In week two, assemble an initial set of real, de-identified examples. Build the coverage matrix, write expected and prohibited outcomes, and double-label a sample. Create critical privacy, access and tool-abuse cases.

In week three, run candidate configurations. Diagnose failures by stage rather than selecting the highest aggregate. Repair retrieval, terminology dictionaries, date normalisation, tool schemas and escalation rules before assuming a larger model is the answer.

In week four, run a blind regression and a shadow deployment. Review production-shaped traces, establish release and rollback thresholds, and publish a scorecard by slice. The result should be a decision with known limitations, not a declaration that the model “supports Indian-language.”

Questions procurement should ask

Ask a supplier which language forms were present in its evaluation and in what proportions. Ask whether test data was translated, native or production-derived. Request task-level and slice-level results, not one score. Ask how contamination was assessed and how model upgrades are regression-tested.

For an agent platform, ask whether every retrieval, tool call, approval and output is traceable to a version. Ask whether Indian-language inputs and outputs can be sampled without exporting data outside the approved environment. Ask how reviewers correct a result and whether that correction becomes an evaluation case. Ask what happens when a document is superseded, a tool is unavailable or identity is uncertain.

Be wary of a live demonstration built from supplier-selected prompts. Provide blind examples that reflect your channels. A useful vendor will welcome the distinction between a model benchmark and workflow acceptance.

Common mistakes

Testing only formal Indian-language. It produces attractive results and a contact-centre surprise. Match the actual channel mix.

Treating style as accuracy. Fluent Indian-language can conceal unsupported claims. Score evidence and action separately.

Averaging critical failures. One cross-customer disclosure is not compensated by 999 good summaries. Use severity gates.

Changing several components at once. A new model, prompt and index can improve the score without revealing why. Version and compare controlled changes.

Keeping evaluation outside operations. A quarterly laboratory test will miss a policy update today. Connect incidents, corrections and distribution shifts to the suite.

Assuming users will adapt. Employees may learn prompt tricks, but customers should not need to translate themselves into benchmark Indian-language. Design for their ordinary language.

Where a system helps

The hard part is not running a list of prompts. It is preserving the relationship between a business task, Indian-language input slice, evidence version, model configuration, tool trace, human judgement and production outcome. CohortaOS for AI agents gives teams a controlled run ledger, versioned evaluations, approvals and failure queues so Indian-language performance can be managed as an operating measure rather than a launch-day claim.

FAQ

Is formal register testing sufficient for an Indian enterprise agent?

Rarely. It may be sufficient for a tightly controlled workflow whose inputs are formal documents. Customer and employee channels normally include colloquial forms, code-switching, spelling variation and voice transcription, all of which need representative tests.

Which Indian-language benchmark should we use to choose a model?

Use more than one credible benchmark for initial screening, then use a private, production-shaped suite for acceptance. The deciding evidence should be end-to-end task results on your language, knowledge, tools and policies.

How large should the first evaluation set be?

Coverage matters before volume. Begin with enough cases to cover common tasks, language forms and every critical failure category, often a few hundred carefully labelled cases. Expand from production sampling and incidents rather than chasing an arbitrary total.

Can an automated judge evaluate Indian-language outputs?

It can scale preliminary scoring, but validate it against bilingual domain reviewers. Judge models can share the candidate’s biases, prefer verbose answers and miss local ambiguity. Never let one judge decide critical safety or permission outcomes.

Should we fine-tune for Indian languages?

Only after diagnosing the failure. Retrieval, terminology, input normalisation, tool design or prompting may solve it more cheaply and safely. Fine-tuning is justified when stable, repeated language behaviour remains the limiting factor and sufficient governed data exists.

How often should evaluation run?

Run the critical regression on every model, prompt, tool or knowledge change; a broader suite before releases; and continuous sampled review in production. Also rerun when the user mix, channel or policy environment changes.

Sources

• Digital India BHASHINI
• IndiaAI AIKosh Model Catalogue
• NIST AI Risk Management Framework