A Comprehensive Evaluation Protocol for Vision–Language Models in Diverse Document Ecosystems
With over 1.4 billion people and dozens of widely used scripts, India represents one of the world's most linguistically and visually diverse document ecosystems. Yet document understanding benchmarks remain largely Western- and English-centric. Consequently, the robustness of modern vision–language models (VLMs) in multilingual and low-resource settings remains under-evaluated.
We introduce the first large-scale multilingual benchmark for Indian document understanding, comprising 52,607 documents across 23 languages, 13 domains, 71 document classes, and 52 tasks, with over 2.4M QA annotations across multiple tasks. The dataset captures diverse layouts, scripts, and visual conditions spanning governance, education, finance, archival, and informal content. We additionally include synthetic multilingual charts, tables, and diagrams generated via Patram-Syn, an Indian persona-driven data generation pipeline with human validation.
The benchmark measures perception-level capabilities (OCR, parsing, key information extraction, layout detection) and reasoning-level capabilities (Document VQA with abstractive, multi-hop, and ambiguous questions). We further introduce targeted settings for cross-lingual, transliterated, and code-mixed VQA, and propose two new evaluation metrics — DUCLI and SRS — to quantify cross-lingual reasoning degradation and script robustness.
Real Indian documents across 13 domains and 71 document classes — governance, education, finance, archival, and more.
QA annotations spanning 52 tasks across perception and reasoning, including cross-lingual and code-mixed settings.
From high-resource Hindi/Bengali to low-resource Santali/Kashmiri — exposing critical gaps in multilingual VLM robustness.
A hybrid curation strategy combining native Indian documents and persona-driven synthetic data.
Government notifications, circulars, and forms sourced from national portals across multiple states and languages.
Competitive examination papers, school textbooks, and university academic theses across Indic mediums.
Annual business reports, tax filings, and regulatory filings from BSE & NSE in regional languages.
Scan-rich regional newspapers, historic novels, advertisements, and recipe books spanning decades.
Document domain and class distribution: Spans 13 high-level domains and 71 fine-grained document classes.
We curate a large-scale corpus of Indian PDF documents reflecting real-world document ecosystems. Sources span official governance portals, educational boards, listed corporations, and archival databases — capturing the full diversity of Indian document production.
The corpus covers handwritten, printed, and mixed-modality documents, with degraded scans and multi-column layouts that stress-test VLM visual grounding.
To evaluate VLM performance under strict layout controls, we introduce Patram-Syn, a persona-driven synthetic data generation pipeline. By conditioning on 200,000 synthesized Indian personas, we produce realistic, culturally grounded tables, diagrams, and charts.
All synthetic samples undergo human verification by 14 native-speaker experts reviewing layout, script rendering, and QA annotation quality.
Patram-Syn overview: A five-stage persona-driven generation pipeline for synthetic Indian documents. Press play to walk through each stage.
Content conditioned on roles like educators, public health officers, and accountants from 200k Indian personas.
Retrieves background from public Indian databases keeping visual assets factually grounded.
Outputs Matplotlib charts, LaTeX tables, and Mermaid diagrams with full Indic font support.
14 native-speaker experts review layout, script rendering, and QA annotation correctness.
Patram-Bench spans 52 distinct document understanding tasks across perception and reasoning.
Task taxonomy: Real examples pulled live from the Patram-Bench dataset — evaluation axes across text recognition, document segmentation, element parsing, and multi-lingual VQA. Click a family chip to filter, or switch to Explore for a focused per-family view. Click any card to inspect the full example.
Large-scale unified evaluation of 29 vision-language models and specialized document processors.
Performance across English (en), five high-resource Indic languages (hi, bn, te, ta, mr), and five low-resource Indic languages (mni, sat, sd, ks, kok). The degradation slope from left to right is the robustness story. Select models using the chips below — or switch to the Radar sub-tab to pin up to 4 models and compare their full language profile at a glance.
Detailed model performance across 8 task categories. Radar compare overlays up to 4 models; the heatmap surfaces outliers (e.g. Pixtral EP = 0.15 near-black cell).
Best, best open-source, and worst performing models per task group.
| Type | TR | DS | DP | EP | TDV | SER | AKR | LR |
|---|---|---|---|---|---|---|---|---|
| Best | G3P | iDLP | C4S | G3P | C4S | GPT5 | GPT5 | G3P |
| Best OS | Chan | iDLP | Mrkr | QW32 | QW32 | QW235 | GLM4 | QW30 |
| Worst | LLaVA | PPDoc | LLaVA | Pix12 | LLaVA | IVL8 | LLaVA | Mol8 |
TR: Text Recognition • DS: Document Segmentation • DP: Document Parsing • EP: Element Parsing • TDV: Text-Centric DocVQA • SER: Structured Element Reasoning • AKR: Analytical/Knowledge Reasoning • LR: Linguistic Robustness
Question and document language mismatch settings.
Average Normalized Levenshtein Similarity (ANLS) is the standard evaluation metric for Document VQA. Unlike exact-match (which is overly strict on minor typos or OCR scan noise), ANLS measures character-level edit distance to tolerate slight transcription noise.
where $a$ is the model answer, $o$ is the ground-truth, and $\text{NLD}(a, o) = \frac{\text{Levenshtein}(a, o)}{\max(|a|, |o|)}$ is the Normalized Levenshtein Distance. Any edit distance $\ge 50\%$ yields a score of 0.
Document Understanding Cross-Lingual Index and Script Robustness Score, decomposed into the settings each metric is built from. Sort either chart, and hover a bar or table row to link the two.
Does the model actually read the page, or guess from language priors? Ranking is based on how much accuracy survives when the document image is removed at inference time — the smaller the drop, the more the model was actually looking at the page.
Three independent results on this page point to the same conclusion. Cross-script robustness (Language Robustness, above) tracks script diversity and cross-lingual supervision, not parameter count — the strongest low-resource model isn't the largest one. Visual grounding (VL Reliance, above) tracks architectural balance, not scale — the mid-sized Qwen3-VL models stay more image-grounded than their 235B sibling. And the largest models lean hardest on language priors rather than the page in front of them. Read together, these aren't three separate observations — they're one critique of scale-maximalism as a strategy for multilingual document understanding.
If you find our work or dataset useful in your research, please consider citing:
@inproceedings{patrambench2026,
title = {Patram-Bench: A Multi-task, Multi-domain and Multi-lingual
Benchmark for Indian Document Image Understanding},
author = {Srinivasan, Anirudh and Jena, Pratyush and Topale, Arya
and Venna, Venkat and Sarvadevabhatla, Ravi Kiran},
booktitle = {Proceedings of the International Conference on Document
Analysis and Recognition (ICDAR)},
year = {2026},
}