By the Monlam AI Engineering Team, November 2025
Language is more than a tool for communication—it carries culture, history, philosophy, and identity. Tibetan preserves centuries of intellectual and spiritual knowledge, yet modern AI systems have historically focused on high-resource languages, leaving Tibetan and many other low-resource languages underserved.
Melong 2.0 improves Tibetan benchmark performance by +10.0 percentage points over its Gemma 3 foundation model while retaining approximately 94% of its general capabilities. The model also preserves—and on some benchmarks improves—long-context performance up to 130,000 tokens, demonstrating that deep Tibetan specialization can be achieved without sacrificing broad reasoning performance.
Melong 2.0 was developed to address this gap. Built by the Monlam AI Engineering Team in Dharamshala, India, the model represents a shift from simple language adaptation toward a specialized architecture optimized for Tibetan linguistic understanding, cultural alignment, retrieval-grounded reasoning, and long-context reading.
| Component | Specification |
|---|---|
| Foundation Model | Gemma 3 12B |
| Training Strategy | Tibetan Continual Pretraining |
| Corpus Size | 22B+ Tokens |
| Context Length | 130K Tokens |
| Tibetan Benchmark Gain | +10.0pp (BoBench) |
| General Capability Retention | 94% |
| Retrieval Support | Retrieval-Augmented Generation (RAG) |
Rather than pursuing larger parameter counts alone, development focused on solving three fundamental challenges that affect low-resource language AI systems.
Challenge 1: Continual Learning Without Forgetting
A major risk when specializing a model for a single language is catastrophic forgetting. As a model becomes deeply optimized for Tibetan, it can lose capabilities in mathematics, reasoning, coding, translation, and multilingual understanding.
Our Solution: Specialized Continual Pre-training
Melong 2.0 uses a continual pretraining approach designed to strengthen Tibetan linguistic understanding while preserving the broader reasoning capabilities inherited from its foundation model.
The training process exposes the model to both Tibetan and multilingual data throughout specialization, allowing it to internalize Tibetan syntax, vocabulary, and cultural context without substantially degrading the underlying reasoning abilities that make modern language models useful across a wide range of tasks.
General Capability Retention
| Category | Retention |
|---|---|
| Machine Translation | 101% |
| Commonsense & Science | 100% |
| Math & Reasoning | 92% |
| Expert Knowledge QA | 89% |
| Reading Comprehension QA | 89% |
Overall, Melong 2.0 retains approximately 94% of the baseline model's general capabilities while delivering substantial gains in Tibetan understanding.
Challenge 2: Data Scarcity and Domain Imbalance
Digital Tibetan resources remain limited compared with major world languages. Existing data is heavily concentrated in classical literature, historical archives, and religious texts, creating significant gaps in modern domains such as science, technology, medicine, mathematics, and everyday conversation.
Our Solution: Corpus Normalization & Synthetic Diversification
To address these limitations, the Monlam AI team curated a corpus containing more than 22 billion tokens and implemented two complementary strategies.
The corpus combines classical literature, historical materials, modern educational content, multilingual resources, and synthetically generated training data designed to address gaps in contemporary Tibetan usage. This combination allows the model to remain grounded in traditional linguistic sources while expanding coverage across modern domains.
Classical Corpus Normalization
A large-scale automated preprocessing pipeline standardizes Tibetan Unicode text, removes encoding artifacts, resolves digitization inconsistencies, and improves corpus quality for large-scale pre-training.
Synthetic Data Generation
Custom synthetic generation pipelines create modern Tibetan training data covering:
These datasets help bridge vocabulary gaps and expose the model to linguistic patterns rarely found in historical corpora.
Challenge 3: Compute Efficiency and Factual Grounding
Training extremely large language models requires enormous computational resources and often reduces accessibility for local deployment.
Our Solution: A Lean Reader Grounded in Retrieval
Instead of relying primarily on parameter memorization for factual recall, Melong 2.0 was designed as a highly capable reader paired with Retrieval-Augmented Generation (RAG).
This strategy focuses on:
The result is a system that remains lightweight, improves factual grounding by retrieving information from source documents at generation time, lowers training costs, and can run efficiently on consumer-grade GPUs.
Baseline evaluation across multiple candidate foundation models showed that Gemma 3 12B possessed the strongest native Tibetan capabilities prior to specialization. As illustrated in the comparison benchmarks below, it provided the strongest starting point for continual pretraining and subsequent adaptation.
Melong 2.0 was therefore built on Gemma 3 and subsequently specialized through Tibetan continual pre-training before integration into the retrieval pipeline.
Base Model Selection: Accuracy on Tibetan Benchmark
Qwen3-14B
Qwen3-32B
Gemma 3 12B
Gemma 3 27B
One of the largest limitations of Melong 1.0 was its approximately 2,000-token context length, restricting long conversations and large-scale document understanding.
Melong 2.0 inherits the long-context architecture of Gemma 3 and undergoes additional long-context training during specialization. A key objective was ensuring that Tibetan adaptation did not compromise the model's ability to retrieve, track, and reason over information distributed across extremely long sequences.
The resulting model supports context windows of up to 130,000 tokens, enabling book-length document analysis, multi-document reasoning, large retrieval contexts, and extended research workflows.
Long-Context Evaluation
| Benchmark | Gemma 3 12B | Melong 2.0 | Delta |
|---|---|---|---|
| LongBench v2 | 30.4% | 29.4% | -1.0pp |
| RULER (120K) | 57.4% | 63.3% | +5.9pp |
Evaluation on LongBench v2 and RULER (120K) demonstrates that long-context performance remains largely preserved after specialization. Notably, Melong 2.0 exceeds the foundation model on RULER, indicating that Tibetan adaptation did not merely preserve long-context capability, but improved performance on challenging long-range retrieval tasks.
To evaluate Tibetan language capabilities accurately, the Monlam AI team developed and benchmarked against BoBench v1.0.0, a Tibetan evaluation suite created in collaboration with 20 scholars, linguists, and historians.
BoBench evaluations were conducted using a standardized 5-shot evaluation configuration across all compared models to ensure consistency and comparability.
The benchmark contains more than 6,000 original questions spanning:
Tibetan Performance Gains
| Topic | Base | Melong 2.0 | Gain |
|---|---|---|---|
| Literature | 76.4% | 92.7% | +16.4pp |
| Grammar | 62.1% | 73.7% | +11.6pp |
| Poetry | 48.9% | 60.2% | +11.4% |
| History | 70.1% | 79.9% | +9.8pp |
| Synonyms | 90.4% | 90.8% | +0.4pp |
Overall Tibetan benchmark performance improved from 63.4% to 73.4%, representing a +10 percentage-point gain over the base model.
A major goal of Melong 2.0 was maintaining strong standalone reasoning while maximizing performance when paired with retrieval.
| Model | Closed-Book | RAG (10 Documents) |
|---|---|---|
| Melong 2.0 | 68.75% | 76.73% |
| DeepSeek v3.2 | 62.37% | 76.26% |
| Gemini 2.5 Flash | 66.63% | 73.59% |
When retrieval context increases to 20 documents, Melong 2.0 reaches 79.2% accuracy, demonstrating the effectiveness of combining specialized language understanding with retrieval-grounded reasoning.
Melong 2.0 is not the final destination. The next phase of development focuses on three major research directions.
The team is continuing to expand both classical and modern Tibetan corpora while improving preprocessing and synthetic generation systems. Future training phases aim to scale data volume to approximately 3× the current corpus size.
The team is continuing to expand both classical and modern Tibetan corpora while improving preprocessing and synthetic generation systems. Future training phases aim to scale data volume to approximately 3× the current corpus size.
Future versions of Melong are expected to interact directly with tools, APIs, databases, and retrieval systems, enabling:
Future versions of Melong are expected to interact directly with tools, APIs, databases, and retrieval systems, enabling:
Future research directions include native vision, speech, and document-understanding capabilities designed specifically for Tibetan-language workflows.
These capabilities could support speech-to-speech translation, automated dictation, digital preservation projects, and visual understanding of both historical and modern documents.
Future research directions include native vision, speech, and document-understanding capabilities designed specifically for Tibetan-language workflows.
These capabilities could support speech-to-speech translation, automated dictation, digital preservation projects, and visual understanding of both historical and modern documents.
Melong 2.0 demonstrates that meaningful progress in low-resource language AI does not require the largest models. Through specialized continual pre-training, retrieval-grounded reasoning, long-context optimization, and culturally aligned data engineering, it is possible to build systems that preserve linguistic heritage while advancing state-of-the-art performance.
Our goal remains clear: to create next-generation language technologies and digital knowledge systems that ensure Tibetan culture remains accessible, vibrant, and deeply integrated into the future of artificial intelligence.