Raabta
Roman-Urdu questions, Urdu-script answers
Results — hover any figure for the baseline and the caveat
MRR@10
vs previous retrieval
5.8× on the same frozen split. The gain is almost entirely the romanized-title route: uroman transliterates every Urdu article title once, and character 2–4 gram matching then survives missing vowels, abbreviations and informal spellings that defeat both keyword and dense search.
Passages indexed
150-token windows with 30-token overlap over 4,000 Urdu Wikipedia articles from the 20231101 snapshot.
Evidence-verified questions
Split 120 dev / 60 test and frozen before evaluation. Every report is audited for zero locked-test queries; the test half has still not been scored.
GPUs required
Runs entirely on CPU — multilingual-e5-small for dense retrieval, gte-multilingual-reranker-base for reranking, uroman for title romanization.
The problem
Millions of Pakistanis type Urdu in Latin characters — 'kya', 'kia', 'kiya' are the same word — while the documents that answer their questions are written in Urdu script. Neither keyword search nor a single transliteration bridges that gap: keyword search finds nothing across scripts, and one transliteration commits to one guess about spelling that is usually wrong.
What I built
Rather than transliterating once, Raabta generates several controlled views of the same question — the original, a normalised form, an Urdu-script transliteration, and a conservative retrieval form — and gates each against semantic drift so a bad rewrite is discarded rather than searched. Each view goes down three routes: Unicode BM25 with title boosting, a multilingual dense retriever, and character 2–4 gram matching against uroman-romanized article titles. Weighted Reciprocal Rank Fusion merges the rankings, a multilingual cross-encoder reranks the top 20 over title plus passage, and an extractive stage returns the answering span only if it clears all seven validation checks.
The table
| System | Recall@1 | Recall@5 | Recall@10 | MRR@10 | nDCG@10 |
|---|---|---|---|---|---|
| Previous application retrieval | 0.075 | 0.117 | 0.192 | 0.101 | 0.122 |
| Current — with romanized-title route | 0.392 | 0.875 | 0.983 | 0.583 | 0.680 |
Regression check on the same 120 evidence-verified development questions, before and after the title route. The 60-question test split is frozen and unused. Earlier baseline and ablation runs are kept in docs/development_results.md.
What the results actually say
Romanizing the titles beat every improvement to the query side.
Reformulation and reranking had taken MRR@10 to 0.101. Adding one route — uroman over article titles, matched on character 2–4 grams — took it to 0.583 and Recall@10 to 0.983. The bottleneck was never the question; it was that no route could match an informally spelled entity name to its Urdu-script title.
Single transliteration is worse than doing nothing.
In the earlier four-system ablation it scored 0.007 MRR@10 against direct dense retrieval's 0.062 — committing to one spelling actively destroys signal. That result is what ruled out the simpler pipeline and led to multi-view reformulation.
Abstention is a feature, not a failure.
Retrieving the right article is necessary but not sufficient, so seven checks sit between retrieval and display — including whether the sentence contains the relation actually asked for, and whether a question about the present is being answered from a historical statement. A failed check shows its reason; the system never generates a replacement fact.
What this doesn’t prove
- The 0.983 is retrieval, not answers: it measures whether the right evidence reaches the top ten, not whether the final response is correct.
- The 120 diagnostic questions are title-oriented, which is exactly what the new route is best at — so the measured gain is an upper bound on what a broader question mix would show.
- The 60-question test split has not been scored, so every number here is a development-set result and should be read as provisional.
- No independent native-speaker review of answer quality yet, so relevance judgements rest on my own annotation.
- Evaluated on 4,000 articles of Urdu Wikipedia — cleaner, more formal, and far narrower than the text a real deployment would face. The live fallback is not a dependable source for prices or breaking news.
This section is here on purpose. A result without its limits isn’t a result, it’s a claim.