Language Datasets
This page brings the Appendix of the book to life. Discover what datasets exist, find who's working on it, and see what AI tools actually support it and where the gaps are.
By the numbers
Book Appendices
Every table and researcher profile from the appendices of Large Language Models for Linguists, now explorable.
| Foundation Model | Created By | Open Source? |
|---|---|---|
| GPT-3.5 / GPT-4.0 | OpenAI | No |
| Qwen3 (119 languages) | Alibaba | Yes |
| Gemini / Gemini 2.5 Pro | Open Weights | |
| Gamma | Open Weights | |
| Fanar | Qatar / HBKU | No |
| SEA-LION | HuggingFace | Yes |
| DeepSeek | China | Yes |
| LLaMA | Meta | Yes |
| Awarri (Igbo, Yoruba, Hausa) | Nigerian Government | No |
| mT5 / Flan-T5 / T5 / BERT | Yes | |
| Mistral | Mistral AI, France | Mixed |
| Aya | Cohere Labs | Yes |
| Swiss AI Initiative (60% European, 40% other) | Swiss AI Initiative | Yes |
| Language | ChatGPT | Qwen3 (Alibaba) | Zoom AI Companion | Zoom Meeting Summary | Zoom Background Generator |
|---|---|---|---|---|---|
| Albanian | Yes | Yes | No | No | No |
| Amharic | Yes | Yes | No | No | No |
| Arabic | Yes | Yes | Yes | Yes | Yes |
| Armenian | Yes | Yes | No | No | No |
| Bengali | Yes | Yes | No | No | No |
| Bosnian | Yes | Yes | No | No | No |
| Bulgarian | Yes | Yes | Yes | Yes | Yes |
| Burmese | Yes | Yes | No | No | No |
| Catalan | Yes | Yes | No | No | No |
| Chinese (Simplified) | Yes | Yes | Yes | Yes | Yes |
| Chinese (Traditional) | Yes | Yes | Yes | Yes | Yes |
| Croatian | Yes | Yes | No | No | No |
| Czech | Yes | Yes | Yes | Yes | Yes |
| Danish | Yes | Yes | Yes | Yes | Yes |
| Dutch | Yes | Yes | Yes | Yes | Yes |
| English | Yes | Yes | Yes | Yes | Yes |
| Estonian | Yes | Yes | Yes | Yes | Yes |
| Finnish | Yes | Yes | Yes | Yes | Yes |
| French | Yes | Yes | Yes | Yes | Yes |
| Georgian | Yes | Yes | No | No | No |
| German | Yes | Yes | Yes | Yes | Yes |
| Greek | Yes | Yes | Yes | Yes | Yes |
| Gujarati | Yes | Yes | No | No | No |
| Hindi | Yes | Yes | Yes | Yes | Yes |
| Hungarian | Yes | Yes | Yes | Yes | Yes |
| Icelandic | Yes | Yes | No | No | No |
| Indonesian | Yes | Yes | Yes | Yes | Yes |
| Italian | Yes | Yes | Yes | Yes | Yes |
| Japanese | Yes | Yes | Yes | Yes | Yes |
| Kannada | Yes | Yes | No | No | No |
| Kazakh | Yes | Yes | No | No | No |
| Korean | Yes | Yes | Yes | Yes | Yes |
| Latvian | Yes | Yes | Yes | Yes | Yes |
| Lithuanian | Yes | Yes | Yes | Yes | Yes |
| Macedonian | Yes | Yes | No | No | No |
| Malay | Yes | Yes | Yes | Yes | Yes |
| Malayalam | Yes | Yes | No | No | No |
| Marathi | Yes | Yes | No | No | No |
| Mongolian | Yes | Yes | No | No | No |
| Norwegian | Yes | Yes | Yes | Yes | Yes |
| Persian | Yes | Yes | No | No | No |
| Polish | Yes | Yes | Yes | Yes | Yes |
| Portuguese | Yes | Yes | Yes | Yes | Yes |
| Punjabi | Yes | Yes | No | No | No |
| Romanian | Yes | Yes | Yes | Yes | Yes |
| Russian | Yes | Yes | Yes | Yes | Yes |
| Serbian | Yes | Yes | No | No | No |
| Slovak | Yes | Yes | Yes | Yes | Yes |
| Slovenian | Yes | Yes | Yes | Yes | Yes |
| Somali | Yes | Yes | No | No | No |
| Spanish | Yes | Yes | Yes | Yes | Yes |
| Swahili | Yes | Yes | No | No | No |
| Swedish | Yes | Yes | Yes | Yes | Yes |
| Tagalog | Yes | Yes | No | No | No |
| Tamil | Yes | Yes | No | No | No |
| Telugu | Yes | Yes | No | No | No |
| Thai | Yes | Yes | Yes | Yes | Yes |
| Turkish | Yes | Yes | Yes | Yes | Yes |
| Ukrainian | Yes | Yes | Yes | Yes | Yes |
| Urdu | Yes | Yes | No | No | No |
| Vietnamese | Yes | Yes | Yes | Yes | Yes |
| Welsh | Yes | Yes | No | No | No |
| Language | LLM Model | Credits / Research Team |
|---|---|---|
| Ainu | AI Pirika | SARAC / Prof. Kenji Araki, Hokkaido University |
| Korean | Synatra / Motif / Polyglot-ko | Korean NLP community |
| British English | BritLLM: Caernarfon | UCL (University College London) |
| Portuguese | AMÁLIA / Open PT LLM Leaderboard | Portuguese NLP community |
| Italian | LLaMantino | University of Bari |
| Vietnamese | VinaLLaMA | Vietnamese NLP team |
| Dutch | Dutch Llama v2 13B | Dutch open-source community |
| SE Asian languages | SeaLLM | DAMO Academy / Alibaba |
| Odia | Odia Generative AI | Odia Generative AI initiative |
| Hindi | OpenHathi | Sarvam AI |
| Tamil | TamilLLAMA | Tamil NLP community |
| Hausa | HausaLlama | HausaNLP team |
| Swahili | UlizaLlama | Jacaranda Health |
| Yoruba | YorubaGPT | Yoruba NLP community |
| Xhosa / Zulu | Xhosa_ZuluLlama3_v1 | Jacaranda Health |
| IsiZulu / Yoruba / Hausa / Swahili / IsiXhosa | InkubaLM | Lelapa AI |
| Ghanaian languages | Ghana NLP | Ghana NLP |
| Ethiopian languages | EthioNLP models | EthioNLP |
| Swahili / Hausa / Yoruba / Xhosa / Zulu | UlizaLlama (maternal health) | Jacaranda Health |
| Japanese | Gemma 2 JPN | Japanese research community |
| Spanish | LeNIA-Chat-1.5B | Spanish NLP community |
| Spanish / Catalan / Basque / Galician | LA Leaderboard models | SomosNLP / María Grandury (Technical Univ. Madrid) |
| 60% European languages | Swiss AI Initiative LLM | Swiss AI Initiative |
| Benchmark | Language(s) | Details |
|---|---|---|
| bgGLUE | Bulgarian | Bulgarian General Language Understanding Evaluation benchmark |
| INCLUDE | 44 languages | Inclusive multilingual evaluation benchmark |
| Belebele | 122 languages | Meta's massive multilingual reading comprehension benchmark |
| Aya Evaluation Suite | 101 languages | Cohere Labs multilingual evaluation suite |
| HELM | English + multilingual | Holistic Evaluation of Language Models — Stanford University |
| CLEVA | Chinese | Chinese Language Model Evaluation — Stanford & CUHK |
| HellaSwag | English | Commonsense NLI: can a machine complete a sentence? |
| BIG-Bench | Multilingual | Beyond the Imitation Game Benchmark — 200+ tasks |
| COPA | English | Choice of Plausible Alternatives — causal reasoning |
| BoolQ | English | Yes/no question answering — 15,942 Q&A pairs |
| TruthfulQA | English | 817 questions across 38 categories testing for truthfulness |
| WiC | English | Word-in-Context — word sense disambiguation |
| SentiNeg | Multilingual | Sentiment and negation benchmark |
| LMArena (formerly Chatbot Arena) | Multilingual | Human preference-based ranking of LLMs |
| AlGhafa | Arabic | Arabic benchmark — Technology Innovation Institute, Abu Dhabi |
| Evalverse | Korean / English | Unified evaluation framework for LLMs |
| Voice Dataset | Source | Details |
|---|---|---|
| Common Voice | Mozilla | Crowd-sourced multilingual speech dataset — one of the largest open ASR datasets |
| NaijaVoice | Lanfrica (grant-funded) | 500+ Nigerian languages; speech data for low-resource Nigerian language ASR |
| Zambezi Voice | Zambian multilingual research team | Covers Bemba, Nyanja, Tonga, Lozi, Lunda — Zambian national languages |
| LibriSpeech | Johns Hopkins University | 1,000 hours of English read speech, derived from LibriVox audiobooks |
| Dataset | Language(s) | Details |
|---|---|---|
| Aya Dataset | 65 languages | 513 million instruction-response instances — Cohere Labs |
| xP3x | 277 languages | 16 NLP tasks; broad multilingual instruction dataset |
| Aya Collection | 101 languages | Curated multilingual instruction collection — Cohere Labs |
| DataProvenance Collection | Multilingual | Licensing and provenance metadata for training datasets |
| Alpaca Cleaned | English | Cleaned version of Stanford's Alpaca instruction-tuning dataset |
| BramVanroy Dutch Chat | Dutch | Dutch conversational instruction dataset |
| Synthetic Datasets | Various | Synthetically generated instruction datasets for low-resource languages |
| NEU-ESC | Vietnamese | Vietnamese emotional speech corpus — Northeastern University |
| OpenOrca (Tamil) | Tamil | Tamil adaptation of the OpenOrca reasoning dataset |
| Alpaca Cleaned Tamil | Tamil | Tamil-language version of Stanford Alpaca |
| Indic Alpaca | Telugu, Sindhi, Malayalam, Punjabi, Tamil, Urdu, Nepali, Assamese, Marathi, Konkani, Gujarati, Bengali, Odia, Hindi | Alpaca-style dataset covering 14 Indic languages |
| Inkuba-Mono | 5 African languages | Monolingual corpus — Lelapa AI |
| Bambara Datasets | Bambara | NLP resources for Bambara (Mali) — Allahsera Auguste Tapo / RobotMali |
| Spanish NLP Datasets | Spanish | AQuAS, ClinTreatES, ClinDiagnosES, HumorQA, SpaLawEx, TELEIA, RAGQuAS |
| PHOENIX-2014-T | German Sign Language | Sign language translation from TV broadcast — weather domain |
| AI4KS KSL Dataset | Kenyan Sign Language | Kenyan Sign Language recognition dataset |
| Mendeley African Food Dataset | English (Ghana / Cameroon) | Food classification dataset covering Ghanaian and Cameroonian dishes |
| Nigerian Food Dataset | English (Nigeria) | Nigerian food image dataset — Kaggle |
| AGE: Amharic/Ge'ez/English | Amharic, Ge'ez, English | Parallel translation dataset for Ethiopian languages |
If your language isn't in our database, that's exactly the gap we're trying to close. Let us know and we'll work to add it — or point you to researchers already working on it.