BUILDING CULTURALLY SENSITIVE AI

Language Datasets

Find out what is known about your language.

This page brings the Appendix of the book to life. Discover what datasets exist, find who's working on it, and see what AI tools actually support it and where the gaps are.

By the numbers

What's inside the book

🌍
Languages covered
75+
Extensive and Inclusive
👥
Grassroots researchers
50+
[Placeholder — update with exact number from book Appendix]
🗄️
Datasets & benchmarks
35+
19 datasets + 16 benchmarks
🤟
Sign language AI projects
6
ASL, KSL, German SL, Brazilian/Argentinian SL, Visual SL (USTC), AI4KS KSL
🌱
Endangered language projects
6
Ainu, Owens Valley Paiute, Seneca, Bardi, Bo (Andaman), Tok Pisin
🌐
Language families represented
12
Niger-Congo, Dravidian, Japonic, Koreanic, Sino-Tibetan, Indo-European, Austronesian, Afro-Asiatic, Mande, Turkic, Tai-Kadai, Sign Languages

Book Appendices

From the Book — Data, Benchmarks & the People Behind It

Every table and researcher profile from the appendices of Large Language Models for Linguists, now explorable.

Foundation ModelCreated ByOpen Source?
GPT-3.5 / GPT-4.0OpenAINo
Qwen3 (119 languages)AlibabaYes
Gemini / Gemini 2.5 ProGoogleOpen Weights
GammaGoogleOpen Weights
FanarQatar / HBKUNo
SEA-LIONHuggingFaceYes
DeepSeekChinaYes
LLaMAMetaYes
Awarri (Igbo, Yoruba, Hausa)Nigerian GovernmentNo
mT5 / Flan-T5 / T5 / BERTGoogleYes
MistralMistral AI, FranceMixed
AyaCohere LabsYes
Swiss AI Initiative (60% European, 40% other)Swiss AI InitiativeYes
LanguageChatGPTQwen3 (Alibaba)Zoom AI CompanionZoom Meeting SummaryZoom Background Generator
AlbanianYesYesNoNoNo
AmharicYesYesNoNoNo
ArabicYesYesYesYesYes
ArmenianYesYesNoNoNo
BengaliYesYesNoNoNo
BosnianYesYesNoNoNo
BulgarianYesYesYesYesYes
BurmeseYesYesNoNoNo
CatalanYesYesNoNoNo
Chinese (Simplified)YesYesYesYesYes
Chinese (Traditional)YesYesYesYesYes
CroatianYesYesNoNoNo
CzechYesYesYesYesYes
DanishYesYesYesYesYes
DutchYesYesYesYesYes
EnglishYesYesYesYesYes
EstonianYesYesYesYesYes
FinnishYesYesYesYesYes
FrenchYesYesYesYesYes
GeorgianYesYesNoNoNo
GermanYesYesYesYesYes
GreekYesYesYesYesYes
GujaratiYesYesNoNoNo
HindiYesYesYesYesYes
HungarianYesYesYesYesYes
IcelandicYesYesNoNoNo
IndonesianYesYesYesYesYes
ItalianYesYesYesYesYes
JapaneseYesYesYesYesYes
KannadaYesYesNoNoNo
KazakhYesYesNoNoNo
KoreanYesYesYesYesYes
LatvianYesYesYesYesYes
LithuanianYesYesYesYesYes
MacedonianYesYesNoNoNo
MalayYesYesYesYesYes
MalayalamYesYesNoNoNo
MarathiYesYesNoNoNo
MongolianYesYesNoNoNo
NorwegianYesYesYesYesYes
PersianYesYesNoNoNo
PolishYesYesYesYesYes
PortugueseYesYesYesYesYes
PunjabiYesYesNoNoNo
RomanianYesYesYesYesYes
RussianYesYesYesYesYes
SerbianYesYesNoNoNo
SlovakYesYesYesYesYes
SlovenianYesYesYesYesYes
SomaliYesYesNoNoNo
SpanishYesYesYesYesYes
SwahiliYesYesNoNoNo
SwedishYesYesYesYesYes
TagalogYesYesNoNoNo
TamilYesYesNoNoNo
TeluguYesYesNoNoNo
ThaiYesYesYesYesYes
TurkishYesYesYesYesYes
UkrainianYesYesYesYesYes
UrduYesYesNoNoNo
VietnameseYesYesYesYesYes
WelshYesYesNoNoNo
LanguageLLM ModelCredits / Research Team
AinuAI PirikaSARAC / Prof. Kenji Araki, Hokkaido University
KoreanSynatra / Motif / Polyglot-koKorean NLP community
British EnglishBritLLM: CaernarfonUCL (University College London)
PortugueseAMÁLIA / Open PT LLM LeaderboardPortuguese NLP community
ItalianLLaMantinoUniversity of Bari
VietnameseVinaLLaMAVietnamese NLP team
DutchDutch Llama v2 13BDutch open-source community
SE Asian languagesSeaLLMDAMO Academy / Alibaba
OdiaOdia Generative AIOdia Generative AI initiative
HindiOpenHathiSarvam AI
TamilTamilLLAMATamil NLP community
HausaHausaLlamaHausaNLP team
SwahiliUlizaLlamaJacaranda Health
YorubaYorubaGPTYoruba NLP community
Xhosa / ZuluXhosa_ZuluLlama3_v1Jacaranda Health
IsiZulu / Yoruba / Hausa / Swahili / IsiXhosaInkubaLMLelapa AI
Ghanaian languagesGhana NLPGhana NLP
Ethiopian languagesEthioNLP modelsEthioNLP
Swahili / Hausa / Yoruba / Xhosa / ZuluUlizaLlama (maternal health)Jacaranda Health
JapaneseGemma 2 JPNJapanese research community
SpanishLeNIA-Chat-1.5BSpanish NLP community
Spanish / Catalan / Basque / GalicianLA Leaderboard modelsSomosNLP / María Grandury (Technical Univ. Madrid)
60% European languagesSwiss AI Initiative LLMSwiss AI Initiative
BenchmarkLanguage(s)Details
bgGLUEBulgarianBulgarian General Language Understanding Evaluation benchmark
INCLUDE44 languagesInclusive multilingual evaluation benchmark
Belebele122 languagesMeta's massive multilingual reading comprehension benchmark
Aya Evaluation Suite101 languagesCohere Labs multilingual evaluation suite
HELMEnglish + multilingualHolistic Evaluation of Language Models — Stanford University
CLEVAChineseChinese Language Model Evaluation — Stanford & CUHK
HellaSwagEnglishCommonsense NLI: can a machine complete a sentence?
BIG-BenchMultilingualBeyond the Imitation Game Benchmark — 200+ tasks
COPAEnglishChoice of Plausible Alternatives — causal reasoning
BoolQEnglishYes/no question answering — 15,942 Q&A pairs
TruthfulQAEnglish817 questions across 38 categories testing for truthfulness
WiCEnglishWord-in-Context — word sense disambiguation
SentiNegMultilingualSentiment and negation benchmark
LMArena (formerly Chatbot Arena)MultilingualHuman preference-based ranking of LLMs
AlGhafaArabicArabic benchmark — Technology Innovation Institute, Abu Dhabi
EvalverseKorean / EnglishUnified evaluation framework for LLMs
Voice DatasetSourceDetails
Common VoiceMozillaCrowd-sourced multilingual speech dataset — one of the largest open ASR datasets
NaijaVoiceLanfrica (grant-funded)500+ Nigerian languages; speech data for low-resource Nigerian language ASR
Zambezi VoiceZambian multilingual research teamCovers Bemba, Nyanja, Tonga, Lozi, Lunda — Zambian national languages
LibriSpeechJohns Hopkins University1,000 hours of English read speech, derived from LibriVox audiobooks
DatasetLanguage(s)Details
Aya Dataset65 languages513 million instruction-response instances — Cohere Labs
xP3x277 languages16 NLP tasks; broad multilingual instruction dataset
Aya Collection101 languagesCurated multilingual instruction collection — Cohere Labs
DataProvenance CollectionMultilingualLicensing and provenance metadata for training datasets
Alpaca CleanedEnglishCleaned version of Stanford's Alpaca instruction-tuning dataset
BramVanroy Dutch ChatDutchDutch conversational instruction dataset
Synthetic DatasetsVariousSynthetically generated instruction datasets for low-resource languages
NEU-ESCVietnameseVietnamese emotional speech corpus — Northeastern University
OpenOrca (Tamil)TamilTamil adaptation of the OpenOrca reasoning dataset
Alpaca Cleaned TamilTamilTamil-language version of Stanford Alpaca
Indic AlpacaTelugu, Sindhi, Malayalam, Punjabi, Tamil, Urdu, Nepali, Assamese, Marathi, Konkani, Gujarati, Bengali, Odia, HindiAlpaca-style dataset covering 14 Indic languages
Inkuba-Mono5 African languagesMonolingual corpus — Lelapa AI
Bambara DatasetsBambaraNLP resources for Bambara (Mali) — Allahsera Auguste Tapo / RobotMali
Spanish NLP DatasetsSpanishAQuAS, ClinTreatES, ClinDiagnosES, HumorQA, SpaLawEx, TELEIA, RAGQuAS
PHOENIX-2014-TGerman Sign LanguageSign language translation from TV broadcast — weather domain
AI4KS KSL DatasetKenyan Sign LanguageKenyan Sign Language recognition dataset
Mendeley African Food DatasetEnglish (Ghana / Cameroon)Food classification dataset covering Ghanaian and Cameroonian dishes
Nigerian Food DatasetEnglish (Nigeria)Nigerian food image dataset — Kaggle
AGE: Amharic/Ge'ez/EnglishAmharic, Ge'ez, EnglishParallel translation dataset for Ethiopian languages
Jared Coleman
Owens Valley Paiute
Loyola Marymount University, USA
Language documentation & NLP for endangered indigenous languages
Eduardo Garcia
Portuguese
Brazil
Brazilian Portuguese AI research
Xinwen "Enru" Zhang
Brazilian Portuguese
Stanford University — CLAS
Cross-lingual alignment and semantic research
Guijin Son
Korean
Yonsei University / OneLine AI
KO-REAson-0831 — Korean reasoning dataset & evaluation
Idris Abdulmumin
Hausa
Ahmadu Bello University / University of Pretoria
HausaNLP — NLP tools and datasets for the Hausa language
Ife Adebara
517 African languages
University of Alberta — CIFAR AI Chair
Massively multilingual African language NLP research
Allahsera Auguste Tapo
Bambara
RIT / Google PhD Fellow
RobotMali — Bambara language NLP & robotics research
Abhinaya Mahendiran
Tamil
Nunnari Labs — CTO
Tamil language AI tools and applications
Craig Alan Volker
Tok Pisin
James Cook University
Tok Pisin (Papua New Guinea Creole) language research
Robbie Jimerson Jr PhD
Seneca
Seneca Nation / University of Rochester
Seneca Nation language revitalization with NLP
Prof. Kenji Araki
Ainu (Japanese)
Hokkaido University
AI Pirika — LLM for the Ainu language of Japan
Bonaventure Dossou
Fon
Lanfrica co-founder / McGill University PhD
Fon language NLP; co-created Lanfrica African language data hub
Prof. Vukosi Marivate
African languages
University of Pretoria — ABSA-UP Chair
African language data and NLP infrastructure
Tejumade Afonja
Nigerian languages
Nigeria / Germany
AI Saturdays Lagos co-founder — AI education & community in Africa
Chris Emezue
African languages
Lanfrica co-founder
African language dataset curation and open-source NLP
Key-Sun Choi
Korean
KAIST — Professor Emeritus
Korean NLP & knowledge graph research
Junbum Lee
Korean
Google — ML Expert
KcBERT & KoAlpaca — Korean BERT and instruction-tuning models
Prof. Naoaki Okazaki
Japanese
Institute of Science Tokyo
Swallow series — large Japanese language models
Kengatharaiyer Sarveswaran
Tamil
University of Jaffna
Tamil NLP tools and morphological analysis
Dr. Sripriya N
Tamil
SSN College, India
Tamil language processing and educational NLP
Bharathi B
Tamil
Shiv Nadar University
ASR for elderly and transgender Tamil speakers
Bharathi Raja Chakravarthi
Tamil, English, German, Spanish, Kannada
University of Galway
Multilingual and code-switched NLP
Rajeswari Natarajan
Dravidian languages (Malayalam, Tamil, Telugu)
SASTRA University
Multimodal hate speech detection in Dravidian languages
Preslav Nakov
Bulgarian / Slavic languages
MBZUAI, UAE
Fact-checking, disinformation, and Slavic language NLP
Isaac Caswell
Multilingual
Stanford / Google NLP
Google Translate language expansion and data quality
Ravi Theja Desetty & Ramsri Goutham
Telugu
Mistral AI
Telugu LLaMA — instruction-tuned model for Telugu
Sara Hooker
114+ languages
Cohere Labs — Head of Research
Efficient & multilingual model research; Aya project leadership
Claire Louise Bowern
Bardi (Aboriginal Australian)
Yale University
Bardi language documentation and endangered language NLP
Prof. Vassil Panayotov
English
Johns Hopkins University
LibriSpeech — foundational English ASR dataset
María Grandury
Spanish variants
Technical University of Madrid
LA Leaderboard — evaluation for Spanish, Catalan, Basque, Galician
Michael Boone
American Sign Language (ASL)
NVIDIA — Trustworthy AI
Sign language recognition and AI accessibility
Georgine Auma
Kenyan Sign Language (KSL)
Studio KSL / eKitabu — Mandela Washington Fellow
KSL AI dataset creation and deaf education technology
Prof. Houqiang Li & Prof. Wengang Zhou
Visual Sign Language
University of Science & Technology of China (USTC)
Computer vision for sign language recognition
João Pedro Santos
Brazilian & Argentinian Sign Language
Brazil
Sign language NLP for South American deaf communities
Aremu Aanuoluwapo
English, Yoruba, Nigerian Pidgin, isiZulu, Swahili, seSotho, Hausa
Lelapa AI — Lead Data Officer
African language data curation and InkubaLM development
Henok Biadglign Ademtew
Underrepresented languages
Vella — ML Engineer
NLP for low-resource and underrepresented languages
Anvita Abi
Bo (Andaman Islands, India)
Andaman Island, India
Documentation of the Bo language — extinct as of 2010
Binyam Ephrem Seyoum, Seid Muhie Yimam & Surafel M. Lakew
Ethiopian languages
EthioNLP founders / Amazon AI
EthioNLP — Ethiopian language NLP datasets and models

Don't see your language? Tell us.

If your language isn't in our database, that's exactly the gap we're trying to close. Let us know and we'll work to add it — or point you to researchers already working on it.

Scroll to Top