Research
Three questions run through my work: who governs speech technology, how to build it for languages with little data, and how people and machines read meaning beyond the words.
Speech technology ethics
Speech technology now decides who gets heard in public agencies, classrooms and courts, but the law governing it assumes a single claimant with a grievance. When a system fails a language, it fails everyone who speaks it. I work on that gap: fairness and bias, consent over speech data, and the standing of language communities under AI law and minority-language instruments. Frisian is the recurring test case.
Roles
- Ethics Chair, IEEE ASLI 2027
- Ethics Chair, INTERSPEECH 2025–2026
- Chair, ISCA Ethics Committee
- Management Committee, GRADE COST Action
- Working group member, CoARA
- Chair, Campus Fryslân Ethics Committee
In the news
AI for small data
Most of the world’s languages have little recorded speech, and the systems in widest use simply skip them. We build speech recognition and synthesis that work from small amounts of data, using multilingual and cross-lingual transfer, data augmentation and evaluation methods designed for low-resource settings. Frisian, Bildts and Luxembourgish are current cases.
The linguistic groundwork comes from years of fieldwork on Aymara and other Andean languages: a reference grammar, studies of its phonology and verbal art, and an early rule-based machine translation system.
Publications
- Improving low-resource ASR using bilingual fine-tuning with language identification: A cross-linguistic evaluation.
- The Blizzard Challenge 2025.
- Enhancing standard and dialectal Frisian ASR: Multilingual fine-tuning and language identification for improved low-resource performance.
- Overcoming data scarcity in multi-dialectal Arabic ASR via Whisper fine-tuning.
- Meta learning text-to-speech synthesis in over 7000 languages.
Low-resource speech → · Andean languages → · Language diversity →
In the news
Datasets: luxembourgish-asr-rtl-lu, a Luxembourgish speech corpus from RTL.lu (Hugging Face, 2022).
Meaning beyond words
Speech carries more than words: emotion, intent, irony. Sarcasm is our central test case, since speakers routinely mean the opposite of what they say. We combine acoustic, linguistic and visual cues to detect it, and we synthesize it to find out which cues listeners actually use. Related work looks at how listeners perceive prosody, including through cochlear implants and in Parkinson’s speech, and at how people experience sound in everyday places.
Publications
- What makes synthetic speech sound sarcastic? A prosody-controlled perception study.
- Modeling sarcastic speech: Semantic and prosodic cues in a speech synthesis framework.
- AMuSeD: An attentive deep neural network for multimodal sarcasm detection incorporating bimodal data augmentation.
- SarcasmMiner: A dual-track post-training framework for robust audio-visual sarcasm reasoning.
- A neural model for CDST: Resonance, attunement, and prosodic transfer in L2 development.
In the news
Datasets: MCSD 1.0, Multimodal Chinese Sarcasm Dataset (DataverseNL, 2025).