IJCAI–ECAI · Workshop on Foundation Models for Social Good ·
Basilectal-Inspired Health Questions Expose Robustness Gaps in Small Medical QA Models
Accepted · Workshop · Clinical NLP
Fluent answers can lose medical detail when a health question is phrased in nonstandard English.
Abstract
Small local language models are attractive for medical question answering in settings where connectivity, cost, and privacy limit cloud-based use. Yet users in these settings may express health questions in forms that move away from benchmark-standard English, including compressed, phonetic, and basilectal-inspired phrasing. We present a controlled robustness study using 102 TREC LiveQA Medical questions, paired L0–L3 synthetic basilectal-inspired variants, six small instruction-tuned models, and 7,416 free-form generations. Across the baseline setting, answer quality is stable under mild variation but degrades sharply in the L3 basilectal-inspired condition: BERTScore-style F1 falls from 0.707 at L0 to 0.614 at L3, ROUGE-L from 0.136 to 0.095, and Medical Concept Overlap recall from 0.129 to 0.046. The dominant failure mode is fluent incompleteness: models often continue producing plausible medical text while preserving fewer medically relevant concepts from trusted reference answers. We also examine prompt intervention, quantization, feature ablations, behavioral proxies, and claim-level support. The study is intentionally bounded: it is a controlled robustness benchmark, not a clinical safety certification, not a naturally collected dialect corpus, and not a claim about any specific speech community. Code is available at https://github.com/ankitsblade/Basilectal-Robustness-in-SLMs
Findings
- Evaluates 102 TREC LiveQA Medical questions, six models, and 7,416 generated responses.
- Medical Concept Overlap recall drops from 0.129 to 0.046 between the standard and most perturbed conditions.
- The study uses synthetic phrasing variants; it is neither a clinical safety certification nor a claim about a specific speech community.
Sources and citation
View BibTeX citation
@inproceedings{medical-qa-2026,
title = {{Basilectal-Inspired Health Questions Expose Robustness Gaps in Small Medical QA Models}},
author = {Saiankit Shankar and BHVSP Subrahmanyam and Neeraj Choudhary},
booktitle = {IJCAI–ECAI · Workshop on Foundation Models for Social Good},
year = {2026},
url = {https://scholar.google.com/citations?view_op=view_citation&hl=en&user=_kzC6EsAAAAJ&citation_for_view=_kzC6EsAAAAJ:2osOgNQ5qMEC},
note = {Accepted}
}