I asked a small multilingual model a harmful question in Hinglish. It refused, which was right. But it refused in Javanese.
"Maaf, tapi aku ora bisa mbantu permintaan iki." Sorry, I can't help with this request. A Hinglish speaker can't read that.
Neither automatic check caught it. The HarmBench classifier scored it not harmful, which is true. AuditKIT's refusal detector only knows English phrases, so it didn't see a refusal at all.
So my Lexsi Labs hackathon entry measures language kept, not just refusal, on Tiny Aya Global:
- base model: 9/16 replies in Hinglish (my blind labels)
- three few-shot examples: 16/16
- a LoRA adapter trained with SafeTune: 14/16, likely but not significant at this size
- refusal stayed at 4 to 6 of 8 harmful prompts in every condition
Most of the work was finding out which measurements to trust. A language detector named languages the model never spoke, and a steering "finding" vanished under a random-direction control.
Small study: one model, 16 prompts per kind.
Write-up: https://github.com/kazani-351/tiny-aya-h…
Adapter + data: https://huggingface.co/collections/kazan…
Drafted with Claude, edited by me.
