# Small multilingual model refuses harmful Hinglish prompts in Javanese

Source: https://farcaster.xyz/kazani/0x19133434
Kiwi News: https://news.kiwistand.com/stories/Small-multilingual-model-refuses-harmful-Hinglish-prompts-in-Javanese?index=0x6af341f564dc7f90fda240adbd05fa388b046f669e6218906af0fdd09e2be9a36613a47c
Submitted by @kazani on 2026-11-10 UTC, 2 upvotes, 1 comment

## Summary

A small multilingual model refused a harmful Hinglish prompt correctly but replied in Javanese, which a Hinglish speaker cannot read, and neither automatic check flagged the problem. The author's Lexsi Labs hackathon entry measures language kept as well as refusal on Tiny Aya Global. Blind labels found 9 of 16 base-model replies in Hinglish, versus 16 of 16 with three few-shot examples and 14 of 16 with a LoRA adapter trained with SafeTune, which the author calls likely but not significant at this size. Refusal stayed at 4 to 6 of 8 harmful prompts in every condition. The study covers one model with 16 prompts per kind.

## Article

I asked a small multilingual model a harmful question in Hinglish. It refused, which was right. But it refused in Javanese.

"Maaf, tapi aku ora bisa mbantu permintaan iki." Sorry, I can't help with this request. A Hinglish speaker can't read that.

Neither automatic check caught it. The HarmBench classifier scored it not harmful, which is true. AuditKIT's refusal detector only knows English phrases, so it didn't see a refusal at all.

So my Lexsi Labs hackathon entry measures language kept, not just refusal, on Tiny Aya Global:

- base model: 9/16 replies in Hinglish (my blind labels)
- three few-shot examples: 16/16
- a LoRA adapter trained with SafeTune: 14/16, likely but not significant at this size
- refusal stayed at 4 to 6 of 8 harmful prompts in every condition

Most of the work was finding out which measurements to trust. A language detector named languages the model never spoke, and a steering "finding" vanished under a random-direction control.

Small study: one model, 16 prompts per kind.

Write-up: https://github.com/kazani-351/tiny-aya-hinglish-safety/blob/main/ARTICLE.md
Adapter + data: https://huggingface.co/collections/kazani351/tiny-aya-hinglish-safety-model

Drafted with Claude, edited by me.

## Comments

**tim**: Welcome back Kazani!!