A New Method Improves AI Fluency in Smaller Languages

A new method aims to make AI-based language models more fluent even for languages with limited data and less developed tools. Researchers propose a way to further train models so that they maintain language fluency, even when guided by evaluation models that themselves produce clumsy or unnatural text.

The underlying concept is so-called preference-based optimization, where a language model is taught to produce responses that people—or "reward models" resembling another AI model—consider better. Previous work has primarily focused on English and Chinese, for which there is a wealth of high-quality training data and effective models available. For many smaller languages, such as Norwegian Bokmål, there is no equivalent data or models that naturally produce fluent text.

The researchers' goal is to build a fluently writing language model adapted to user preferences without requiring pre-instructed training data in the target language. Their approach is based on so-called on-policy training, where the model learns in feedback cycles based on its own generated responses.

The method is compared to two common solutions: supervised further training with machine-translated data and multilingual further training, where the model is taught simultaneously in multiple languages. The study conducts a case study with Norwegian Bokmål and specifically evaluates the fluency of the produced text.

The work is part of an effort to bring advanced AI applications to smaller languages that do not have the data abundance of larger languages. The researchers emphasize that maintaining fluency is crucial when fine-tuning models to suit user preferences in languages where there is limited instructional material produced by human writers available.

Source: Fluent Alignment with Disfluent Judges: Post-training for Lower-resource Languages, ArXiv (AI).