Going Global
Language beyond English.
Most language technology was built on English, and tested on English. But most people in the world speak something else. When Bright Books goes global, everything that you've learned needs a second look. Tokenising, sentiment, search, and even what politeness means.
- Built and tested on English
- Most people speak something else
- Everything needs a second look
What changes across languages
Chinese and Japanese have no spaces between words. Arabic is written right to left, and it changes its letters by position. Finnish and Turkish build giant words from smaller pieces. Many people mix languages in one sentence, as in Hinglish. And many languages have very little training data. So don't assume that a tool that works in English will work anywhere else.
Which approach?
You've got a few choices. Translate everything to English, then process it. That's simple, but it loses tone, and it adds errors. Use multilingual embeddings, which place all languages in one space. Or use large language models, which handle many languages well, with weaker quality on those with little data. Whichever you pick, test each language separately.
| Approach | Trade-off |
|---|---|
| Translate to English | Simple, loses tone |
| Multilingual embeddings | One index for all |
| LLM directly | Good, varies by language |
| Local staff review | Essential for quality |
Summary
Test every language separately, and work with native speakers. Next, how did we get from simple rules to today's models?
Key takeaways- Languages differ deeply
- Don't assume English tools transfer
- Test each language