The Mongolian startup defying Big Tech with its own LLM

Silicon Valley firms are rushing to train artificial intelligence models with tens of thousands of graphics processing units, and offering multimillion-dollar salaries to star researchers. Meanwhile, in Mongolia, Badral Sanlig is building a large language model with only 128 GPUs and a small team of engineers to serve just his country.

Sanlig’s startup, Egune AI, is part of a growing movement to build LLMs in low-resource languages to expand AI access for underserved populations. Despite a shortage of training data, compute power, talent, and funding, these small models are attracting government clients and individual users keen to safeguard their language, cultural identity, and sovereignty in the face of dominance by American and Chinese firms. 

A software engineer trained in Germany, Sanlig began developing Mongolian speech recognition models in 2019, after noticing that smart devices like Alexa often struggled to understand his native tongue. Weeks after OpenAI launched ChatGPT in November 2022, Sanlig started working on an LLM. He named the startup Egune, which means “it” in Mongolian, he told Rest of World. 

“The early stage was very difficult, because we didn’t have a lot of GPUs or a lot of money,” the 45-year-old said. “But we believed that LLMs could solve our language [recognition] problem because OpenAI had proved it.” 

Due to Mongolia’s close ties with China and Russia, getting U.S. approval for purchasing the Nvidia chips commonly used to train AI models can be a complicated and long-drawn-out affair — Sanlig flies to Berlin frequently to buy gaming-grade chips. Training data is another hurdle. Egune relies on Mongolian university texts, library archives, and synthetic data. The team in Ulaanbaatar of about 40 consists mostly of young engineers who studied abroad and do not have much work experience. 

Top-ranked LLMs such as GPT, Anthropic’s Claude, and Google’s Gemini generally have hundreds of billions of parameters, or variables within a model that determine how it behaves. In contrast, the first model from Egune had 5 billion parameters and could only understand Mongolian. The next model, with 34 billion parameters, was built with multi-language data and more general knowledge, Sanlig said. The latest Egune model has 70 billion parameters and improved coding and reasoning capability. 

Despite their larger scale, the performance of bigger models in low-resource languages can be unpredictable: Companies are more focused on getting the models to excel in English and other widely used languages such as Spanish, French, and German. With their own models, smaller communities can make improvements and correct inaccuracies more quickly, Sanlig said. 

Egune focuses on serving a small base of 3.5 million people in Mongolia. Its models are better at not only the language but also the history, culture, and nomadic lifestyle, he said. 

“Mongolian culture is very different from Western culture, and the nomadic people may have some [questions like]: What’s the problem with this horse?” said Sanlig. “We can decide which information we feed into the models.”

Egune Chat currently has about 24,000 daily users. The company has also developed AI applications for a telecom company, a bank, and several government agencies, Sanlig said.

Several similar efforts around the world are seeking similar outcomes with their own models. These include Latin America’s Latam-GPT, Singapore’s Sea-Lion, India’s BharatGPT, and Nigerian startup Awarri, which is building an LLM trained in five local languages and accented English. 

There are real benefits to such LLMs. The unreliability of mainstream models in low-resource languages can exacerbate inequality, causing communities to miss out on AI’s economic benefits, while suffering more from its biases and hallucinations, Caroline Meinhardt, policy research manager at the Stanford Institute for Human-Centered Artificial Intelligence, told Rest of World. 

Besides exploring AI to better serve local populations and represent local values, researchers and authorities are concerned with the national security implications of relying on mostly U.S. AI technologies, said Meinhardt, who studies low-resource language LLMs.

“There is this desire to be able to develop your own AI systems that align with your own specific cultural and linguistic context, but also perhaps your own political context,” she said. 

But a lack of resources makes it hard for these models to compete with those made by American and Chinese tech giants. ChatGPT is much more popular than Egune in Mongolia, said Tserennyam Sukhbaatar, a native Mongolian and a marketing professor at Brigham Young University–Hawaii. “Even though it’s a Mongolian AI, [it’s] not as good as ChatGPT,” he told Rest of World. 

Still, it’s important that a local company has control over Mongolian models and training data, and he tries to use Egune occasionally to show his support, Sukhbaatar said. 

“The Mongolian economy is heavily focused on mining, so if somebody is trying to do business in technology, we have to raise both hands to support that business,” he said. 

Mongolia, with its vast deposits of coal and copper, derives about a quarter of its gross domestic product from mining. The country has a small tech industry, with a startup ecosystem worth only about $156 million, according to a 2022 report by the Japan International Cooperation Agency. Egune is valued at about $39 million, with a recent investment of $3.5 million from Golomt Bank, one of the biggest banks in the country. 

While Mongolia is not a tech heavyweight, Egune can help other small countries build their own LLMs, Sanlig said. The startup showcases Mongolia’s potential to build a homegrown tech industry, even as the country suffers from a brain drain of young workers seeking education and jobs abroad, he said.

“If they use Egune, the young generation will understand the importance of this fundamental work,” said Sanlig. “The investment climate will also be warmer.”