Can You Translate a Conversation in Real Time Offline?

You have just landed in a foreign country, with no internet and no signal. You need to activate a local SIM, but to do that you need to talk to a local shopkeeper. Unfortunately, though, to do that you need a translator… which doesn't work without internet.

This paradox explains why we chose to research how to introduce offline translation into one of our products, VoiceLink.

Where we started from: what VoiceLink is

VoiceLink is a mobile app by Mabiloft built for conversational translation. It comes from the need to overcome the clunky UX of classic translation apps: select the language, speak, hand the phone to the other person and so on, often accompanied by a disappointing translation.

Thanks to VoiceLink, two people who speak different languages can have a natural conversation. The app understands who is speaking and in what language, translates into the other person's language and reads the translation out loud. You can see it in action here.

What the product is missing

VoiceLink is live on the stores and is enjoying a fair amount of success, but as of today it is still tied to the network. In practice, if you are in the mountains, with no signal, and the only person you can ask for information is an expert German climber, you can only hope that they know the basics of Italian.

It was precisely from this consideration that we started when we set our minds on developing an offline version of the product, not meant for constant use, but as an as-needed mode, adaptable to the situation.

The architecture we built, therefore, makes it possible to choose piece by piece what runs on the phone and what runs in the cloud, depending on the need of the moment. Quality inevitably drops when the cloud is not available, but it makes it possible to provide a service even in remote areas or when the server does not respond.

How a translated conversation is built

A translated conversation is not a single block, but an assembly line:

• The voice is detected
• The language being used is recognized
• What has been heard is transposed into writing
• The text is translated
• The translation is read out by a synthetic voice

To carry out our research we built a test bench: a test app in Flutter that reconstructed the entire chain, measuring the times and the quality of every step. Not an estimate, then, but an actual measurement, in order to optimize starting from data.

The languages we used for our test are the most common ones, such as English or German, but also the less well-served ones, such as Persian, precisely because they are typically less covered by translation services, especially those geared towards conversation.

We also tested the results on three different pieces of hardware to compare performance. Specifically, we used:

• A budget Android from 2021
• A more recent high-end Android
• A high-performing iPhone from a few years ago

The goal was to develop a complete downloadable language pack, with speech recognition, translation and synthetic voice, of a sustainable size, roughly 1-1.5 GB (basically, the size of two or three episodes of a TV series downloaded in HD from a streaming platform).

One model, three phones and three different outputs

The first translation model took 1.4 seconds per sentence on the iPhone, an excellent time. But performance dropped drastically on the two Androids: while the more recent one required 4 seconds, still an acceptable time, the older model needed a full 12 seconds.

This immediately taught us a lesson: the same experience has to be designed in tiers. While it makes sense to use the model that delivers the best translation on more powerful phones, it becomes necessary to use lighter models for cheaper phones, with a shift in quality handled elegantly, so that the user does not perceive a “second-rate” product.

A matter of architecture

On the other hand, it is not the model alone that makes the difference. The architecture chosen also clearly changed the quality of the result.

First of all, we found that specialized models, dedicated to individual languages, produced a transcription 3 or 4 times faster than generic speech recognition models.

The specificity of the model also achieved better results when it came to translation: targeted models that focused on a single language pair showed better performance than multilingual models. On the other hand, the latter come in handy for covering more unusual and less widespread languages.

Finally, we also tested the models that used English as an intermediate language, that is, models that in order to translate, for example, from German into Italian, first translated the German text into English and then translated from English into Italian.

The result? An unnatural translation, in which linguistic nuance was lost between translations. For example, both Italian and German have a polite form of address (the Italian “Lei”), which does not exist in English. As a result, leaning on English led to this linguistic aspect being lost.

Perception counts as much as substance

While testing, we realized that UX did not depend only on the speed and the quality of the translation, but also on other factors that are easy to counteract. A large part of the perceived slowness, in fact, depended on the dead time: the technical silence needed to determine the end of an exchange, for example, and the delay in the audio starting.

To counter this perception we orchestrated techniques that radically improved the experience without improving the models, such as:

• Giving the user immediate feedback, showing the transcription of what was heard straight away.
• Starting to load the model while the user is still speaking.

Where we stand with offline translated conversations

What is the current situation on the market with offline translations? On the one hand, a great many tools now make it possible to translate texts offline. Google Translate has been doing it forever and Apple has its own version too, albeit limited to about twenty languages.

When it comes to conversation mode, on the other hand, is it possible to find offline alternatives? We were certainly not the first to think of it: on the Google support forums users have been asking for it for years (as in this ticket, for example). At the moment, it seems that Google Translate is working on it, but it is not an available feature yet. As for Apple, the function does exist, but the limitation to a few of the most popular languages remains.

A growing landscape

So we have seen that it makes sense to invest in offline translation, as the big tech companies are doing, because the landscape still has its gaps.

It is not only the giants that are on the move, though: the open-source world is teeming with new projects and you can find speech recognition models for over 1,600 languages, natural synthetic voices, high-quality compact translators, almost all of it freely usable, even for commercial products.

Moreover, from a hardware point of view, phone chips have dedicated AI accelerators. The very models that up until a couple of years ago would have needed a server to run today work smoothly on the device.

The limitations that are still there

Despite the acceleration of recent years, there are still obstacles to face. We have already mentioned some of them in the results of our research. Older phones need lighter models in order to achieve acceptable performance. This means having to design differently for different users.

Moreover, as already mentioned, less popular languages have few arrows in their quiver. For non-European languages, premium synthetic voices or transcription models often do not exist. What is more, even languages that appear to be covered, such as Arabic, in reality often only include the “standard language”, with no consideration whatsoever for the dialects actually spoken by potential users.

Finally, adding complication to an already complex landscape, there are also licenses: some models, especially those that perform better on rarer languages, are released under research-only licenses (and therefore cannot be used in a commercial product such as VoiceLink) or cannot be used in the European Union because of territory-related clauses.

What we take away from this research

What we understood from this research is that on-device AI has gone from being a hypothesis to being feasible, even if with a gap between devices.

Above all, we understood that orchestrating small, specialized models produces better results than using a single all-inclusive model, and that there is still a long way to go for smaller language communities.

What we have learned will probably open the door to new features on VoiceLink in the near future. Don't miss what's in the pipeline; subscribe to the Mabiloft newsletter (you will find it in the footer) to stay up to date, or download VoiceLink for the next updates.