What are the major challenges in Audio transcriptions via AI?

Hello, everyone. Welcome back to another episode of Question Minute. And today, I'm going to handle a question related to transcription handling using AI. So the question is, what are the major challenges in audio transcription via AI?

There are multiple questions. But personally, what I have seen is that some of the major challenges are related to, number one, poor audio quality. Sometimes there is lots of background noise, echoes, maybe microphone quality is not good, or compression is really high. So poor audio quality is one of the major challenges in doing correct audio transcription.

And another one is accent and pronunciation. Many transcriptions don't have any specific defined language. Today, we are living in a world of multiple languages. So people talk about things in multiple languages using multiple kinds of words. So in many situations, our model will just get confused about which language parser they need to parse or transcribe your audio with. And accent is also a major challenge because throughout the world, we have different people having different kinds of pronunciation or accent. Similarly, different languages also have major challenges. Many Indian languages, many other languages have these kinds of challenges.

And sometimes there can be multiple speakers in one of the transcription candidate audios, and it becomes really difficult to identify who is speaking when, because transcription happens in a way like there is a start to finish discussion. We are only doing transcription for the words, we are not identifying anything. So you need to go through another request to your AI model, a different LLM model entirely, which will do the difficult and tricky job of identifying the characters within the transcription.

And yes, sometimes, for example, somebody may be watching TV while doing the call recording. So in such a situation, things become very specific as far as the challenges are concerned.

Another challenge that I have seen is that sometimes people talk about very personal or even sometimes financial things during the transcriptions, which may not be the intended purpose of the transcription in the end. Like, for example, there may be an exchange of a credit card number during one of the transactions. So you need to be careful about all such situations.

And yes, there will always be contextual ambiguity, and hallucination is one of the major challenges.

One thing I usually recommend, and you can take it as a bonus, is that whenever you are creating a private AI deployment pipeline within your own network, please always try to build some kind of process like taking a hash of the file that is being transcribed, that is your transcription candidate in that case. If you can use the hash of that file to identify that file in the future, then maybe you can save lots of compute resources on processing the same file again and again. Sometimes the same files may be sent by multiple applications to your AI deployment, or to your AI model actually. If you are using such a hashing mechanism to identify if this has already been transcribed by the system earlier, then you may save lots of system resources by just returning the existing cache, unless they want it in a different way or through a different model.

Similarly, numbers, dates, etc. are also a major challenge. You need to put some kind of layered approach in identifying payment information, because the transcription process is not supposed to handle or identify those things.

Sometimes recordings are really long, and sometimes there is a good amount of accuracy issue. So whenever you are getting back a result from your transcription model, please always check that it is not repeating the words. Please always make sure that you are having some kind of quality control, some kind of ranking management around the output that your transcription model is generating.

Thank you.