What is automatic speech re­cog­ni­tion (ASR)?

Automatic speech re­cog­ni­tion is a process for auto­mat­ic­ally con­vert­ing speech into text. ASR tech­no­lo­gies use machine learning methods to analyse, process and output speech patterns as text. From gen­er­at­ing meeting tran­scrip­tions and subtitles to virtual voice as­sist­ants, automatic speech re­cog­ni­tion is suitable for a wide range of use cases.

What does automatic speech re­cog­ni­tion mean?

Automatic speech re­cog­ni­tion (ASR) is a subfield of computer science and com­pu­ta­tion­al lin­guist­ics focused on de­vel­op­ing methods that auto­mat­ic­ally translate spoken language into a machine-readable form. When the output is in text form, it’s also referred to as Speech-to-Text (STT). ASR methods are based on stat­ist­ic­al models and complex al­gorithms.

How does automatic speech re­cog­ni­tion work?

Automatic speech re­cog­ni­tion consists of multiple con­sec­ut­ive steps that seam­lessly integrate. Below we outline each phase:

1. Capturing speech (automatic speech re­cog­ni­tion): The system captures spoken language through a mi­cro­phone or other audio source.
2. Pro­cessing speech (natural language pro­cessing): First, the audio recording is cleaned of back­ground noise. Then, an algorithm analyses the phonetic and phonemic char­ac­ter­ist­ics of the language. Next, the captured features are compared to pre-trained models to identify in­di­vidu­al words.
3. Gen­er­at­ing text (speech to text): In the final step, the system converts the re­cog­nised sounds into text.

Close

Comparing ASR al­gorithms — Hybrid approach vs deep learning

There are generally two main ap­proaches to automatic speech re­cog­ni­tion. In the past, tra­di­tion­al hybrid ap­proaches like stochast­ic Hidden Markov models were primarily used. Recently, however, deep learning tech­no­lo­gies have been in­creas­ingly employed, as the precision of tra­di­tion­al models has plateaued.

Tra­di­tion­al hybrid approach

Tra­di­tion­al models require force-aligned data, meaning they use the text tran­scrip­tion of an audio speech segment to determine where specific words occur. The tra­di­tion­al hybrid approach combines a lexicon model, an acoustic model and a language model to tran­scribe speech:

• The lexicon model defines the phonetic pro­nun­ci­ation of words. A separate data or phoneme set must be created for each language.
• The acoustic model focuses on modelling the acoustic patterns of the language. Using force-aligned data, it predicts which sound or phoneme cor­res­ponds to different segments of speech.
• The language model learns which word sequences are most common in a language, aiming to predict the words most likely to come next in a given sequence.

The main drawback of the hybrid approach is the dif­fi­culty in in­creas­ing the accuracy of speech re­cog­ni­tion using this method. Ad­di­tion­ally, training three separate models is very time- and cost-intensive. However, due to the extensive knowledge available on how to create a robust model using this approach, many companies still go for this option.

Deep learning with end-to-end processes

End-to-end systems can directly tran­scribe a sequence of acoustic input features. The algorithm learns how to convert spoken words using a large amount of paired data. The data pairs are comprised of an audio file con­tain­ing a spoken sentence and the cor­res­pond­ing tran­scrip­tion of the sentence.

Deep learning ar­chi­tec­tures such as CTC, LAS and RNNT can be trained to deliver precise results even without using force-aligned data, lexicon models or language models. Many deep learning systems are still paired with a language model though, as it can further enhance tran­scrip­tion accuracy.

The end-to-end approach for automatic speech re­cog­ni­tion offers greater accuracy than tra­di­tion­al models. These ASR systems are also easier to train and require less human labour.

What are the main ap­plic­a­tions for automatic speech re­cog­ni­tion?

Thanks to advances in machine learning, ASR tech­no­lo­gies are becoming in­creas­ingly accurate and more powerful. Automatic speech re­cog­ni­tion can be used across various in­dus­tries to increase ef­fi­ciency, improve customer sat­is­fac­tion and/or boost ROI. The most important areas of ap­plic­a­tion include:

• Tele­com­mu­nic­a­tions: Contact centres use ASR tech­no­lo­gies to tran­scribe and analyse customer con­ver­sa­tions. Accurate tran­scrip­tions are also needed for call tracking and for phone solutions im­ple­men­ted via cloud servers.
• Video platforms: The creation of real-time subtitles on video platforms has now become an industry standard. Automatic speech re­cog­ni­tion is also helpful for content cat­egor­isa­tion.
• Media mon­it­or­ing: ASR APIs make it possible to analyse TV shows, podcasts, radio broad­casts and other types of media for brand or topic mentions.
• Video con­fer­en­cing: Meeting solutions like Zoom, Microsoft Teams and Google Meet rely on accurate tran­scrip­tions and content analysis to generate key insights and guide relevant actions. Automatic speech re­cog­ni­tion can also provide live subtitles for video con­fer­ences.
• Voice as­sist­ants: Virtual as­sist­ants like Amazon Alexa, Google Assistant and Apple’s Siri rely on automatic speech re­cog­ni­tion. This tech­no­logy allows the as­sist­ants to answer questions, perform tasks and interact with other devices.

What role does ar­ti­fi­cial in­tel­li­gence play in ASR tech­no­lo­gies?

Ar­ti­fi­cial In­tel­li­gence helps improve the accuracy and overall func­tion­al­ity of ASR systems. In par­tic­u­lar, the de­vel­op­ment of large language models has led to a sig­ni­fic­ant im­prove­ment in pro­cessing natural language. A large language model can not only perform trans­la­tions and create complex texts that are highly relevant, it can also recognise spoken language. ASR systems benefit greatly from ad­vance­ments in this area. AI is also be­ne­fi­cial for the de­vel­op­ment of accent-specific language models.

What are the strengths and weak­nesses of automatic speech re­cog­ni­tion?

Compared to tra­di­tion­al tran­scrip­tion, automatic speech re­cog­ni­tion offers several ad­vant­ages. A key strength of modern ASR processes is their high accuracy, stemming from the ability to train these systems with large datasets. This enables improved quality in subtitles or tran­scrip­tions, which can also be provided in real time.

Another major benefit is increased ef­fi­ciency. Automatic speech re­cog­ni­tion allows companies to scale, expand their service offerings faster and reach a larger customer base. ASR tools also make it easier for students and pro­fes­sion­als to document audio content, for example, during a business meeting or uni­ver­sity lecture.

While more accurate than ever before, ASR systems still cannot match human accuracy though. This is largely due to the many nuances in spoken language. Accents, dialects, tone vari­ations and back­ground noise remain chal­len­ging for these systems, with even the most powerful deep learning models unable to handle input that doesn’t match expected or typical patterns. Another concern is that ASR tech­no­lo­gies often process personal data, raising issues regarding privacy and data security.