How to set up voice recognition of subscribers’ messages in your chatbot

With SendPulse, you can connect Whisper — OpenAI's automated speech recognition system — to your chatbots. The integration allows you to recognize and transcribe voice messages from subscribers, which allows them to easily interact with your chatbot.

In this article, we will talk about how the Whisper model works and how to set up it.

What is whisper?

Whisper is an automated speech recognition (ASR) system developed by OpenAI. ASR technology is designed to convert spoken language into written text.

Whisper supports popular languages, including English, Spanish, French, German, Italian, Portuguese, Dutch, Russian, Swedish, Chinese, Japanese, Korean, and Arabic. You can use the model, for example, for transcription, voice-controlled applications, as a voice assistant, etc.

The Whisper model is primarily focused on transcribing speech and converting it into text.

How does the whisper integration work?

The Whisper system is available for OpenAI chatbot integration, so it uses the same token as ChatGPT. You can set it up in the same modal window.

Read more: How to connect GPT to your chatbot.

You can connect ChatGPT and Whisper separately or together.

If only Whisper is enabled, the recognized text will be displayed in your chat.

If Whisper and ChatGPT are enabled, then when a subscriber sends a voice message to your bot, it will be recognized and sent to ChatGPT. Next, ChatGPT will generate a reply and send it to your subscriber as text.

If a subscriber uses a trigger word in their voice message, the linked flow will be launched.

If there is an active user input element in your chatbot flow, subscribers can record and send voice messages that will be converted into text and added to variables.

How to connect whisper to your chatbot

Choose a chatbot, and go to the Bot settings > Integrations.

You can enable voice recognition when integrating with OpenAI. Once you have set it up, click Enabled > Edit next to ChatGPT by OpenAI.

In the modal window, turn on the Whisper: Speech Recognition for Voice Messages toggle.

If you want ChatGPT to generate text replies, then turn on the ChatGPT: AI Text Response Generation toggle. If you want to use only speech recognition, turn off this toggle.

How to view voice messages

Go to the Conversations tab, your bot audience, or the Chats widget on the right to view all chats with subscribers.

When a subscriber sends a voice message to your chatbot, you'll see a message transcription at the bottom of their message.

If you have turned on the ChatGPT: AI Text Response Generation toggle in the integration settings, ChatGPT will generate a reply for you.

OpenAI uses an internal database to process your clients’ requests and send a reply based on the available information. It does not take your subscribers’ personal data into account.

Last Updated: 17.06.2024