Voice assistant

People often come to me with questions as an expert in PDP-11 assembly language. I used to answer them in text messages, then switched to voice messages to save time. It was still a waste of time.

Build a voice assistant into my website manwe.ru/bk0010 that would give people consultations based on the knowledge base I’ve built up. The website is static HTML, with articles in Markdown.

I chose x.ai as the provider. Connecting to it from the browser requires an API key, but storing it in a static website isn’t safe. The solution: the browser requests a token from a small PHP script, and x.ai issues a temporary token valid for 5 minutes. With it, the browser opens a WebSocket connection to the agent. The real key stays only on the server, with access closed. The script checks where the request comes from and limits the number of tokens per IP address per hour.

While implementing it, Claude wrote a Cloudflare Worker for issuing tokens, although I hadn’t asked for it. I had to explain how to do it in PHP. Over-engineering is a typical problem of AI assistants.

Even though the widget is added with a single line of code, transcription of the user’s voice didn’t work. Explicitly enabling the option in the session settings helped.

I wanted users to be able to talk and type at the same time, but by default that didn’t work: a typed message split the voice utterance into parts. I had to bind the voice transcription to the utterance ID and create a separate ID for typed text.

A pause in dictation also split a phrase into parts, and with a repeat: after a pause the server created a new utterance, and for some reason its transcription contained what had already been said in the previous one. Comparing the content of the last two utterances helped: if they match, they are merged under one ID.

The website is a single page: articles load without a reload. The agent receives a system message with the content of the page (conveniently, it is stored in Markdown in the first place). The builder instructions tell it to answer from the context of the current page first, and only then turn to the knowledge base.

This turned out to use fewer tokens than passing the agent the page’s web address for it to parse.

Every article sent stays in the conversation history, so the context grows. But for short dialogues that’s even a plus.

The voice assistant sends plain text, and my widget finds addresses in it (like https://) and styles them as clickable links.

The agent’s instructions say: if the answer comes from an external website, add ‘Source: address’ at the end. The widget shows the source link as a tag and removes the word ‘Source’ from the text.

I quickly realised that the UX of the x.ai widget was no good. For example, why can’t you start typing straight away? The input field is inactive until the connection to the server is established. That makes no sense. In 99% of cases the connection is established before the user finishes typing their question. I allowed typing and pressing Enter to send right away, but the message is actually sent only once the connection is established.

Why write ‘Send a message’ in the input field and put a microphone button on the right? It confuses the user: should they type or talk? Besides, the right side is usually where the Send button goes. I simplified the interface as much as possible: the field has no placeholder, but the cursor always blinks, and the microphone is on and shows the signal level. At first glance it’s clear that the system is active and accepts both voice and text input at the same time. What you say appears in the chat right away, and what you type appears after you press Enter. The red colour shows that recording is on. Pressing a red button is a familiar pattern, so it’s easy to guess how to stop recording.

The work took one day, not counting the knowledge base, which I had been building up and structuring for years.

One session covering several topics costs about $1. And I no longer spend my own time on it.

It’s difficult to set up different rules for text and voice in a single model. It became clear that two models are needed: one to generate the text and another to voice it. Then the text can contain links, and they won’t reach the voice model. Specialised terms can be converted into phonetic transcription before being sent to the voice model. There is plenty of room for improvement.