Every voice AI demo I found started with a telecom bill. This one starts with a chat app I already had.
I wanted to talk to my agent from my phone without holding it. Every route I looked at, the hosted voice platforms, wanted a monthly charge per number plus a per minute rate on top. Nobody has ever wanted a phone number. They wanted to be reachable, and I am reachable by other means.
So the loop is: a voice note or a voice channel in a chat app, speech to text, the agent, text to speech, audio back. All of it running on a small VPS I already pay for.
phone: voice note
|
v
[ speech to text ] --> text
|
v
[ the agent ] tools, memory, a reply
|
v
[ text to speech ] --> audio
|
v
phone: it talks back
The models are the easy part now. Transcription is largely solved and there are several good options. For speech out I benchmarked two on the same three sentences. One was a 31 million parameter model quantized to q8, the other was Kokoro in fp16. The small one ran at roughly four times real time and used about 300 MB of memory. Kokoro managed under twice real time and needed about 630 MB.
I went with Kokoro anyway. The faster model sounded like a phone menu, and once you hear both side by side you cannot unhear it. The price is latency and memory, both of which hurt on a box this small.
The part nobody warns you about is turn taking. Knowing that a person has finished talking is not solved, and getting it wrong is worse than any model choice.
too eager: you: "so the thing is--"
bot: "SO THE THING IS"
too patient: you: [done 2s ago]
bot: ... ... "hello?"
Too eager and it talks over you. Too patient and every reply lands after an awkward silence. Most of the tuning went into that gap.
There is also a failure mode with no error message. When the machine runs low on memory the bot does not crash. It reads your message and says nothing, and you find out hours later from the silence.
What this cannot do is make a phone call. There is no ordinary telephone leg anywhere, so nobody can ring it and it only works inside the app. That is the trade, honestly stated. A real number costs money every month and buys reach. This costs almost nothing and buys a demo that only the people you send an invite to can use.
For what I actually wanted, which is to talk to something I built while walking around the house, that was enough.