Surogate Speech

We build agents that talk to people, on the phone and in chat, and often not in English. The speech models we tried kept getting amounts and dates wrong, or they needed a large GPU in someone else's cloud. So we trained our own. Surogate Speech is the result: Jackrabbit turns speech into text (ASR) and Amami turns text into speech (TTS).

We started with Romanian:

• jackrabbit-110m-ro: speech recognition, 116M, TDT and CTC decoders, optional 4-gram LM
• jackrabbit-110m-ro-streaming: the same model for live audio, ~1 s lookahead
• amami-357m-ro: text-to-speech, three voices (Doina, Tudor, Radu), runs on a plain CPU

Each model is named after a rabbit. The jackrabbit has the biggest ears of any hare, and the Amami rabbit is one of the few rabbits that makes a sound.

Jackrabbit: Automatic Speech Recognition (ASR)

Jackrabbit listens. We train a separate model for each language. For Romanian there are two, built on the same 116M FastConformer, and both write Romanian with capital letters and punctuation:

• jackrabbit-110m-ro: for recorded audio files. TDT and CTC decoders share one encoder, and a 4-gram language model improves the CTC output
• jackrabbit-110m-ro-streaming: for live audio. It looks about one second ahead

Both models run on an ordinary CPU. The offline model is also available as a GGUF file for NVIDIA's NeMo-Speech.cpp runtime, so you don't need PyTorch to run it.

While you speak, the streaming model shows a draft of what you said. When you pause, it reads the whole sentence again and writes the final text. On FLEURS-ro the final text has a 7.03% word error rate, and it appears about 0.72 seconds after you stop talking on an RTX 5090.

Benchmark results

A lower word error rate is better.

Amami: Text-To-Speech (TTS)

Amami speaks, with its own voices for each language. amami-357m-ro is a 357M Magpie-TTS model with three built-in voices: Doina, Tudor and Radu. It reads numbers, dates, IBANs and abbreviations the way a Romanian speaker would. It runs on a CPU, without a GPU or PyTorch.

There are four builds:

• cpu/: native runtime for Linux x86-64, on any CPU with AVX2
• gpu/: the same runtime for NVIDIA GPUs from Ampere to Blackwell (A100, L4, RTX 30/40/50, H100, B200). On an RTX 5090 it speaks about 25 times faster than real time
• macos/: Apple Silicon with Metal, about real time on an Apple M3 with 8 GB
• nemo/: the PyTorch package, for everything else

We care most about being understood. If an agent reads out the wrong amount or the wrong date, the call has failed, however pleasant the voice. It also has to start speaking quickly, and it has to run on the same machine as the rest of the agent, which usually has no GPU.

We tested it on the Romanian part of the MiniMax multilingual test set, 100 sentences, with Whisper large-v3 writing down what each voice said:

The three voices are fixed, and this release can't clone voices. If you need a custom voice, talk to us.

How it fits together

Surogate Speech is one part of a larger stack:

• surogate: our engine for training and serving models. It serves all three speech models through OpenAI-compatible audio endpoints, with no Python in the serving path
• surogate-speech: the local clients, the evaluation code behind every number in this post, and the test sets we used
• surogates: our open platform for running managed agents at scale. These models are how its agents listen and talk
• surogate.ai: the hosted version of Surogate, if you'd rather build and run agents without managing the infrastructure
• the weights: CC-BY-NC-4.0 for research and non-commercial use. For a commercial license, contact Invergent

Run it

You can serve all three models with the Surogate engine:

surogate serve --stt surogate/jackrabbit-110m-ro             # files, /v1/audio/transcriptions
surogate serve --stt surogate/jackrabbit-110m-ro-streaming   # live, /v1/audio/streams
surogate serve --tts surogate/amami-357m-ro --device 0       # speech, /v1/audio/speech

curl http://localhost:8000/v1/audio/speech -H "Content-Type: application/json" \
  -d '{"model": "surogate/amami-357m-ro", "voice": "Doina", "input": "Bună ziua! Cu ce vă pot ajuta?"}' -o doina.wav

Or use the local clients:

pip install "surogate-speech[asr,tts] @ git+https://github.com/invergent-ai/surogate-speech"
surogate-speech transcribe interviu.wav --lm
surogate-speech listen
surogate-speech speak "Plata de 327,45 lei a fost programată pentru 3 octombrie." --voice Tudor -o tudor.wav

Or load Jackrabbit directly in NeMo:

from nemo.collections.asr.models import ASRModel
model = ASRModel.from_pretrained("surogate/jackrabbit-110m-ro")
print(model.transcribe(["audio.wav"])[0].text)

What's next ?

Next we'll add more European languages, one at a time. We'll start with the ones large models handle worst and the ones customers ask for. Each will get its own Jackrabbit and Amami, tested and published the same way as this release.

Models: collection · engine: surogate · toolkit and evals: surogate-speech

Get started on surogate.ai.