Interruptible Conversational Voice AI — Deployment Guide

View source on GitHub
Preset

Cloud or OpenAI-Compatible LLM

Speech runs on the device; replies are generated by Qwen API or the OpenAI-compatible endpoint you enter. The default is Alibaba Cloud Model Studio's Beijing endpoint with qwen3.5-flash; replace the base URL, key, and model ID for another provider.

  • Microphone: a reSpeaker XVF3800 USB microphone array is required (a plain USB mic causes echo and false interruptions).
  • Network: required for every conversation turn.
  • API key: enter an API Key and a Model ID at deploy time; the model must support streaming Chat Completions.

Optional wake word: select Wake word required in the deployment form and enter a short Chinese or English phrase; a short tone plays when it is detected. Always listening is the default.

1

Deploy the Cloud-Backed Voice Terminal

After deployment, you can interrupt a spoken answer at any time.

Deployment method

Run speech on RK3576 and connect to a cloud or LAN model. Requires at least 12 GB free disk.

Wiring
  1. Connect the reSpeaker XVF3800 and speaker
  2. Fill in the SSH details and the model endpoint settings
What you will be asked for(filled in the SenseCraft Solution app)
  • Conversation language*default zh
    2 options
    • · Chinese
    • · English

    Spoken language for recognition and for the synthesized reply. The list is the RK runtime's 30-language set (openvoicestream matrix language_catalog_source), the narrower of the two available lists - Qwen3-ASR upstream advertises 52 and Whisper 99. RK3576 serves Chinese and English only: its TTS bundle is Matcha icefall zh-en, and the multilingual Kokoro bundle on this board is TTS-only, so no profile pairs it with an ASR backend yet. The deployment resolves (language, device) to one speech profile before any service starts and stops with an error if this board cannot serve the language.

  • Device IP*
  • SSH Username*default cat
  • SSH Password*
  • OpenAI-compatible Base URL*default https://dashscope.aliyuncs.com/compatible-mode/v1
  • API Key*
  • Model ID*default qwen3.5-flash
  • Assistant personality*default You are a natural, witty voice assistant. Reply in the user's language. Keep responses conversational and brief—usually one or two sentences. Add light humor and personality when appropriate, but never at the expense of accuracy. Avoid rambling, repetition, lectures, Markdown, headings, and lists. If details are missing, ask one short question. For spoken Chinese, write numbers, dates, times, money, and units as natural spoken Chinese.

    Editable system prompt applied to every conversation.

  • RK Performance Mode*default performance
    2 options
    • · Maximum performance
    • · System default / balanced

    Lock CPU, NPU, DDR, and GPU (when present) to their highest reported frequencies, or keep the system governor unchanged.

4 more with defaults you can keep
  • Conversation modedefault always_on
    2 options
    • · Always listening
    • · Wake word required
  • Wake worddefault 你好小智

    Any short Chinese or English phrase compiled at startup by the open-vocabulary KWS backend.

  • Wake-word sensitivitydefault 0.25
    3 options
    • · Strict (0.35)
    • · Balanced (0.25)
    • · Sensitive (0.15)

    Lower threshold is more sensitive and may increase false triggers.

  • Reply after how much silencedefault 0
    4 options
    • · Balanced (image default
    • · Snappy (0.9 s)
    • · Tolerant (2 s)
    • · Very tolerant (3 s)

    Server-side speech endpoint: how long the assistant waits after you stop speaking before it replies. The image default (1.5 s) tolerates a natural mid-sentence pause; a smaller value feels snappier but can cut one sentence into two; a larger value tolerates long thinking pauses at the cost of a slower reply. Must stay above the device-side VAD silence (600 ms). Only affects the paths where the device itself decides the end of speech — a separate microphone/agent application keeps its own setting.

Troubleshooting
SymptomAction
API returns 401Confirm the API key belongs to the endpoint's region
Speech is recognized but no reply playsCheck the agent log for LLM request errors and confirm the model supports streaming
No audio after replugging the microphoneConfirm the new Agent image is running
After Deployment

Ask a question. Within one second of playback starting, speak again; the current answer should stop immediately.

2

Verify Conversation and Barge-in

Watch the listening, thinking, speaking, and barged-in states on the dashboard.

Troubleshooting
SymptomAction
State changes but there is no soundCheck that the default playback device is the speaker
Room noise triggers interruptionsConfirm the microphone is the reSpeaker XVF3800, then raise the client VAD threshold slightly; do not mute capture
After Deployment

Three normal turns plus an immediate stop when you say “wait” during playback pass this check.

3

Verify the Selected Language

Confirm that recognition and spoken replies use the language chosen at deploy time.

Troubleshooting
SymptomAction
Replies come back in the wrong languageCheck the assistant personality prompt
Deployment never startedThe selected language is not supported on this board; pick a supported one
Transcript language is right but audio is wrongReport the language and the board; do not edit the speech profile yourself
After Deployment

Acceptance checklist

  1. curl -fsS http://<device-ip>:8621/health returns success.
  2. Ask one question; a spoken reply starts within a few seconds.
  3. Speak again within one second of playback starting; the current answer stops immediately.
  4. Two or three turns are transcribed and answered in the selected language.
  5. On the device, docker compose -p conversational_voice_ai -f ~/conversational_voice_ai/assets/docker/docker-compose.<target>.yml logs --since 10m | grep -i error returns nothing (<target> is rk3576, rk3588, jetson, or rpi5).