AI Voice Module - Edge-Cloud AI Interaction System
In ProgressAI Engineering
An edge-cloud AI voice interaction system experiment for multiple device carriers, exploring how STT, LLM, TTS, agent tools, runtime configuration, and observability form a low-latency maintainable voice pipeline.
Project Profile
- Phase
- In Development
- Visibility
- Project Archive
- Next Step
- Continue Trace, latency, Apply / Reload, tool-list consistency, and end-to-end stability work.
- Project Relation
- A deeper AI Engineering project for edge interaction, cloud orchestration, Agent tools, and runtime configuration.
- Boundary
- Multi-device collaboration and one-brain-many-forms remain future directions; no internal endpoints, secrets, accounts, MQTT defaults, or production APIs are public.
- Tech Stack
- Python
- Rust
- WebSocket
- gRPC
- SenseVoice
- Qwen TTS
- LLM
- MCP
- MySQL
- Docker
- Kubernetes
One-Line Positioning
The AI Voice Module is an edge-cloud AI interaction system experiment. The core question is not only whether a model can be called, but whether a real voice system can stay stable, low-latency, configurable, observable, and continuously iterated.
It can be understood as a voice intelligence layer that may be mounted on different terminal carriers. The edge handles real-time interaction and device access, while the cloud handles model orchestration, tool use, runtime configuration, and trace observability.
Project Background
Voice interaction puts audio capture, VAD, network transport, STT, LLM, TTS, playback queues, interruption control, and device state into the same chain. Any jitter becomes visible to the user immediately.
This project is where I study those engineering problems: how to separate edge and cloud responsibilities, how to organize the STT / LLM / TTS pipeline, how to make agent tool calls auditable, and how runtime configuration can safely affect behavior instead of staying in static files.
Key Pipeline
User speech -> Edge VAD -> Audio upload -> STT -> LLM -> Tool / Agent -> TTS -> Streaming playback -> Interruption controlThe key challenge is not merely connecting the modules. It is controlling end-to-end latency, handling interruptions, managing playback queues, and keeping graceful degradation paths when a model or network segment fluctuates.
Technical Architecture
Edge
Real-Time Interaction
- Wake-up / audio capture
- WebRTC VAD / interruption
- TTS playback / audio ducking
- Device access / WebSocket client
Cloud
Intelligence Orchestration
- Gateway / session runtime
- STT Service / LLM Service / TTS Service
- MCP / Tool Service
- Trace / observability
Admin
Configuration Governance
- Bot / Robot / Agent config
- MCP Server management
- Model parameters / runtime config
- Apply / reload / status view
Edge Layer
- Wake-up, audio capture, WebRTC VAD, TTS playback, voice interruption, local audio ducking, device access, and WebSocket client.
- The edge prioritizes interaction latency and device stability, not complex model inference.
Cloud Layer
- Gateway, STT Service, LLM Service, TTS Service, MCP / Tool Service, Runtime Config, and Trace / Observability.
- Client-to-cloud communication uses WebSocket. Service-to-service communication is designed around gRPC. Admin configuration is stored in runtime configuration tables.
Current Model Direction
- STT: SenseVoiceSmall.
- LLM: Qwen / OpenAI-compatible API.
- TTS: Qwen TTS Realtime / qwen3-tts-flash-realtime.
- Tool protocol: MCP / Function Calling / local Agent.
Current Progress
Completed
- Basic edge-cloud chain and WebSocket client access
- Basic STT / LLM / TTS orchestration
- Basic interruption handling and partial MCP tool calls
- Admin configuration prototype
In Progress
- End-to-end latency optimization and trace observability
- Agent tool inventory governance
- Admin Apply / Reload stability
- End-to-end testing and runtime state governance
Future Plan
- Multi-agent scenario templates and robot task orchestration
- More complete Admin UI and public demo page
- Stronger observability
- Multi-device collaboration remains a future direction, not a delivered capability
Engineering Signals
First Audio Packet
Target: keep close to 800ms where possible.
Why it matters: determines whether the interaction feels real-time.
End-to-End Reply
Target: keep close to 2s where possible.
Why it matters: STT, LLM, and TTS latency compounds quickly.
Interruption Response
Target: stop playback and switch state quickly.
Why it matters: makes the system feel conversational instead of scripted.
Degradation Path
Target: models, tools, and TTS providers remain replaceable.
Why it matters: prevents one dependency from breaking the full loop.
Observability
Target: correlate STT, LLM, TTS, and tool traces by session ID.
Why it matters: makes slow requests, failed tools, and abnormal states debuggable.
Relationship with Other Projects
The AI Voice Module is one AI Engineering practice inside the project map. It is not the only narrative of wzk.icu and is no longer mixed with AscendLab. AscendLab focuses on public tool experiments; this page records the engineering boundary and staged progress of the edge-cloud voice system.
Risks and Boundaries
- This is a personal AI engineering experiment, not a public product with an SLA.
- Completed capabilities, in-progress work, and future vision are separated to avoid presenting unfinished multi-device collaboration as delivered functionality.
- This page does not expose internal addresses, secrets, servers, accounts, MQTT defaults, production APIs, or company-sensitive information.
- Multi-device collaborative execution remains a future direction, not a fully delivered capability.
Project Asset
Asset Connections
How this project connects to topics, reading paths, series, and future writing.Reading Paths
- AI Engineering TopicSTT / LLM / TTS / Agent application path
- AI Infra TopicModel services, Trace, observability, and cost governance
Related Series
- LLM Application DevelopmentAgent, tool calling, and RAG foundations
- AI Engineering & ResearchAI engineering context and trends
Related Entries
- Voice Interaction LatencyLatency breakdown entry
- LLM Production EngineeringEngineering issues from demo to production
Project Timeline
- Base PathEdge-cloud split and STT / LLM / TTS orchestration
- Stability StageTrace, latency, interruption, Apply / Reload, and end-to-end tests
Next Content Plan
- Add a clearer edge / cloud / admin architecture explanation.
- Keep recording STT / LLM / TTS, MCP / Tool / Agent, Trace / Observability, and Apply / Reload.
- Keep one-brain-many-forms marked as a future direction.
Project FAQ
- Is one-brain-many-forms fully delivered?
- No. It is a future direction; the public page records the edge-cloud voice module as an experiment.