Production Self-Hosted Voice AI Platform For Data-Residency-Sensitive Teams

The problem

Voice AI platforms like Parloa, Cognigy, Vapi, and Retell are useful, but enterprise teams often need more control than a hosted voice SaaS gives them. The recurring concerns:

• Where does call data go?
• Can the system run in a trusted VPC?
• Can STT, LLM, and TTS providers be swapped?
• Can costs be controlled at scale?
• Can learnings from calls feed back into the business?
• Can the workflow be reviewed before changes reach production?

The solution

A production-oriented self-hosted voice AI setup with operational control as the design centre:

• Dograh for open-source voice-agent orchestration.
• GPU-backed local STT, LLM, and TTS inference behind OpenAI-compatible endpoints so individual providers can be swapped without rewiring callers.
• Persistent model/runtime volume so compute can be ramped down and back up without long re-download cycles.
• Evidence artifacts for machine proof, model preload, health checks, benchmark, and smoke tests.
• Runbook for recreating the setup on STACKIT, PlusServer, or in a client-selected VPC.

Client context before the first word

A key design point is pre-call context lookup. Before the caller starts speaking, the system can retrieve known account or client information and use it to adapt: greeting, tone, product context, support tier, next best question, routing decision, handoff threshold. This makes the agent behave less like a generic bot and more like a prepared operator who knows who is calling.

What we measured

The validation environment ran on STACKIT in DE-Frankfurt with a dual-GPU stack — an NVIDIA L40S handling ASR and LLM orchestration, an NVIDIA L4 streaming TTS under the 200 ms latency threshold — and a dedicated 300 GB persistent volume. The pairing is the DACH industry standard for multilingual voice AI handling regional dialects (Swiss German, Austrian Bavarian). The local voice AI layer completed chat response generation, text-to-speech audio generation, speech-to-text transcription, health checks, and a warm benchmark.

Business outcome loop

The system is designed to write structured session outcomes back to a backend so calls become measurable business progress, not just “a voice bot answered.”

Per-session metrics tracked: call answered, call completed, successful handoff, resolved without handoff, qualified lead, disqualified lead, reason for disqualification, unresolved question, objection category, follow-up needed, estimated value, cost per completed call, cost per qualified lead.

A successful handoff means the agent identified that a human should take over, the destination was correct, the human received context, and the caller did not need to repeat the whole story. Sample payload:

{
  "session_id": "sess_123",
  "handoff_target": "sales_engineering",
  "caller": {
    "company": "Acme GmbH",
    "support_tier": "priority"
  },
  "reason": "VPC deployment and security review question",
  "summary": "Caller wants self-hosted voice AI in their own VPC and asked about data residency.",
  "recommended_next_action": "Schedule technical architecture call."
}

{
  "session_id": "sess_123",
  "type": "knowledge_gap",
  "source": "voice_call",
  "text": "Caller asked whether STT can run fully inside an EU VPC.",
  "recommended_action": "Add VPC-local STT section to security FAQ.",
  "priority": "high"
}

POST /call-session/start
POST /call-session/event
POST /call-session/outcome
POST /handoff-summary
POST /learning-items

Data model: caller_profiles, call_sessions, call_events, handoffs, outcomes, learning_items, agent_versions, workflow_versions.

Guardrail

Do not let one session automatically rewrite production behavior. The recommended flow:

1. Write learning item.
2. Group similar items.
3. Review.
4. Update prompt / workflow / knowledge base.
5. Version change.
6. Test.
7. Publish.

Five capabilities this proves

1. Ultra-low latency path through local GPU inference.
2. Provider-agnostic model flexibility.
3. Human-style interaction handling through context and handoff logic.
4. Transient specialist agents behind the live voice agent.
5. Enterprise workflow control through visual guardrails and persistence.

Honest status

The infrastructure and local inference proof are complete. The next production proof is to wire Dograh orchestration into the local inference endpoints, add pre-call client lookup, and add backend learning writeback.

For orientation first: a full vendor comparison — Vapi vs Retell vs Parloa vs Cognigy vs Dograh with NIS2 scoring and cost matrix is available as a separate guide.

If you are running voice AI on a hosted SaaS today and the data-residency, provider-swap, or business-outcome questions above are starting to bite, I am happy to walk through what a self-hosted deployment would look like for your specific stack — thirty minutes, no slide deck.