Real-Time Conversational AI & Multi-LLM Orchestration

System Architecture

[ Client Web App ]
       │ (SSE Stream)
       ▼
[ FastAPI Gateway / Auth ] ────► [ Redis Session / Cache ]
       │
       ├──► [ Dynamic LLM Router ]
       │         │
       │         ├──► [ Primary: OpenAI GPT-4o ]
       │         ├──► [ Fallback: Anthropic Claude 3.5 ]
       │         └──► [ Local/Swiss: Self-Hosted Mistral ]
       │
       └──► [ Background Celery Workers ]
                 │
                 ├──► [ Document Parsing & Chunking ]
                 └──► [ Vector Indexing / Swiss PostgreSQL + pgvector ]

Core Technical Challenges

• Provider Rate Limits & Degradation: Enterprise SLA required 99.9% uptime despite unpredictable upstream vendor latency spikes and HTTP 429 rate limit errors from LLM providers.
• Low Latency Token Delivery: Standard HTTP REST endpoints introduced unacceptable latency for conversational interfaces, necessitating asynchronous Server-Sent Events (SSE).
• Data Privacy & Compliance: Enforcing Swiss DSG and European GDPR data sovereignty rules for document embeddings without sacrificing RAG query performance.

Implementation & Engineering Highlights

Asynchronous Multi-LLM Dynamic Routing

Designed a resilient provider routing engine in FastAPI that evaluates health checks, rate limits, and latency profiles in real time. If the primary provider experiences performance degradation, traffic transparently fails over to secondary endpoints (e.g. Anthropic Claude 3.5 Sonnet or self-hosted Mistral) with zero dropped connections.

Token Streaming with Server-Sent Events (SSE)

Engineered an asynchronous HTTP response pipeline streaming tokens directly from LLM response generators to the client web application. This architecture cut Time-To-First-Token (TTFT) to under 300ms while maintaining memory-efficient non-blocking socket loops.

Swiss DSG-Compliant RAG Pipeline

Built background document ingestion pipelines using Celery and Redis. Documents undergo automated text extraction, chunking, and embedding creation. Embeddings are stored and queried inside PostgreSQL using pgvector hosted in Swiss AWS data privacy boundaries with strict tenant-scoped access controls.

Key Impact & Metrics

• < 300ms TTFT: Delivered near-instantaneous streaming responses to end users across all model providers.
• 99.95% Gateway Availability: Multi-provider fallback routing eliminated downstream service outages caused by vendor rate limits.
• Zero Swiss DSG Violations: Guaranteed strict data isolation and in-region vector processing for all enterprise tenant documents.