The Problem: 700 Million People, Zero Good Solutions
Here's a scene that plays out in millions of homes: a Hindi-speaking grandmother wants to teach her English-speaking daughter-in-law how to make biryani. They jump on a video call. Mom starts explaining the layering technique — "Ab chawal ke upar dum lagao" — and the daughter-in-law stares blankly at her screen.
700 million people worldwide live in multilingual family households. Cross-cultural cooking is one of the most intimate forms of knowledge transfer, yet every existing tool fails at it.
❌ Video Calls (FaceTime, WhatsApp): No translation whatsoever.
❌ Interpreter Apps: No video, no visual context, no understanding of "that looks done."
❌ Kitchen Noise: Sizzling oil, clattering utensils, and running water trigger phantom translations in every speech-to-text system.
The core challenge isn't just translation. It's real-time, emotionally-aware, noise-resilient, vision-contextual translation — during a high-chaos activity (cooking) where timing is everything.
The Solution: BabelChef
BabelChef is a real-time bilingual video call translator purpose-built for cross-cultural cooking. Two people join a video call, each speaking their own language. The AI translates in real-time — preserving emotional tone, filtering kitchen noise, and understanding visual cooking context.
Wife speaks English → Gemini translates → Mom hears English audio as Hindi in real-time.
Both cameras → AI sees what's cooking → "That looks done" is translated with visual context.
7 Gemini Live API Features That Make This Work
BabelChef isn't a toy demo. It uses every major feature of the Gemini Live API to deliver a production-grade experience. Here's how each one contributes:
Session Resumption
Cooking takes 30–60 minutes. WebSocket connections have limits. Session Resumption via
SessionResumptionConfig ensures the translation stays alive through the entire
biryani — no dropped connections, no lost context.
Context Window Compression
After an hour of conversation, the context window fills up. Context Window Compression automatically compresses older dialogue, so translation quality stays sharp even in marathon cooking sessions.
Input Audio Transcription
Shows the original speech as subtitles in real-time. Mom says "Masale daal do!" — and the original Hindi appears on screen, so nothing is lost.
Output Audio Transcription
Shows the translated output as subtitles. "Add the spices!" appears below the original — both speakers can read along while hearing the translation.
Affective Dialog
When Mom excitedly shouts "Wah! Bahut accha!" — the translation doesn't come out flat. Affective Dialog preserves the joy, urgency, and tenderness of the original speaker's tone.
Proactive Audio (Noise Filtering)
This is the game-changer. Kitchens are noisy — sizzling oil, running water, clattering pans. Without Proactive Audio, these sounds trigger phantom translations. With it, only intentional speech is translated. Silence stays silent.
Function Calling (Cooking Tools)
Mid-conversation, ask "How many grams is 2 cups of flour?" or "What does tadka mean?" The agent invokes cooking tools — measurement converter, timer suggestions, and culturally-aware term explanations — without breaking the conversation flow.
How I Built It: Google AI Models + Google Cloud
The Agent Architecture
BabelChef runs three concurrent ADK agents per room, all using the ADK Bidi Streaming Toolkit:
| Agent | Model | Purpose |
|---|---|---|
| 🔵 EN→HI Translation | gemini-live-2.5-flash-native-audio |
Real-time English→Hindi audio translation with vision |
| 🔵 HI→EN Translation | gemini-live-2.5-flash-native-audio |
Real-time Hindi→English audio translation with vision |
| 🟢 Cooking Assistant | gemini-2.0-flash-live-001 |
Video analysis → cultural cooking insights (text output) |
Why two separate translation agents instead of one? Independent pipelines prevent crosstalk. Each agent has its own system prompt, voice configuration, and language model parameters. When Mom speaks Hindi, only the HI→EN agent processes it — the EN→HI agent stays quiet.
Why a separate cooking assistant model? The translation agents use
gemini-live-2.5-flash-native-audio which is optimized for audio-in/audio-out. But the
cooking assistant needs to output text (for insight badges on the UI), so it uses
gemini-2.0-flash-live-001 — a Live API model that supports text output.
The Bidi Streaming Pipeline
Each WebSocket connection creates an ADK LiveRequestQueue + Runner.run_live()
pipeline:
async for message in websocket:
audio_data = base64.b64decode(message["audio"])
live_queue.send_realtime(audio_data)
# Downstream: Gemini → Browser
async for event in runner.run_live(config):
if event.content:
websocket.send_json({
"type": "audio",
"audio": base64_encode(audio)
})
Google Cloud Architecture
| Google Cloud Service | How It's Used |
|---|---|
| ☁️ Cloud Run | Hosts the FastAPI + WebSocket server with session affinity and 1-hour timeout for long cooking sessions |
| 🧠 Vertex AI | Serves Gemini Live models (gemini-live-2.5-flash-native-audio +
gemini-2.0-flash-live-001) in us-central1 |
| 📦 Cloud Build | Builds Docker containers from source for Cloud Run deployment |
| 🔑 IAM | Service account authentication for Vertex AI API access |
Deployment: One Command
gcloud run deploy babelchef-live-translator \
--source . \
--region us-central1 \
--allow-unauthenticated \
--session-affinity \
--timeout 3600
What I Learned Building This
-
🔊
Native audio models can't output text
gemini-live-2.5-flash-native-audiois optimized for audio-only I/O. When I tried using it for the cooking assistant (which needs text output for badges), it crashed with "Text output is not supported for native audio output model." The fix: usegemini-2.0-flash-live-001for text-output agents. -
🍳
Proactive Audio is essential in kitchens
Without it, sizzling oil and running water constantly trigger phantom translations. The other person hears nonsensical translated "speech" every time you wash a pot. Proactive Audio completely solves this by detecting intentional speech vs. ambient noise.
-
⏱️
Session Resumption is non-negotiable for real cooking
A typical biryani takes 45–60 minutes. Without Session Resumption, connections drop after ~10 minutes and context is lost. With it, the translation stays coherent through the entire cooking session.
-
💬
Affective Dialog makes translation feel human
Cooking is emotional — excitement when a dish comes together, urgency when something is burning, tenderness when sharing a family secret recipe. Flat, monotone translation kills the experience. Affective Dialog preserves the human connection.
-
📐
Vertex AI model names ≠ AI Studio model names
When
GOOGLE_GENAI_USE_VERTEXAI=TRUEis set, the model names are different.gemini-2.5-flashbecomesgemini-live-2.5-flash-native-audio. This caused runtime errors until I mapped the correct identifiers.
Try BabelChef / View the Code
BabelChef is open source and live on Google Cloud Run.
The Bigger Picture
BabelChef started as a personal problem — my Mom speaks Hindi, my wife speaks English, and I'm tired of being the human translator during cooking sessions. But the implications are much bigger.
43% of the world's population speaks more than one language at home. Cross-cultural cooking is just the beginning. The same architecture — real-time, vision-aware, emotionally-intelligent translation — can power:
- Multilingual medical consultations where tone matters
- Cross-cultural craft workshops (pottery, textile weaving)
- International cooking classes without language prerequisites
- Family reunions where grandkids can talk to grandparents
The Gemini Live API doesn't just translate words. It preserves intent, emotion, and context. That's what makes this a Live Agent, not just a translation tool.