Skip to content

feat(mllm): add ThunderPhone speech-to-speech extension - #2316

Open
kolchinski wants to merge 1 commit into
TEN-framework:mainfrom
kolchinski:add-thunderphone-mllm
Open

kolchinski wants to merge 1 commit into
TEN-framework:mainfrom
kolchinski:add-thunderphone-mllm

Conversation

@kolchinski

@kolchinski kolchinski commented Sep 10, 2026

Copy link
Copy Markdown

Summary

Adds thunderphone_mllm_python: ThunderPhone voice agents as a speech-to-speech (MLLM) extension, alongside openai_mllm_python, gemini_mllm_python, azure_mllm_python, glm_mllm_python and stepfun_mllm_python.

ThunderPhone's Realtime WebSocket API speaks the OpenAI Realtime protocol, so the extension mirrors openai_mllm_python (same mllm-interface.json, same event mapping) and adds the ThunderPhone-specific parts:

  • the endpoint, its query parameters (saved agent, engine, language, caller/callee numbers, wire audio, call_events) and sk_live_ keys;
  • saved agents (agent_id): prompt, voice, engine, languages, tools and greeting live on ThunderPhone, the graph only moves audio;
  • inline sessions (prompt, product, voice, language): graph tools are collected and sent in the single session.update that starts the call, because ThunderPhone freezes instructions, tools and voice at that point (the extension waits up to 500 ms for late tool_registers unless audio arrives first);
  • ThunderPhone's call.* events (hang-up reason, transfer, keypad) are parsed and logged, and call.ended stops the extension from reconnecting, since a reconnect would start a new, separately billed call.

Also registers the extension in examples/voice-assistant-realtime (manifest dependency + README provider list) and adds THUNDERPHONE_API_KEY / THUNDERPHONE_AGENT_ID to .env.example.

Validation

  • tests/test_realtime.py (no credentials): 6 passed, covering the connect query, config validation, the inline session.update, and parsing of ThunderPhone's session.created, GA conversation.item.added and call.* events.
  • tests/test_extension.py: a credential-free load test through AsyncExtensionTester, plus a live session-ready test when THUNDERPHONE_API_KEY is set.
  • Verified live against the ThunderPhone API by driving ThunderPhoneRealtimeExtension directly (base class stubbed to record emits): saved-agent greeting → caller transcript → reply → hang-up (call.ended, no reconnect); inline session with a graph tool → mllm_server_function_callfunction_call_output → reply using the tool result → hang-up.

Type of Change

  • Bug fix
  • New feature
  • Documentation update
  • Performance improvement
  • Code refactoring

Testing

  • Tests added/updated
  • All repository tests pass. The full repository suite was not run.
  • Manual testing completed against the live ThunderPhone API (see Validation)

tests/test_realtime.py needs no TEN runtime and passes locally. tests/test_extension.py runs through AsyncExtensionTester, which needs the ten_agent_build dev container; I haven't run it there yet and can if you'd like. No upstream CI result is claimed.

Documentation

  • Documentation updated
  • Examples provided if needed

The extension README covers configuration, saved agents vs. inline sessions, graph wiring and test commands. examples/voice-assistant-realtime lists the provider, and .env.example has the two ThunderPhone variables.

Breaking Changes

None. The extension is opt-in and doesn't change existing MLLM extensions or graphs.

Disclosure: I'm the founder of ThunderPhone.

ThunderPhone's Realtime WebSocket API speaks the OpenAI Realtime protocol, so
thunderphone_mllm_python mirrors openai_mllm_python and adds the ThunderPhone
parts: endpoint and query parameters (saved agent, engine, language, phone
numbers, wire audio, call events), sk_live_ keys, saved agents that bring
their own prompt/voice/tools, one complete session.update for inline sessions
(ThunderPhone freezes instructions and tools when the call starts), call.*
events, and no reconnect after the platform ends the call.
@kolchinski

Copy link
Copy Markdown
Author

Checking in on this. The branch is still mergeable and the only red check is the repo's own claude-review job, which needs a maintainer-side secret rather than a change here. Happy to rebase or rework anything if there's feedback.

@kolchinski

Copy link
Copy Markdown
Author

@wangyoucao577 @diyuyi-agora, could one of you take a look when you have a moment? You've handled the recent extension PRs (#2276, #2320), so you seemed like the right people. The workflows haven't run here yet since this is my first PR to the repo, so it also needs an approval to run CI. I've added the Type of Change and Testing sections to match #2320, and I can run tests/test_extension.py in the ten_agent_build container before review if that helps.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant