Skip to content

Add audio chat API support (gpt-4o-audio-preview, gpt-audio-1.5) - #397

Open
BugorBN wants to merge 3 commits into
MacPaw:mainfrom
BugorBN:feature/audio-chat-api
Open

BugorBN wants to merge 3 commits into
MacPaw:mainfrom
BugorBN:feature/audio-chat-api

Conversation

@BugorBN

@BugorBN BugorBN commented Nov 28, 2025

Copy link
Copy Markdown

What

Adds audio support through the existing ChatQuery surface rather than a parallel API:

  • gpt-audio-1.5, gpt-audio and gpt-audio-mini model constants
  • gpt-4o-audio-preview marked deprecated (shut down 7 May 2026) and dropped from allModels(satisfying:), as other deprecated constants are
  • a README section, "Audio input and output with Chat Completions", built on ChatQuery.modalities and ChatQuery.audioOptions — audio in and out, text in with audio out, streaming, multi-turn via the audio id, and audio token usage
  • coding tests for input_audio content parts, assistant messages referencing a previous audio response, and decoding message.audio and the streaming delta.audio

Why

The first version of this PR added AudioChatQuery, AudioChatResult, AudioChatStreamResult, audioChats/audioChatsStream and an AudioConversationManager. Per review, that was a second way to call /chat/completions with a smaller surface (6 voices against 11, fewer parameters), added four protocol requirements that the API-breakage gate flags, and put conversation state management inside the SDK. All of it is removed, so the library keeps one way to make a chat completion.

What remains is what the library was actually missing: the current audio model constants, and documentation for audio over Chat Completions, which the README did not cover at all.

Affected Areas

  • Sources/OpenAI/Public/Models/Models/Models.swift — three new constants, one deprecation
  • README.md — new section under Audio
  • Tests/OpenAITests/ — audio cases added to the existing ChatQuery, ChatResult and ChatStreamResult test files

No API breakage: swift package diagnose-api-breaking-changes against 0.5.1 reports none, so no allowlist entry is needed. The branch is rebased on main and all 218 tests pass.

@BugorBN
BugorBN force-pushed the feature/audio-chat-api branch from 7d5804b to 7d385dd Compare November 29, 2025 09:47
@BugorBN BugorBN changed the title Add audio chat API support for gpt-4o-audio-preview model Add audio chat API support (gpt-4o-audio-preview, gpt-audio-1.5) Apr 3, 2026

@Krivoblotsky Krivoblotsky left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for the work here, and apologies for the wait. It merges cleanly with main, builds, and all 217 tests pass. I want to be upfront about the direction, because it affects most of the diff.

ChatQuery already covers audio chat. On main today: ChatQuery.modalities, ChatQuery.audioOptions (11 voices including ash, ballad, coral, sage, verse; formats mp3, opus, flac, wav, pcm16), ContentPartAudioParam for input_audio, ChatResult.Choice.Message.audio (id, expiresAt, data, transcript) and the same on the streaming delta. AudioChatQuery, AudioChatResult and AudioChatStreamResult are a second way to call /chat/completions with a smaller surface (6 voices, fewer parameters), and audioChats / audioChatsStream add four protocol requirements, which the new API-breakage gate flags. The library should have one way to do a chat completion.

AudioConversationManager is conversation state management, which belongs in the app (or the Demo), not in the SDK.

What we would like to take from this PR:

  • the gpt-audio-1.5 constant (plus gpt-audio and gpt-audio-mini), with gpt-4o-audio-preview marked deprecated since it was shut down on 7 May 2026;
  • a README section "Audio input and output with Chat Completions" built on ChatQuery, which the README lacks entirely today; your examples adapt almost one to one;
  • the coding tests, adapted to ChatQuery where they add coverage we do not have.

If you are up for reshaping the PR along those lines, that would be great and it would land quickly. If not, we can open a smaller PR that carries those pieces with you credited as co-author. Either way, thank you for pushing on audio support.

@BugorBN

BugorBN commented Sep 14, 2026

Copy link
Copy Markdown
Author

Thank you for the work here, and apologies for the wait. It merges cleanly with main, builds, and all 217 tests pass. I want to be upfront about the direction, because it affects most of the diff.

ChatQuery already covers audio chat. On main today: ChatQuery.modalities, ChatQuery.audioOptions (11 voices including ash, ballad, coral, sage, verse; formats mp3, opus, flac, wav, pcm16), ContentPartAudioParam for input_audio, ChatResult.Choice.Message.audio (id, expiresAt, data, transcript) and the same on the streaming delta. AudioChatQuery, AudioChatResult and AudioChatStreamResult are a second way to call /chat/completions with a smaller surface (6 voices, fewer parameters), and audioChats / audioChatsStream add four protocol requirements, which the new API-breakage gate flags. The library should have one way to do a chat completion.

AudioConversationManager is conversation state management, which belongs in the app (or the Demo), not in the SDK.

What we would like to take from this PR:

  • the gpt-audio-1.5 constant (plus gpt-audio and gpt-audio-mini), with gpt-4o-audio-preview marked deprecated since it was shut down on 7 May 2026;
  • a README section "Audio input and output with Chat Completions" built on ChatQuery, which the README lacks entirely today; your examples adapt almost one to one;
  • the coding tests, adapted to ChatQuery where they add coverage we do not have.

If you are up for reshaping the PR along those lines, that would be great and it would land quickly. If not, we can open a smaller PR that carries those pieces with you credited as co-author. Either way, thank you for pushing on audio support.

Hey, thanks for the feedback, I'll reshape it in the next couple days

BugorBN and others added 2 commits September 16, 2026 19:42
- Add AudioChatQuery, AudioChatResult, AudioChatStreamResult models
- Implement audioChats() and audioChatsStream() methods
- Add type-safe enums: AudioFormat, Voice, Modality
- Add AudioConversationManager for multi-turn conversations
- Add support for gpt-4o-realtime-preview and gpt-4o-mini-realtime-preview models
- Add dated variants: 2024-12-17 snapshots
- Comprehensive test coverage (38 tests across 4 test suites)
- Update README with audio chat documentation and examples
- Format requirements: wav/mp3 for input, pcm16 recommended for streaming output
- Relaxed parsing support for handling missing fields
@BugorBN
BugorBN force-pushed the feature/audio-chat-api branch from 58b84a0 to a090543 Compare September 16, 2026 19:14
BugorBN added a commit to BugorBN/OpenAI that referenced this pull request Sep 16, 2026
Following review feedback on MacPaw#397, drop the parallel audio chat API and
build on the ChatQuery surface the library already has.

Removed: AudioChatQuery, AudioChatResult, AudioChatStreamResult, the
audioChats/audioChatsStream protocol requirements, and
AudioConversationManager (conversation state belongs in the app).

Kept, per review:
- gpt-audio-1.5, gpt-audio and gpt-audio-mini constants, with
  gpt-4o-audio-preview deprecated since it was shut down on 7 May 2026
  and dropped from allModels(satisfying:), as other deprecated
  constants are
- a README section, "Audio input and output with Chat Completions",
  built on ChatQuery modalities and audioOptions, covering audio in and
  out, text in and audio out, streaming, multi-turn via the audio id,
  and audio token usage
- coding tests adapted to ChatQuery, covering input_audio content
  parts, assistant messages referencing a previous audio response, and
  decoding message and delta audio, none of which was covered before

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Following review feedback on MacPaw#397, drop the parallel audio chat API and
build on the ChatQuery surface the library already has.

Removed: AudioChatQuery, AudioChatResult, AudioChatStreamResult, the
audioChats/audioChatsStream protocol requirements, and
AudioConversationManager (conversation state belongs in the app).

Kept, per review:
- gpt-audio-1.5, gpt-audio and gpt-audio-mini constants, with
  gpt-4o-audio-preview deprecated since it was shut down on 7 May 2026
  and dropped from allModels(satisfying:), as other deprecated
  constants are
- a README section, "Audio input and output with Chat Completions",
  built on ChatQuery modalities and audioOptions, covering audio in and
  out, text in and audio out, streaming, multi-turn via the audio id,
  and audio token usage
- coding tests adapted to ChatQuery, covering input_audio content
  parts, assistant messages referencing a previous audio response, and
  decoding message and delta audio, none of which was covered before
@BugorBN
BugorBN force-pushed the feature/audio-chat-api branch from a090543 to 76af3ea Compare September 16, 2026 19:15
@BugorBN

BugorBN commented Sep 16, 2026

Copy link
Copy Markdown
Author

@Krivoblotsky done, let me know if I can help with anything else

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants