Do not forward the audio codec config buffer as an encoded frame - #2202
Open
RomanHerbstmann wants to merge 1 commit into
Open
RomanHerbstmann wants to merge 1 commit into
RomanHerbstmann wants to merge 1 commit into
Conversation
MediaCodec emits the audio codec config (for AAC the AudioSpecificConfig) as its first output buffer, flagged with BUFFER_FLAG_CODEC_CONFIG. AudioEncoder.checkBuffer only validated the timestamp, so that buffer was forwarded like a regular frame: senders transmitted it as a tiny bogus AAC frame and AndroidMuxerRecordController wrote it into the recording as a sample. The resulting MP4 files start with a 2-byte audio sample that is identical to the track extradata. FFmpeg and VLC skip it, but Chrome aborts playback of the whole file with PIPELINE_ERROR_DECODE. Dropping the buffer is safe because the configuration is derived independently on every path: RTMP builds the AudioSpecificConfig in AacPacket.sendAudioInfo from sample rate and channel count, RTSP signals it through the SDP body, and the muxer takes it from the MediaFormat passed to setAudioFormat, which already carries csd-0. This mirrors how VideoEncoder keeps SPS/PPS out of the frame path.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
MediaCodecemits the audio codec configuration as its first output buffer, flagged withBUFFER_FLAG_CODEC_CONFIG. For AAC that buffer is the 2-byte AudioSpecificConfig.BaseEncoder.processOutputforwards every buffer thatcheckBufferaccepts:VideoEncoder.checkBufferkeeps SPS/PPS out of that path, butAudioEncoder.checkBufferonly validates the timestamp, so the config buffer is forwarded like a regular encoded frame. It then reachesrecordAudio→AndroidMuxerRecordController.write→MediaMuxer.writeSampleData, which stores it as a media sample.Impact
Recorded MP4 files start with a 2-byte audio sample that is identical to the track's
extradata. FFmpeg and VLC skip it withInput buffer exhausted before END element found, but Chrome aborts playback of the entire file:Verified on a 1.4 GB, 66-minute recording produced from an RTMP stream: the first audio packet is
12 10(AAC-LC, 44.1 kHz, stereo), byte for byte the same as the track'sextradata. Remuxing the same file with that single packet dropped makes it play in Chrome; nothing else changes.Fix
Reject buffers carrying
BUFFER_FLAG_CODEC_CONFIGinAudioEncoder.checkBuffer, mirroring how video keeps SPS/PPS out of the frame path.This is safe because the configuration is derived independently on every path:
AacPacket.sendAudioInfo(sampleRate, isStereo, codec)buildsAacAudioSpecificConfigfrom sample rate and channel count.SdpBodysignals it from sample rate and channel count.MediaFormatpassed tosetAudioFormatinonFormatChanged, which already carriescsd-0.The change covers every
MediaCodec-based audio codec (AAC, HE-AAC, Opus). G711 uses the software path inBaseEncoder.processG711()and has no codec-config buffer, so it is unaffected. The filter is deliberately kept inAudioEncoderrather thanBaseEncoder, because video still relies on the config buffer for manual SPS/PPS extraction whenformatChangedarrives late.Tests
Added
encoder/src/test/java/com/pedro/encoder/AudioEncoderCheckBufferTest.ktwith two cases: a buffer flagged as codec config is rejected, a normal frame with a valid timestamp is accepted../gradlew :encoder:testDebugUnitTestpasses 24/24.Note for anyone writing further tests in this module: it sets
unitTests.isReturnDefaultValues = true, soMediaCodec.BufferInfo.set(...)is a no-op stub and the fields have to be assigned directly.