My goal is to audio stream over WebSocket. Right now, I am trying to compress the audio data using AAC encoder. But I am having trouble decoding it.
My expected result is:
The PCM data recorded with AudioRecord class can be encoded, the encoded audio can be decoded back and played with AudioTrack class.
My actual result is:
Sometimes the AACDecoderPCM.java fails to decode, causing the entire app to crash.
How it works:
- Invoke a ReactMethod to start recording, using AudioRecord class.
- Encode the audio data using PCMEncoderAAC.java
- Emit the data to the JS bridge.
- On the JS side, add an event listener with callback function
(event) => send += ` {event}`
- Invoke a ReactMethod to play the audio, using AudioTrack class.
@ReactMethod public void streamAndPlayAsync(String data) { String[] chunks = data.split(" "); for (String chunk : chunks) { byte[] audioData = base64ToBytes(chunk); aacDecoderPCM.decode(audioData); } }
Possible cause:
- I believe this is caused because of for 1 raw PCM data will generate 4 encoded AAC data. The issue could be because of this line of code.
AudioPlayerModule.java line 72.
@ReactMethod public void streamAndPlayAsync(String data) { String[] chunks = data.split(" "); for (String chunk : chunks) { byte[] audioData = base64ToBytes(chunk); aacDecoderPCM.decode(audioData); } }
PCMEncoderAAC.java line 55
while (outputBufferIndex >= 0) { ByteBuffer outputBuffer = outputBuffers[outputBufferIndex]; outputBuffer.position(bufferInfo.offset); outputBuffer.limit(bufferInfo.offset + bufferInfo.size); byte[] encodedData = new byte[bufferInfo.size]; outputBuffer.get(encodedData); String stringData = Base64.encodeToString(encodedData, Base64.NO_WRAP); mReactContext .getJSModule(DeviceEventManagerModule.RCTDeviceEventEmitter.class) .emit("read", stringData); mediaCodec.releaseOutputBuffer(outputBufferIndex, false); outputBufferIndex = mediaCodec.dequeueOutputBuffer(bufferInfo, 0); }
This is the Encoder and Decoder full source code
PCMEncoderAAC.java
package com.satpam.AudioRecorder;import android.media.MediaCodec;import android.media.MediaCodecInfo;import android.media.MediaFormat;import android.util.Base64;import com.facebook.react.bridge.ReactApplicationContext;import com.facebook.react.modules.core.DeviceEventManagerModule;import java.io.IOException;import java.nio.ByteBuffer;public class PCMEncoderAAC { private ReactApplicationContext mReactContext; private MediaCodec mediaCodec; private ByteBuffer[] inputBuffers; private MediaCodec.BufferInfo bufferInfo; private ByteBuffer[] outputBuffers; public PCMEncoderAAC(ReactApplicationContext reactContext) { mReactContext = reactContext; setEncoder(); } private void setEncoder() { MediaFormat format = MediaFormat.createAudioFormat(MediaFormat.MIMETYPE_AUDIO_AAC, 8000, 1); format.setInteger(MediaFormat.KEY_BIT_RATE, 9600); format.setInteger(MediaFormat.KEY_AAC_PROFILE, MediaCodecInfo.CodecProfileLevel.AACObjectLC); try { mediaCodec = MediaCodec.createEncoderByType(MediaFormat.MIMETYPE_AUDIO_AAC); } catch (IOException e) { e.printStackTrace(); } mediaCodec.configure(format, null, null, MediaCodec.CONFIGURE_FLAG_ENCODE); mediaCodec.start(); inputBuffers = mediaCodec.getInputBuffers(); outputBuffers = mediaCodec.getOutputBuffers(); bufferInfo = new MediaCodec.BufferInfo(); } public void encode(byte[] data) { int inputBufferIndex = mediaCodec.dequeueInputBuffer(-1); if (inputBufferIndex >= 0) { ByteBuffer inputBuffer = inputBuffers[inputBufferIndex]; inputBuffer.clear(); inputBuffer.put(data); inputBuffer.limit(data.length); mediaCodec.queueInputBuffer(inputBufferIndex, 0, data.length, 0, 0); } int outputBufferIndex = mediaCodec.dequeueOutputBuffer(bufferInfo, 0); while (outputBufferIndex >= 0) { ByteBuffer outputBuffer = outputBuffers[outputBufferIndex]; outputBuffer.position(bufferInfo.offset); outputBuffer.limit(bufferInfo.offset + bufferInfo.size); byte[] encodedData = new byte[bufferInfo.size]; outputBuffer.get(encodedData); String stringData = Base64.encodeToString(encodedData, Base64.NO_WRAP); mReactContext .getJSModule(DeviceEventManagerModule.RCTDeviceEventEmitter.class) .emit("read", stringData); mediaCodec.releaseOutputBuffer(outputBufferIndex, false); outputBufferIndex = mediaCodec.dequeueOutputBuffer(bufferInfo, 0); } }}
AACDecoderPCM.java
package com.satpam.AudioPlayer;import android.media.AudioFormat;import android.media.AudioManager;import android.media.AudioTrack;import android.media.MediaCodec;import android.media.MediaCodecInfo;import android.media.MediaFormat;import android.util.Log;import java.io.IOException;import java.nio.ByteBuffer;public class AACDecoderPCM { private AudioTrack audioTrack; private MediaCodec mediaCodec; private ByteBuffer[] inputBuffers; private MediaCodec.BufferInfo bufferInfo; private ByteBuffer[] outputBuffers; public AACDecoderPCM() { audioTrack = new AudioTrack(AudioManager.STREAM_MUSIC, 8000, AudioFormat.CHANNEL_CONFIGURATION_MONO, AudioFormat.ENCODING_PCM_16BIT, 4096, AudioTrack.MODE_STREAM); audioTrack.play(); setDecoder(); } private void setDecoder() { MediaFormat format = MediaFormat.createAudioFormat(MediaFormat.MIMETYPE_AUDIO_AAC, 8000, 1); format.setInteger(MediaFormat.KEY_BIT_RATE, 9600); format.setInteger(MediaFormat.KEY_AAC_PROFILE, MediaCodecInfo.CodecProfileLevel.AACObjectLC); int profileLevel = MediaCodecInfo.CodecProfileLevel.AACObjectLC; // AAC LC int sampleRateInId = 11; // 8KHz int channelConfig = 1; // Mono ByteBuffer csd = ByteBuffer.allocate(2); csd.put(0, (byte) (profileLevel << 3 | sampleRateInId >> 1)); csd.put(1, (byte)((sampleRateInId & 0x01) << 7 | channelConfig << 3)); format.setByteBuffer("csd-0", csd); try { mediaCodec = MediaCodec.createDecoderByType(MediaFormat.MIMETYPE_AUDIO_AAC); } catch (IOException e) { e.printStackTrace(); } mediaCodec.configure(format, null, null, 0); mediaCodec.start(); inputBuffers = mediaCodec.getInputBuffers(); outputBuffers = mediaCodec.getOutputBuffers(); bufferInfo = new MediaCodec.BufferInfo(); } public void decode(byte[] data) { Log.d(getClass().getName(), "data length " + data.length); int inputBufferIndex = mediaCodec.dequeueInputBuffer(-1); if (inputBufferIndex >= 0) { Log.d(getClass().getName(), "triggered"); ByteBuffer inputBuffer = inputBuffers[inputBufferIndex]; inputBuffer.clear(); inputBuffer.put(data); mediaCodec.queueInputBuffer(inputBufferIndex, 0, data.length, 0, 0); } bufferInfo = new MediaCodec.BufferInfo(); int outputBufferIndex = mediaCodec.dequeueOutputBuffer(bufferInfo, 0); Log.d(getClass().getName(), "outputBufferIndex " + outputBufferIndex); while (outputBufferIndex >= 0) { Log.d(getClass().getName(), "triggered 2"); ByteBuffer outputBuffer = outputBuffers[outputBufferIndex]; outputBuffer.position(bufferInfo.offset); outputBuffer.limit(bufferInfo.offset + bufferInfo.size); byte[] decodedData = new byte[bufferInfo.size]; outputBuffer.get(decodedData); Log.d(getClass().getName(), String.valueOf(decodedData.length)); audioTrack.write(decodedData, 0, decodedData.length); mediaCodec.releaseOutputBuffer(outputBufferIndex, false); outputBufferIndex = mediaCodec.dequeueOutputBuffer(bufferInfo, 0); } }}
TL:DR
I am using these as references:1. [PCM -> AAC -> PCM, Android only](https://stackoverflow.com/questions/21804390/pcm-aac-encoder-pcmdecoder-in-real-time-with-correct-optimization)2. [PCM -> AAC, Android only](https://www.programmersought.com/article/78314556957/)3. [AAC -> PCM, solve error W/SoftAAC2: AAC decoder returned error 4098, substituting silence](https://stackoverflow.com/a/36278858/13285583)More references, I have not understand how ADTS works. I will be implementing these within 6 hours:1. [ADTS](https://stackoverflow.com/a/17357008/13285583)2. [AAC -> PCM, Android only, with switch case](https://stackoverflow.com/questions/56106877/how-to-decode-aac-formatmp4-audio-file-to-pcm-format-in-android)