Android Audio: Problems, Hidden Limitations and OpenSL ES

Thomas Edison and one of his early phonographs

Lately I’ve been digging into Android audio APIs. Earlier I wrote an introductory article that describes the three available APIs for WiseAndroid. Now this article assumes you are familiar with AudioTrack, SoundPool and MediaPlayer at the basic level.

What I want to present in this post is my experience with the existing audio APIs on Android, including the issues and problems I personally faced. I will also shortly cover OpenSL ES, the standard that is expected to be supported in one of the upcoming Android releases.

Known problems with existing APIs

All three of the existing audio APIs (SoundPool, AudioTrack and MediaPlayer) have bugs (behavior inconsistent with documentation), hidden (undocumented) limitations and design issues. Let’s review some of them per each API.

AudioTrack

AudioTrack is marketed as the most flexible and advanced audio API, allowing you to push raw audio data to the hardware. While it does work to some extent, and might allow you to solve some use cases, it still fails in some very basic scenarios.

I faced both of the following issues personally before I found that they had already been submitted to the Android bug database by Piotr Buła. Unfortunately, neither of them are resolved as of today:

Issue 3197: Audio breaks after fast, repeated attempts to replay sound using AudioTrack (in static mode)

This bug actually prevents developes from using AudioTrack for software synths and games (where you could benefit from its low latency and ability to play long audio samples, which is not possible with SoundPool).

From what I’ve seen, it seems to be a miserable race condition. If you add a delay (SystemClock.sleep() for 30-40 ms on Droid), this issue goes away. Of course, it’s a pretty wretched solution which I can’t recommend. So consider this use case broken.

Issue 3198: AudioTrack playback state does not reflect reality

Now this is arguably less of an issue, but the problem seems to be that when the sample is finished playing the status is not updated correctly in AudioTrack code.

As you might guess from the serious impact of those two issues, AudioTrack is generally not very stable or well implemented. So if you’re planning to use it in your app, do test your use cases before you go. And read on to learn limitations and problems of other APIs that you might consider as alternative.

SoundPool

SoundPool is the foundation of short, low-latency audio on Android. It is widely used in games and other apps that need sound effects. However, it is rather limited in its functionality, and poorly documented in terms of limitations and use cases it supports.

Although Piotr Buła reported an issue on SoundPool too, it’s not as serious (it’s related to release() method and seems to be not reproducible for me). I faced other rough edges though:

  • Inability to detect whether samples are loaded (fixed in 2.2 but what about older versions, wasn’t it an obvious piece of functionality?)
  • Hidden undocumented limit of sample size to 1 Mb (seems to boil down to the line
    static size_t kDefaultHeapSize = 1024 * 1024; // 1MB
    

    …in MediaPlayerService.cpp)

  • Sudden periods of high latency (up to 1 second) with corresponding reports in the log from the platform such as write delayed for XXX ms
  • Lack of support for some audio formats such as floating-point PCM WAVs. However, this might be documented somewhere and I might’ve just missed that.

As you can see, these issues together might create enough pain for developers who expect SoundPool to do what it promises. Combined with the AudioTrack issues described above, non-trivial low-latency audio on Android becomes a real headache, and this is an area in which the platform is quite backwards compared to iPhone. We have to admit that and work on this issue as a community. This is why I ask you to do the simplest thing you can – VOTE for each of those issues if they affect your current or future apps (the vote star is located at the bottom of each bug page).

MediaPlayer

MediaPlayer is probably the most widely used Audio APIs on Android, and it seems to be the best implemented too. You can hear various complaints in the dev groups, but the worst things I brushed into with MediaPlayer were random log messages and poor problem handling.

Another thing many people want is an option to play from an InputStream. This might be tough to implement given that most of MediaPlayer code is native, but has to be possible.

Native audio API & OpenSL ES

Support for fast native graphics, both 2D and 3D, has been improving with each Android version and seems to have reached the level required for very sophisticated apps. The situation is very different with audio, which is quite sad. After all, audio software has been a large industry on desktop computers for a while, and mobile features such as portability might be very appealing for audio software authors.

However, just about any audio software needs to do its job with high performance, good quality and low latency. All of that kind of suggests that you need native code (and decent APIs). However, as of Froyo, we don’t have that option.

What has been heard from the platform team is that one of the future versions might support OpenSL ES, a not very well known standard that is designed by the same organization as OpenGL and has a similar technical approach.

Having read the specification, I have the following comments:

  • Some of the features OpenSL ES includes are not that demanded, such as MIDI support
  • It’s not clear which profile Android is going to support, and if you ask me, none of them fit
  • The feature set is mostly covered by other existing APIs – yes, with issues, but why not fix the existing APIs instead of introducing a new one
  • No performance, latency etc. levels seem to be imposed by the spec – everything depends on the implementation
  • If platform devs use the existing native code to implement the new API, we will have the same set of problems – just under a different cover

Thus, as you might notice, I’m highly uncertain whether OpenSL ES would fix the issues we are facing today. I think effort should better be spent on fixing the existing APIs. Once they’re good enough, we can implement OpenSL ES on top of them – if anyone actually needs it. For now, I’d just love an AudioTrack-like API in native code, better without the issues that AudioTrack has.

Tags: , , , , , , ,

2 Responses to “Android Audio: Problems, Hidden Limitations and OpenSL ES”

  1. Best phone ever it is way better then an iPhone 4s if you’ve thought between the 2 pick this one

  2. It’s amazing that Android audio issues 3197, 3198, 2563 and 3434 have been ignored by Google for nearly 3 years! Their status is still “new”, no one is assigned to them, and nobody at Google has commented on them. This is inexcusable and it’s making me angry… this is my first time using AudioPlayer and I just don’t know how I should use it.

    I notice that the following code doesn’t produce any output whatsoever on my Samsung Galaxy Tab:

    short[] buf = new short[2048]; // 92.9 ms of audio = minimum buffer size
    for (int i = 0; i < buf.length; i++)
    buf[i] = (short)((short)(i <> 4);

    AudioTrack track = new AudioTrack(AudioManager.STREAM_MUSIC, 22050,
    AudioFormat.CHANNEL_CONFIGURATION_MONO, AudioFormat.ENCODING_PCM_16BIT,
    44100 /* one second buffer */, AudioTrack.MODE_STREAM);
    track.play();
    for (int i = 0; i != 10; i++)
    track.write(buf, 0, buf.length);

    If I use i != 20 instead of i != 10 then I get about 1 second (10 buffers worth) of output. What’s the deal? I can’t tell if this is just the Galaxy Tab or a general Android bug, since the emulator doesn’t seem capable of playing audio. Does AudioPlayer simply ignore all input until its buffer is full? Should I output a bunch of 0s to start to ensure that my audio actually comes out? But once the buffer is full, AudioTrack.write() starts blocking. Are we expected to always use AudioTrack on a separate thread?

    P.S. Google is warning me in no uncertain terms that this site is infected with some drive-by-download thing. Hopefully I am protected by Javascript being off.

Leave a Reply

XHTML: You can use these tags: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>