Guys & Girls, all MTR readers – as you might have noticed, I haven’t been writing for MTR for a while already. So at this point, I would like to announce that I will no longer be able to write new articles for this blog any more.

MTR has been a successful project for me. I exchanged a lot of knowledge, found many friends all over the world and started my own cool Android project with a guy who I met here in this blog.

I am moving to a new exciting position in my company where I will need to keep focused on the higher level picture of IT trends, which includes the mobile world but is not limited to it. I am also highly devoted to my app, Su-Preme MPA, which is supposed to see its first full version released very soon. Thus, to avoid spreading myself too thin, I decided to cease my work on Mind The Robot.

However, this by no means should be the end of MTR itself!

First of all, I will keep the blog hosted and up and will try to answer your comments as often as I can.

Second, if you feel like writing an Android article for MTR, I will be happy to give you an account so you can post it. The blog currently has about 700-800 visits per day so your article will definitely be seen. Just contact me at ivan@mindtherobot.com.

Again, thanks for all your feedback, your comments and everything.

Sincerely yours,
Ivan @ MTR

Check out the first demo vid of Su-Preme MPA – my first Android app to be released on the market in few days!

MPA is a sample based music production app that mimics the look and feel of a classic analog device that is widely popular among urban music producers.

I’m proud to say that MPA is not a toy. We really mean it to be used by actual music producers to make actual beats.

The app was co-designed and will be promoted by Alaric “Supreme” Wilder of Wu-Tang Clan fame. He also did the video!

Why scan the classpath? There are various patterns that are often used in enterprise Java applications that require scanning of classpath and getting the list of all classes that are present in the application.

For example, if you want to discover all classes with a certain annotation (such as @Component in Spring Framework) to process them in a special way, you need a way to go over all classes in your application and select some of them based on which annotations they have.

However, neither Java SE nor Android have built-in facilities to safely get the list of all classes in your application in runtime. The reason for that is the theory behind classloaders in Java – the ability to go over all classes is neither needed in classic OOP nor feasible for all theoretically possible classloader implementations. However, in practice, scanning classpath and discovering the classes you need is quite possible in most cases, both in your web app and on Android. This is always going to be more or less a hack, but if it has been useful in web applications, it can also be useful in Android apps – with some caution, of course.

Thus, in this article I will show and explain a piece of code that does exactly that – scans your classpath and gives you the ability to go over all classes in your app.

Action plan

So here’s how we could approach the problem of going over the set of application classes in runtime:

1. We assume that Android PathClassLoader (see source here) is used for classloading in your app.
2. As you can see in the source above, the list of DEX files (typically one but theoretically more) that contain your app classes, is kept in the private field called mDexs. We can use invasive reflection to get the field value and thus the list of DEXs.
3. Once we get the list of DEXs, we can use the API of the DexFile class to get the list of its entries (which are class names) and there you go.

As I said above, this plan is 90% hacking so we should take whatever assumptions we make with a grain of salt, and do as much error handling and checking as possible, if we actually choose to use this approach in a production app.

Now let’s write some code.

Code example

Say we have an annotation called Foo and few classes decorated with it:

@Retention(RetentionPolicy.RUNTIME)
public @interface Foo {
	String value();
}

@Foo("person")
public class Person {

}

@Foo("order")
public class Order {

}

Here’s a simple class that goes over all classes in your app in runtime, finds those marked with @Foo and get the value of the annotation. Do not copy and paste this code into your app – it has a lot to be fixed and added, as described below the code.

public class ClasspathScanner {
	private static final String TAG = ClasspathScanner.class.getSimpleName();

	private static Field dexField;
	
	static {
		try {
			dexField = PathClassLoader.class.getDeclaredField("mDexs");
			dexField.setAccessible(true);
		} catch (Exception e) {
			// TODO (1): handle this case gracefully - nobody promised that this field will always be there
			Log.e(TAG, "Failed to get mDexs field");
		} 
	}
	
	public void run() {
		try {
			// TODO (2): check here - in theory, the class loader is not required to be a PathClassLoader
			PathClassLoader classLoader = (PathClassLoader) Thread.currentThread().getContextClassLoader();

			DexFile[] dexs = (DexFile[]) dexField.get(classLoader);
			for (DexFile dex : dexs) {
				Enumeration<String> entries = dex.entries();
				while (entries.hasMoreElements()) {
					// (3) Each entry is a class name, like "foo.bar.MyClass"
					String entry = entries.nextElement();
					Log.d(TAG, "Entry: " + entry);

					// (4) Load the class
					Class<?> entryClass = dex.loadClass(entry, classLoader);
					if (entryClass != null) {
						Foo annotation = entryClass.getAnnotation(Foo.class);
						if (annotation != null) {
							Log.d(TAG, entry + ": " + annotation.value());
						}
					}
				}
			}
		} catch (Exception e) {
			// TODO (5): more precise error handling
			Log.e(TAG, "Error", e);
		}
	}
}

Let’s review what we have there.

• (1) A future version of Android might change the internal structure of PathClassLoader and the field might not be found. Production code should somehow deal with it.
• (2) A future version of the platform might use other classloader than PathClassLoader – another thing to be handled in real code.
• (3) To save some time, you might scan only some specific packages rather than everything.
• (4) A real good improvement would be to get class metadata without loading the class – something that javassist can do in Java SE. In this implementation, every class will actually be loaded which is slow.
• (5) And of course you should handle each possible exception type specifically.

As you can see, the solution can’t really be called reliable or future-proof. Use with caution.

My definition or, rather, my perception of an enterprise mobile app is generally the following:

• The app in question is a mobile client for a multi-user client-server application. It can either be the only kind of client for the app, or co-exist with a web interface or, for example, a PC desktop client for the same server app.
• It has a “serious” user interface – rather than trying to impress the end-user with a slick, non-typical UI, it aims to be as straightforward and predictable as possible. It tends to have various forms, charts, trees and lists of various complexity.
• Conceptually, very often a big part of the app is dedicated to managing (CRUDing) a set of entities such as users, documents, reports etc. Domain models can get quite complex, often with tree-like structures, fine-grained user permissions with different access levels and profiles.
• Internally the app uses typical client-server protocols like SOAP, XML-RPC or even CORBA. Often in-house protocols and frameworks are used on top of those generic ones or instead of them. You get the picture.

Enterprise apps might not seem to be the most interesting kind to work on. However, they are attractive for many studios because customers usually pay well for them. In addition, as a developer, you will find that they are often challenging to implement – at least with functional and internal quality that will satisfy both the customer and you.

In this article, I would like to cover a list of typical facilities that Android developers need when developing such kind of apps. The platform does not really have that much to offer (as we will see), and many developers build their own ad-hoc frameworks that they try to reuse as needs arise.

I will share some thoughts that I have about that, and will be happy to know your stories too.

UI Framework

One of the most important facilities you need in a typical enterprise mobile client app is a framework that will allow you to quickly put together a form that consists of a set of fields of various types represented by various UI controls. Have a look at the example on the right – it is Alessandro La Rosa’s example app to calculate your taxes if you live in Italy (click to see full-size).

It might seem that Android has enough built-in UI APIs not to make you think about adding any framework layer of your own. However, some things are missing:

• There is no dedicated layout class that allows to make quick and efficient (especially for various display sizes) form-like UIs where you have a set of controls and labels attached to them. You will probably either use LinearLayout or TableLayout, or build your own FormLayout on top of them.
• There is no data-binding framework. Remember I talked about MVC multiple times already. What I want to say here is that you’d like user’s input automatically bound to your POJO model object rather than kept inside a set of UI controls like TextViews and Spinners. Adapters seem to do a part of the job for lists as they work one-way (from data to UI) rather than two-way. In any case, what we usually want is a control-level solution, such as one that allows you to bind an underlying bean’s property name to a TextView with the ID name_view.
• There is no common way to detect and express validation errors. Almost any form lets the user enter invalid data and then rejects it and helps the user find and fix their error. There are two parts of this facility – an engine that allows you to specify your validation constraints and get the list of errors given an input, and a common way to present those errors in the UI. The platform does not have either of those, so you will have to decide how you do it. If you have created a data binding/MVC solution as described in the previous point, then it will typically take over validation as a phase in its UI-to-data process. When you want to display the errors you found, you can show a popup dialog, or insert a yellow bar at the top of the form, or show each error near the field that it is related to etc. You’ll have to do it manually for each form, or build a set of reusable classes that will help you.
• If you have really many forms in your app, you might consider building a form-generating framework (like those often found in RAD environments) that allows generating dynamic forms based on your POJO beans (that you will perhaps annotate with helper UI annotations such as @MediumText or @Spinner(min=1, max=100) etc.) with minimum work spent on UI.

In addition to the form problem, there are many views missing in the platform that are typically needed for enterprise apps, such as a sortable paginated data table, various charts etc. If you do build those components, make sure they are as generic as possible – there is real value in reusable views when you build your next app.

Server connection

The most obvious missing part in the Android core platform in terms of remote connections is SOAP. There is no SOAP client framework in the platform. This might in fact be a good thing as it should be pretty easy to port the most suitable among the Java SE libraries that do SOAP to Android. The HTTP client that is included in the platform is based on Apache Commons HTTP Client which is a respected and stable piece of code that will serve as a foundation for your SOAP framework.

If your SOAP/XML/JSON-based protocol is complicated, or if you plan well ahead, you will think of including a solution to bind your model beans (POJOs) to XML/JSON and back so that you deal with as little XML as possible. Again, there are existing solutions for that in Java SE, but you will have to think if they fit your performance expectations when ported to Android.

A very common requirement is to keep a server connection open to listen to push-type server events. Luckily, Android has Service (you can read one of the first articles on this blog to learn more about it). If you want to build a higher-level abstract that represents a service-that-keeps-server-connection, I think it could be a good idea and quite reusable in various apps.

What I would also recommend is that if you decide to let a Service do some of your server requests, then it should probably do all of them. You shouldn’t generally do server requests both from Activities and Services. If an Activity needs to issue a server request, you can use IPC to pass it to the Service. Doing all of your server requests in one place allows you to queue them up gracefully, maybe optimize them, and avoid unexpected network load.

In any case, what you would like to do in enterprise apps is to abstractize from the low-level details of opening connections, retrying requests and parsing HTTP and XML data. There is little platform support, but it is possible to do yourself.

Local storage

Android has multiple ways for you to store some data on the device. You can use SQLite behind a ContentProvider, or plain SQLite using SQLiteDatabase, or files using serialization, parcelization or whatever other format.

What I am missing here, again, is a way to bind model data to storage and back. Serialization is not as affected by this problem but it has various other issues such as keeping track of class versions and incompatibility with anything else but Java.

Don’t get me wrong, I’m not saying we need to port Hibernate to Android – at least not now. But a more lightweight and Android-optimized framework to save you from doing it all manually could be very useful. Perhaps a good solution would be to generate source or byte code, or just the necessary SQL strings at the compilation stage. Any automated solution would be better than writing SQL manually as Java strings if you ask me.

In addition, the server connection part of your framework can be combined with the local storage part to build solutions for synchronization, caching and offline access in your app – all of which are appreciated in enterprise applications.

Conclusion

I just covered the “most missing” parts of API that we need and we develop for our enterprise apps. There are other things too, such as building custom UI themes (which is not so easy), security issues when allowing to sign in using different server accounts, encryption etc.

What I was trying to say with this article is not that the Android platform is poor in its enterprise features. Well, maybe a little. But we Android devs as a community should be looking towards sharing and open-sourcing more of what we do when it comes to reusable components, be it enterprise or any other genre of apps. At some point, we should introduce a Spring Framework equivalent, a Hibernate equivalent and a JSF (or whatever) equivalent that are all built specifically for Android. What do you think?

How do you separate application state, user interaction logic and data presentation in your Android apps?

Platform designers did not enforce any high-level application architecture framework upon us but left us enough options to implement our own solutions based on application requirements and scale. Most simple applications will get away with just storing data in their widgets (such as in text fields, spinners etc.) and their state manipulation right in event handlers such as OnClickListener‘s. However, if you are going to write a complex application or plan to sophisticate your simple app further, you should really think how to layer it well so that the architecture supports adding new features and satisfies the expected performance, flexibility, responsiveness and other requirements, and your code does not become a mess.

In this article, I will show you one practical approach to dividing application code into three layers according the MVC paradigm and connecting the view to the controller using the Android messaging framework. I used it in my own code and although it might not be 100% academically correct or applicable for every possible app, I love the benefits it gives to me as my app grows more and more complex.

Intro

I will not go deep into explaining the MVC architecture pattern – there’s a lot of places where you can read about it. I wrote an article on how to apply this architecture to a single component but it can also be applied to the entire application. I expect you understand what the Model, View and Controller layers are expected to be there for.

In addition, if you did not read the article on Handler, please do so. Although it does not cover messaging per se, it gives you understanding of the threading model that is behind it.

I wrote a sample app that I will use throughout this article. You can download the complete source at the end of the article.

Part 1: Model

It is a good sign when the Model layer is generic and is only tied to the domain logic. In our demo app, it is completely obvious and boring, except for few things.

First, there is a separate class that contains the model “data”, in other words, whatever state that might change in the Model layer during the application lifecycle:

@Immutable
public final class ModelData implements Serializable {

	private static final long serialVersionUID = 1L;

	private final int answer;
	
	public ModelData(int answer) {
		this.answer = answer;
	}
	
	public final int getAnswer() {
		return answer;
	}
}

As you can see, I like to put a descriptive annotation like @Immutable so that I remember this class is actually immutable which gives a lot of benefits such as the ability to share instances between threads and others. In addition, I made the class Serializable – serializing it would allow to, for example, implement a “Save/Load” feature.

As for the Model class itself, here it is:

@ThreadSafe
public class Model {
	public interface Listener {
		void onModelStateUpdated(Model model);
	}
	
	private ModelData data = new ModelData(0);
	
	private final List<Listener> listeners = new ArrayList<Listener>();
	
	public Model() {
		
	}
	
	public final ModelData getData() {
		synchronized (this) {
			return data;
		}
	}
	
	public final void updateData() { // takes a while!
		SystemClock.sleep(5000);
		ModelData newData = new ModelData(new Random().nextInt(10) + 1);
		
		synchronized (this) {
			data = newData;
		}
		
		synchronized (listeners) {
			for (Listener listener : listeners) {
				listener.onModelStateUpdated(this);
			}
		}
	}
	
	public final void addListener(Listener listener) {
		synchronized (listeners) {
			listeners.add(listener);
		}
	}
	
	public final void removeListener(Listener listener) {
		synchronized (listeners) {
			listeners.remove(listener);
		}
	}
}

As you can see, the updateData() method is written so that it takes some time to execute, which in the real life could be a request to the server and/or a heavy computation. The data is kept protected from concurrent access using synchronized. Listeners can also be added and removed from concurrent threads. If you actually understood MVC, you should realize that model listeners are a core part of the pattern – they allow you to tie any code to the model without the model knowing anything about it (and thus the model code avoids any non-business logic injection).

Now to the most interesting part – the Controller layer.

Part 2: Controller

At this point you can implement the Controller layer in various ways. The way I chose to present is based on Handler, HandlerThread and Message classes. Again, you should have a basic understanding of how they work.

In essence, with those classes the Android platform allows you to implement effective, safe and powerful messaging between various pieces of your app. In our case, what I’d like to do is:

• Give the Controller an inbox handler – so that the View layer can send messages to it, such as “the user asked us to quit” or “the user asked us to update the data”.
• Give the ability to attach one or multiple outbox handlers to the Controller, so that the Controller can send its own messages to the View (or Views) – such as “model update finished” or “destroy yourself, we are quitting” etc.
• This way, the View can only turn user events into messages and send them to Controller without thinking of how to react to them, and whenever the Controller gives any message to the View, the View will mindlessly execute whatever the Controller tells it while presenting the application status to the user. Thus we get a clean implementation of the MVC paradigm.

Now, since every Handler needs a Looper, we need to choose a thread where we will run the Controller’s inbox handler. We could either do it on the UI thread (and it will process both UI messages and Controller’s messages) or on a new separate HandlerThread. Normally our Controller is going to process its messages without blocking on slow operations so that it can stay responsive to the View. Any slow operations will be run in background threads. Thus, we could use the UI thread for Controller’s messages without worrying much. However, I chose to run a separate thread just because I think that is a cleaner architecture and perhaps it could be a bit more responsive. Since the messaging framework saves us from many concurrency headaches, we can easily jump from one solution to the other without changing much code.

This is how we manage the inbox and outbox handlers in the code:

public class Controller {

        // ... some code omitted ...

	private final HandlerThread inboxHandlerThread;
	private final Handler inboxHandler;
	private final List<Handler> outboxHandlers = new ArrayList<Handler>();
	
        // ... some code omitted ...

	public Controller(Model model) {
		this.model = model;
		
		inboxHandlerThread = new HandlerThread("Controller Inbox"); // note you can also set a priority here
		inboxHandlerThread.start();
		 
        // ... some code omitted ...

		inboxHandler = new Handler(inboxHandlerThread.getLooper()) {
			@Override
			public void handleMessage(Message msg) {
				Controller.this.handleMessage(msg);
			}
		};
	}
	
	public final void dispose() {
		// ask the inbox thread to exit gracefully
		inboxHandlerThread.getLooper().quit();
	}
	
	public final Handler getInboxHandler() {
		return inboxHandler;
	}
	
	public final void addOutboxHandler(Handler handler) {
		outboxHandlers.add(handler);
	}

	public final void removeOutboxHandler(Handler handler) {
		outboxHandlers.remove(handler);
	}
	
	final void notifyOutboxHandlers(int what, int arg1, int arg2, Object obj) {
		if (outboxHandlers.isEmpty()) {
			Log.w(TAG, String.format("No outbox handler to handle outgoing message (%d)", what));
		} else {
			for (Handler handler : outboxHandlers) {
				Message msg = Message.obtain(handler, what, arg1, arg2, obj);
				msg.sendToTarget();
			}
		}
	}

        // ... some code omitted ...
}

This is not the complete code for Controller, we’ll come back to it later. OK, so what you can see is how we initialize and bring down the inbox handler and its thread, how you can subscribe your handler as an outbox one and you how you notify all outbox handlers with a message.

With this code, we can take the inbox handler and send message to it from the View layer and at the same time have a View-bound handler registered as an outbox handler for the Controller, and that will allow the Controller and the View to exchange async messages effectively.

Now in order to actually handle incoming messages and do something, we need to implement the handleMessage() method in the Controller class. In most applications you will want to implement an instance of the State design pattern here because depending on what the application is doing right now it might handle the same messages differently, or ignore some of them. For example, in our application, once you start refreshing the Model, we wouldn’t like the user to allow refreshing it again until we’re done. To illustrate that, here’s the ControllerState interface and its ReadyState implementation:

public interface ControllerState {
	boolean handleMessage(Message msg);
}

final class ReadyState implements ControllerState {

	private final Controller controller;
	
	public ReadyState(Controller controller) {
		this.controller = controller;
	}
	
	@Override
	public final boolean handleMessage(Message msg) {
		switch (msg.what) {
		case V_REQUEST_QUIT:
			onRequestQuit();
			return true;
		case V_REQUEST_UPDATE:
			onRequestUpdate();
			return true;
		case V_REQUEST_DATA:
			onRequestData();
			return true;
		}
		return false;
	}

	private void onRequestData() {
		// send the data to the outbox handlers (view)
		controller.notifyOutboxHandlers(C_DATA, 0, 0, controller.getModel().getData());
	}

	private void onRequestUpdate() {
		// we can't just call model.updateState() here because it will block
		// the inbox thread where this processing is happening.
		// thus we change the state to UpdatingState that will launch and manage
		// a background thread that will do that operation

		controller.changeState(new UpdatingState(controller));
	}

	private void onRequestQuit() {
		controller.quit();
	}
}

By putting the Controller and its states to the same package, we can give them access to Controller’s internals using the default access level. I won’t put UpdatingState code here, it’s in the package at the bottom of the page.

Now here’s how Controller delegates message processing to its state:

public class Controller {

        // ... some code omitted ...
	private ControllerState state;
	
	public Controller(Model model) {
		this.model = model;
		
        // ... some code omitted ...
		 
		this.state = new ReadyState(this);

        // ... some code omitted ...
	}
	
        // ... some code omitted ...

	private void handleMessage(Message msg) {
		Log.d(TAG, "Received message: " + msg);
		
		if (! state.handleMessage(msg)) {
			Log.w(TAG, "Unknown message: " + msg);
		}
	}
	
	final Model getModel() {
		return model;
	}

	final void quit() {
		notifyOutboxHandlers(C_QUIT, 0, 0, null);
	}
	
	final void changeState(ControllerState newState) {
		Log.d(TAG, String.format("Changing state from %s to %s", state, newState));
		state = newState;
	}
}

And this is basically how the Controller layer looks. Now let’s bind a View to it.

Part 3: View

Without many explanations – I guess you’ve got the idea already – here’s the View code (DemoActivity):

public class DemoActivity extends Activity implements Handler.Callback, OnClickListener {
	
	private static final String TAG = DemoActivity.class.getSimpleName();
	
	private Controller controller;
	
    @Override
    public void onCreate(Bundle savedInstanceState) {
        super.onCreate(savedInstanceState);

        setContentView(R.layout.main);
        
        ((Button) findViewById(R.id.update)).setOnClickListener(this);
        ((Button) findViewById(R.id.quit)).setOnClickListener(this);
        
        controller = new Controller(new Model());
        controller.addOutboxHandler(new Handler(this)); // messages will go to .handleMessage()
        
        controller.getInboxHandler().sendEmptyMessage(V_REQUEST_DATA); // request initial data
    }

	@Override
	protected void onDestroy() {
		// I think it is a good idea to not fail in onDestroy()
		try {
			controller.dispose();
		} catch (Throwable t) {
			Log.e(TAG, "Failed to destroy the controller", t);
		} 

		super.onDestroy();
	}

	@Override
	public boolean handleMessage(Message msg) {
		Log.d(TAG, "Received message: " + msg);
		 
		switch (msg.what) {
		case C_QUIT:
			onQuit();
			return true;
		case C_DATA:
			onData((ModelData) msg.obj);
			return true;
		case C_UPDATE_STARTED:
			onUpdateStarted();
			return true;
		case C_UPDATE_FINISHED:
			onUpdateFinished();
			return true;
		}
		return false;
	}

	private void onUpdateStarted() {
		ProgressBar progressBar = (ProgressBar) findViewById(R.id.progress_bar);
		progressBar.setVisibility(View.VISIBLE);
	}

	private void onUpdateFinished() {
		ProgressBar progressBar = (ProgressBar) findViewById(R.id.progress_bar);
		progressBar.setVisibility(View.GONE);
		
		// request the updated data
		controller.getInboxHandler().sendEmptyMessage(V_REQUEST_DATA);
	}

	private void onData(ModelData data) {
		TextView dataView = (TextView) findViewById(R.id.data_view);
		dataView.setText("The answer is "+ data.getAnswer());
	}

	private void onQuit() {
		Log.d(TAG, "Activity quitting");
		finish();
	}

	@Override
	public void onClick(View v) {
		switch (v.getId()) {
		case R.id.update:
			controller.getInboxHandler().sendEmptyMessage(V_REQUEST_UPDATE);
			break;
		case R.id.quit:
			controller.getInboxHandler().sendEmptyMessage(V_REQUEST_QUIT);
			break;
		}
	}
}

In this demo app, the Activity actually owns the Controller instance which holds the model. However, you might consider moving the Controller to a Service if its lifecycle fits the Controller/Model lifecycle better.

Summary

As you can see, the message-based approach really allows building a strong complex Controller and a completely dumb View. That allows us to achieve a good MVC implementation.

As I wrote it in the very beginning, this solution might not be completely valid from the theoretical point of view or not good for any app, however I have experience building a very complex app with many states and it is completely based on messages. There are many weaknesses in this architecture, such as the need to collect message codes somewhere (see ControllerProtocol) and to define message parameters in documentation. But the upside is very good layer separation which gives you enormous benefits as the app grows.

Attachment: async-mvc.zip

After few months of busy work in the underground laboratories, MTR is back again. As you can see, I have even installed a new theme which looks to me kinda less annoying and more effective in terms of content.

What I plan to do is, of course, to continue the series of articles on Android development so we can go on sharing our knowledge and experience. Many things have been changing in the Android world. Tablets are here today, and new versions of the platform are soon to come. Android is becoming a more and more widespread and seriously taken platform and at this point, there is no way back.

As for this blog, it is still going to be about useful content (at least content that I consider useful), not about promo articles or high traffic volumes. I am still very interested in general architecture topics, plus multimedia (especially audio) and, sure enough, your questions, issues and achievements.

Some of the topics that are coming up are:

• Architecture: Building well-layered model/view/controller app architectures
• Audio: Mixing multiple audio streams and generating waveform images
• UI: Making your custom views look good on various devices

However, in addition to those topics that you can count on, I would like to know more about what you dear readers actually would love to get. Hence this little poll – please vote:

Thank you and have a nice day!

There are many cases where we use byte[] in our code. In fact, it is the “rawest” type possible in Java unless you go native. Thus, byte arrays are often used to store raw data such as bitmaps, audio and various binary objects.

The previous two articles on MTR were dedicated to audio decoders, including WAV and MP3. In both cases, raw PCM data that was the result of your decoding was a byte array (which you would later write to AudioTrack).

I already mentioned in one of those articles that you should consider streaming any audio that is longer than the reasonable maximum. However, even if your data will definitely fit into the heap, in most cases you can still do better than just using a byte array. Why? Read on (relevant for non-audio byte[]s as well!)

Some low level GC gore

While we Java developers are used to not take JVM heap topology and garbage collection into account too much (and that is generally the correct mindset), on small devices we should be aware of their existence more than on desktops or all the more on the server side. Yes, JVM will do as much as possible for you to fly free, yet it will not be able to handle extreme cases as smoothly without some assistance from your side.

As you remember from the article on decoding WAVs, a 10 second audio will take about 1.7Mb of heap. This is an acceptable size for Android where most devices will allocate either 16Mb or 24Mb maximum per process. However, storing this data as a large byte[] (which is the straightforward way and the first way that comes to one’s mind) is not a good idea and here is why.

Internally, byte arrays are represented as, well, byte arrays. For obvious reasons they require a continuous heap chunk. That is, if your heap is fragmented, JVM will have to run GC in order to defrag the heap – even if technically you had enough space (total!) to fit the amount of data that you needed.

As the result, you will have a lot of objects being transferred back and forth by GC to make room for that big fat array. Things get worse when you have a handful (more than one) large byte arrays that should definitely fit the heap if you look at the numbers (total size and free heap space), but will give GC a real hard time trying to lay them out in the heap space. The more large objects you have, the longer and scarier GC pauses will become.

Splitting byte[] data into chunks

The obvious solution would be to split the large byte[] into chunks. And in this case, the obvious thing to do is also a good one. Here’s a sketch of the ChunkedByteBlob class that implements a nice split byte container:

public final class ChunkedByteBlob {
	private final int totalSize;
	private final int chunkSize;
	private final int totalChunks;
	private final byte[][] chunks;
	
	public ChunkedByteBlob(int totalSize, int chunkSize) {
		if (totalSize <= 0) {
			throw new IllegalArgumentException("totalSize is <= 0: " + totalSize);
		} else if (totalSize < chunkSize) {
			throw new IllegalArgumentException("totalSize " + totalSize + " is < chunkSize " + chunkSize);
		}
		
		this.totalSize = totalSize;
		this.chunkSize = chunkSize;
		this.totalChunks = (int) Math.ceil((double) totalSize / (double) chunkSize);
		this.chunks = new byte[totalChunks][];
		
		int sizeLeft = totalSize;
		int i = 0;
		while (sizeLeft > 0) {
			int nextChunkSize = Math.min(sizeLeft, chunkSize);
			sizeLeft -= nextChunkSize;
			
			chunks[i] = new byte[nextChunkSize];
			i++;
		}
	}
	
	public int totalSize() {
		return totalSize;
	}
	
	public int totalChunks() {
		return totalChunks;
	}
	
	public byte[] chunk(int i) {
		return chunks[i];
	}	
	
	public void fillFrom(InputStream stream) throws IOException {
		for (int i = 0; i < totalChunks(); ++i) {
			int sizeToRead = chunk(i).length;
			int sizeRead = 0;
			boolean done = false;
			
			while (! done) {
				int readSize = stream.read(chunk(i), sizeRead, sizeToRead - sizeRead);
				if (readSize < 0) {
					throw new EOFException("unexpected EOF");					
				}
				sizeRead += readSize;
				
				if (sizeRead >= sizeToRead) {
					done = true;
				}
			}
		}		
	}

	public void writeTo(OutputStream stream) throws IOException {
		for (byte[] chunk : chunks) {
			stream.write(chunk, 0, chunk.length);
		}
	}
}

It should be obvious how this code works, but please do not just copy-paste it – study it, understand it fully and fix the bugs first.

Introducing this class in my latest app that deals with a lot of audio data removed a big nasty GC pause that happened every time I loaded another audio sample.

AudioTrack won’t let you do that

I already wrote an article on shortcomings of Android audio API. Not being a big fan of blog rants, I can’t but come back to the fact that Android audio APIs are underdocumented, underdesigned and sometimes just broken.

In our case, the problem is that even if you split your large audio sample nicely into chunks, you can only write() once if your AudioTrack is static. Thus, you won’t be able to write chunks one by one and you will have to keep your entire sample as a single large byte[].

However, this does not affect MODE_STREAM or other use cases.

Conclusion

Some GC awareness is always good. On small devices such as Android phones it’s even better, and when dealing with extra-large objects it’s even better. Do not follow this recommendation blindly, but be aware of the problem and this simple solution that might be helpful for you too.

I’m in San Francisco for the JavaOne conference.

If you want to meet for a coffee and discuss anything Android, just let me know (ivan@mindtherobot.com)

In my previous article I outlined the stages you need to go through if you want to manually decode WAVs to PCM to play them on an AudioTrack. I promised to show how to do the same for MP3s and this is what this post is going to be about.

Again, the use case is more common than you might think. The only way you can play an MP3 file via direct Android API is MediaPlayer which is heavyweight, slow and presents only high-level API. If you need to mix or modify audio streams or manage them with low latency, you are on your own. But I will try to help you right now.

Introduction

I really suggest that you read the article about playing a WAV first. It contains some general 101 on digital audio that you need to understand what PCM is and how decoders are used.

While WAV files that we loaded in the previous article contained raw PCM data (that we might need to only trivially preprocess), MP3s contain audio data that is encoded with a complex algorithm that we don’t want to learn or implement. Thus, we need some third-party code that will allow us to convert MP3 data to raw PCM that is playable by AudioTrack.

There are two ways we can go – find a pure Java MP3 decoder or try compile a native MP3 decoder on Android. In this article we will only discuss the pure Java solution in detail. It is fast enough for many purposes, although I have to admit that I use a native solution in my production code (but I still did try the Java one and it worked well – it was just not fast enough for my use case that was very demanding in terms of performance).

Now, let’s see if we can find a good pure Java library and how we use it to play some MP3s.

JLayer

As the result of my endless search for a native MP3 decoder, I can recommend only one library: JLayer. It works perfectly and decodes MP3 with good performance. (I used the Java SE version since Android is almost Java SE. Perhaps the Java ME version would be faster – dunno, try it.) JLayer is under an LGPL license which is friendly to commercial apps.

Before you can use JLayer in your project, you need to include it into your app. I wrote an article on porting existing Java code to Android that might help you. Just be sure to include the JLayer code as source because it might need some modifications (as far as I remember, I just removed some unnecessary parts).

Here’s a piece of code that uses the JLayer API to load some PCM data from an MP3:

	public static byte[] decode(String path, int startMs, int maxMs) 
		throws IOException, com.mindtherobot.libs.mpg.DecoderException {
		ByteArrayOutputStream outStream = new ByteArrayOutputStream(1024);
		
		float totalMs = 0;
		boolean seeking = true;
		
		File file = new File(path);
		InputStream inputStream = new BufferedInputStream(new FileInputStream(file), 8 * 1024);
		try {
			Bitstream bitstream = new Bitstream(inputStream);
			Decoder decoder = new Decoder();
			
			boolean done = false;
			while (! done) {
				Header frameHeader = bitstream.readFrame();
				if (frameHeader == null) {
					done = true;
				} else {
					totalMs += frameHeader.ms_per_frame();

					if (totalMs >= startMs) {
						seeking = false;
					}
					
					if (! seeking) {
						SampleBuffer output = (SampleBuffer) decoder.decodeFrame(frameHeader, bitstream);
						
						if (output.getSampleFrequency() != 44100
								|| output.getChannelCount() != 2) {
							throw new com.mindtherobot.libs.mpg.DecoderException("mono or non-44100 MP3 not supported");
						}
						
						short[] pcm = output.getBuffer();
						for (short s : pcm) {
							outStream.write(s & 0xff);
							outStream.write((s >> 8 ) & 0xff);
						}
					}
					
					if (totalMs >= (startMs + maxMs)) {
						done = true;
					}
				}
				bitstream.closeFrame();
			}
			
			return outStream.toByteArray();
		} catch (BitstreamException e) {
			throw new IOException("Bitstream error: " + e);
		} catch (DecoderException e) {
			Log.w(TAG, "Decoder error", e);
			throw new com.mindtherobot.libs.mpg.DecoderException(e);
		} finally {
			IOUtils.safeClose(inputStream);			
		}
	}

Of course, this code is raw but you should be able to get the general idea from it.

Going the native way

If a pure Java solution just isn’t good enough for you, you can try porting one of the existing open source MP3 decoder libraries that are written in C. Theoretically, a decoder is just a mathematical algorithm that should be rather portable. In practice though, you will need to spend some time finding a library with a good license and then tailoring the code to make it compile and work on Android NDK. However, this is a possible and pretty beneficial way to go since a native implementation might be around 20x faster than the pure Java one.

Conclusion

If you need to decode MP3s in your app, don’t worry, you can do that. I hope this article was helpful in showing you how.

If you’ve managed to hack around the various issues that AudioTrack has, then you are probably enjoying its benefits, such as low latency (in the STATIC mode), ability to generate audio on the fly (in the STREAM mode) and the wonderful ability to access and modify raw sound data before you play it.

However, the problem now is getting that data from a source. Many applications that need to use AudioTrack do not generate PCM audio from scratch (an example of an app that does that would be Ethereal Dialpad and other similar apps). You will probably encounter the need to load and use existing audio samples from file sources such as WAV or MP3 files.

Don’t expect to be able to use MediaPlayer facilities to decode raw audio from WAVs and MP3s. Although MediaPlayer can play those files pretty well, its logic is almost entirely contained in the native layer and there is no option for us to plug in and use the decoders for our own purposes. Thus, we have to decode PCM from audio files manually.

In this article I will cover WAV files and in the next one we will get advanced and read audio from MP3s.

Introduction: Some digital audio terms

If your application is not dedicated to digital audio, then you probably need to know several words and acronyms before we continue our discussion. Don’t worry, it’s all very simple and we don’t need to dig very deep.

• PCM (pulse-code modulation) – the simplest way to turn a physical audio signal into numbers. Basically the signal becomes an array of numbers where each number represents the level of energy (amplitude) of the sound at a specific moment of time. (I’m sorry if this explanation is scientifically inaccurate.) Believe it or not, you can represent a sound of any complexity with this approach, and then play it back very nicely. We will only be talking about linear PCM, that is where each number in the array is a linear representation of the original amplitude. In some cases logarithmic mapping is used to better represent the original amplitude scale – but we won’t discuss those cases.
• Sampling rate – how many samples (amplitude-representing numbers) per second your digital sound has. The more it has, the better sound quality you generally get. Numbers used in consumer audio systems today are usually 22050, 44100 and 48000 Hz (samples per seconds).
• Resolution/sample size/bits per sample – defines the size and format of each number used to represent the amplitude. For example, if you use an 8 bit integer number, you can only represent 256 levels of amplitude, so the original physical waveform will be simplified into 256 discrete levels (and you will lose audio precision/quality). If you use 16 bits, the quality becomes much better. In fact, you will probably use 16 bit audio most of the time. Other options are 24 bits, 32 bits (these are not supported on Android for now) and using float numbers.
• Channels – can be mono (1 channel) or stereo (2 channels), or any greater number (but not on Android). If you want to have stereo audio, you need to have a separate PCM array per each channel, and thus the amount of information doubles.

The definitions above also help you understand the amount of data you need for an audio buffer of a specific format and length. Say you need a buffer to keep 5 seconds of 44100 Hz stereo 16-bit linear PCM data. The calculation is:

5 sec * 44100 samples per sec * 2 bytes per sample * 2 channels = 882,000 bytes

This amount of required memory might be surprising for beginners because when you store audio on disk in an MP3, a 880 Kb file might contain a 1-minute track of the same sampling rate and resolution. That is because advanced formats such as MP3 have very sophisticated ways of compressing audio based on dropping parts of the sound that our brain doesn’t pay much attention to. However, most low-level audio APIs including Android’s AudioTrack can only accept linear PCM. That’s why if we can’t keep the entire sample in memory, we have to deal with streaming, circular buffers and other clever ways to feed audio to the API.

Hope this explanation did not confuse you too much, now let’s continue to actually doing some work with digital audio on Android!

WAV file format

Our goal is to take an InputStream that provides raw bytes from a WAV file and load PCM data from it in one way or another. Then we can push raw PCM data to a correctly configured AudioTrack using AudioTrack.write().

A WAV file has a header chunk and a data chunk. We need to read the header chunk to know the format of the data, such as the sampling rate, resolution etc. In addition, we use the header information to make sure the format is supported. WAV can encapsulate a multitude of formats and we won’t support all of them – probably, only linear PCM that has reasonable sampling rate, resolution and channels.

The details of WAV format are widely available on the internet – just do a Google search. However, no matter how long I searched I did not find a good Java library to read WAVs that would be portable to Android. Thus, I wrote some simple code myself.

Below is the method to read a WAV header:

	private static final String RIFF_HEADER = "RIFF";
	private static final String WAVE_HEADER = "WAVE";
	private static final String FMT_HEADER = "fmt ";
	private static final String DATA_HEADER = "data";
	
	private static final int HEADER_SIZE = 44;
	
	private static final String CHARSET = "ASCII";

        /* ... */

	public static WavInfo readHeader(InputStream wavStream) 
throws IOException, DecoderException {

		ByteBuffer buffer = ByteBuffer.allocate(HEADER_SIZE);
		buffer.order(ByteOrder.LITTLE_ENDIAN);

		wavStream.read(buffer.array(), buffer.arrayOffset(), buffer.capacity());

		buffer.rewind();
		buffer.position(buffer.position() + 20);
		int format = buffer.getShort();
		checkFormat(format == 1, "Unsupported encoding: " + format); // 1 means Linear PCM
		int channels = buffer.getShort();
		checkFormat(channels == 1 || channels == 2, "Unsupported channels: " + channels);
		int rate = buffer.getInt();
		checkFormat(rate <= 48000 && rate >= 11025, "Unsupported rate: " + rate);
		buffer.position(buffer.position() + 6);
		int bits = buffer.getShort();
		checkFormat(bits == 16, "Unsupported bits: " + bits);
		int dataSize = 0;
		while (buffer.getInt() != 0x61746164) { // "data" marker
			Log.d(TAG, "Skipping non-data chunk");
			int size = buffer.getInt();
			wavStream.skip(size);
			
			buffer.rewind();
			wavStream.read(buffer.array(), buffer.arrayOffset(), 8);
			buffer.rewind();
		} 
		dataSize = buffer.getInt();
		checkFormat(dataSize > 0, "wrong datasize: " + dataSize);
		
		return new WavInfo(new FormatSpec(rate, channels == 2), dataSize);
	}

(Sorry for the messy code guys! Please clean it up after copy-pasting before use!)

The missing parts should be obvious. As you can see, I only support 16 bits here but you can modify the code to support 8 bits as well (AudioTrack does not support any other resolution).

This method returns enough information for us to know how to read the rest of the file – the audio data.

	public static byte[] readWavPcm(WavInfo info, InputStream stream) throws IOException {
		byte[] data = new byte[info.getDataSize()];
		stream.read(data, 0, data.length);
		return data;
	}

The byte[] array that we read plus the WavInfo structure that has the sampling rate, resolution and channel info are enough for us to play the audio we read.

If we don’t want to load the entire data array into memory at once, we can make an InputStream out of it and suck it little by little.

Feeding PCM to AudioTrack

At this point there are two cases – either we create a new AudioTrack and are able to choose its format, or we have an existing AudioTrack that might have a different format from the one that our WAV file audio has.

In the first case, things are easy – we just use the AudioTrack constructor with the settings that we read from the WAV header.

In the second case we might need to convert our audio to the target format – the format of the AudioTrack that we are going to feed the audio to. We might need one or several of the following conversions:

• If the sampling rate is different, we need to either drop or duplicate samples to match the target rate
• If the resolution is different, we need to map from the source resolution to the target one – that is map integers from 16 bit to 8 bit or vice versa
• If the channels are different, we either mix stereo channels into one mono channel or duplicate the mono channel data to turn it into quasi-stereo

(Do consider moving those algorithms to the native layer as the native layer does give a big advantage when doing such kind of processing.)

In either case, after we’re sure the formats match, we can use AudioTrack.write() to write the entire buffer or its portions to the AudioTrack for playback.

Remember, if you use the STATIC mode, you have to create an AudioTrack with the exact buffer size to fit our audio and write() the audio data completely before we play(), while in STREAM mode we can write() the data part by part after we play() the AudioTrack.

Conclusion

There may be various reasons why you want to play WAV audio on an AudioTrack. Sometimes the size limitations of SoundPool or the latency and high resource usage of MediaPlayer will make you consider going that way. Sometimes you need to modify the audio or mix it on the fly. In any case, in this article I tried to show you how to do it.

In the next post we will do the same with MP3 audio. Stay tuned!