# OpenMAX AL for Android

This article describes the Android native multimedia APIs based on the Khronos Group OpenMAX AL™ 1.0.1 standard, as of Android API level 14 (Android platform version 4.0) and higher.

OpenMAX AL is a companion API to OpenSL ES, but for multimedia (video and audio) rather than audio only.

Android 4.0 provides a direct, efficient path for low-level streaming multimedia. The new path is ideal for applications that need to maintain complete control over media data before passing it to the platform for presentation. For example, media applications can now retrieve data from any source, apply proprietary encryption/decryption, and then send the data to the platform for display.

Applications can now send processed data to the platform as a multiplexed stream of audio/video content in MPEG-2 transport stream format. The platform de-muxes, decodes, and renders the content. The audio track is rendered to the active audio device, while the video track is rendered to either a Surface or a SurfaceTexture. When rendering to a SurfaceTexture, the application can apply subsequent graphics effects to each frame using OpenGL.

OpenMAX AL provides a C language interface that is also callable from C++, and exposes features similar to these Android APIs callable from Java programming language code:

- android.media.MediaPlayer

As with all of the Android Native Development Kit (NDK), the primary purpose of OpenMAX AL for Android is to facilitate the implementation of shared libraries to be called from Java programming language code via Java Native Interface (JNI). NDK is not intended for writing pure C/C++ applications.

Note: though based on OpenMAX AL, the Android native multimedia API is not a conforming implementation of either OpenMAX AL 1.0.1 profile (media player or media player / recorder). This is because Android does not implement all of the features required by either of the profiles. Any known cases where Android behaves differently than the specification are described in section "Android extensions" below. The Android OpenMAX AL implementation has limited features, and is intended primarily for certain performance-sensitive native streaming multimedia applications such as video players.

The major feature is the ability to play an MPEG-2 transport stream containing a single program stream made up of one H.264 video elementary stream and one AAC audio elementary stream. The application provides the stream via an Android buffer queue data source, which is based on the OpenSL ES buffer queue concept and Android-specific extensions.

The video sink is an `ANativeWindow *` abstract handle, derived from an `android.view.Surface` ("surface"). A Surface from `SurfaceHolder.getSurface()` should be used when displaying an unaltered video within a fixed SurfaceView frame. A Surface from `new Surface(SurfaceTexture)` allows streaming the decoded video frames to an OpenGL ES 2.0 texture, where the frames can be used as input to a shader algorithm in the Graphics Processing Unit (GPU). Be sure to `release()` the Surface as soon as possible after calling `setSurface` or ANativeWindow_fromSurface.

The audio sink is always an output mix, a device-independent mixer object similar to that of OpenSL ES.

## Getting started

### Example code

#### Recommended

Supported and tested example code, usable as a model for your own code, is located in NDK folder `platforms/android-14/samples/native-media/`.

#### Not recommended

The OpenMAX AL 1.0.1 specification contains example code in the appendices (see section "References" below for the link to this specification). However, the examples in Appendix D: Sample Code use features not supported by Android. Some examples also contain typographical errors, or use APIs that are likely to change. Proceed with caution in referring to these; though the code may be helpful in understanding the full OpenMAX AL standard, it should not be used as is with Android.

### Adding OpenMAX AL to your application source code

OpenMAX AL is a C API, but is callable from both C and C++ code.

Add the following lines to your code:

    #include <OMXAL/OpenMAXAL.h>
    #include <OMXAL/OpenMAXAL_Android.h>

### Makefile

Modify your Android.mk as follows:

    LOCAL_LDLIBS += libOpenMAXAL

### Multimedia content

The only supported way to supply multimedia content is via an MPEG-2 transport stream.

Finding or creating useful multimedia content for your application is beyond the scope of this article.

Note that it is your responsibility to ensure that you are legally permitted to play the content.

### Debugging

For robustness, we recommend that you examine the `XAresult` value which is returned by most APIs. Use of `assert` vs. more advanced error handling logic is a matter of coding style and the particular API; see the Wikipedia article on assert for more information. In the supplied example, we have used `assert` for "impossible" conditions which would indicate a coding error, and explicit error handling for others which are more likely to occur in production.

Many API errors result in a log entry, in addition to the non-zero result code. These log entries provide additional detail which can be especially useful for the more complex APIs such as `Engine::CreateMediaPlayer`.

Use adb logcat, the Eclipse ADT plugin LogCat pane, or ddms logcat to see the log.

## Supported features from OpenMAX AL 1.0.1

This section summarizes available features. In some cases, there are limitations which are described in the next sub-section.

### Global entry points

Supported global entry points:

- `xaCreateEngine`
- `xaQueryNumSupportedEngineInterfaces`
- `xaQuerySupportedEngineInterfaces`

### Objects and interfaces

The following figure indicates objects and interfaces supported by Android's OpenMAX AL implementation. A green cell means the feature is supported.

### Limitations

This section details limitations with respect to the supported objects and interfaces from the previous section.

#### Audio

The audio stream type cannot be configured; it is always `AudioManager.STREAM_MUSIC`.

Effects are not supported.

#### Dynamic interface management

`RemoveInterface` and `ResumeInterface` are not supported.

#### Engine

Supported:

- `CreateMediaPlayer`

Not supported:

- `CreateCameraDevice`
- `CreateRadioDevice`
- `CreateLEDDevice`
- `CreateVibraDevice`
- `CreateMetadataExtractor`
- `CreateExtensionObject`
- `GetImplementationInfo`

For `CreateMediaPlayer`, these restrictions apply:

- audio sink is an output mix data locator
- video sink is a native display data locator
- soundbank, LED array, and vibra sinks must be `NULL`

#### MIME data format

In the current Android implementation of OpenMAX AL, a media player receives its source data as an MPEG-2 transport stream via a buffer queue.

The source data locator must be `XA_DATALOCATOR_ANDROIDBUFFERQUEUE` (see "Android extensions" below).

The source data format must be `XADataFormat_MIME`. Initialize `mimeType` to `XA_ANDROID_MIME_MP2TS`, and `containerType` to `XA_CONTAINERTYPE_MPEG_TS`.

The contained transport stream must have a single program with one H.264 video elementary stream and one AAC audio elementary stream.

#### Object

`Resume`, `RegisterCallback`, `AbortAsyncOperation`, `SetPriority`, `GetPriority`, and `SetLossOfControlInterfaces` are not supported.

#### StreamInformation

Use the StreamInformation interface on a media player object to discover when the video metrics (height/width or aspect ratio) are available or changed, and to then get the sizes.

Supported:

- `RegisterStreamChangeCallback`
- `QueryMediaContainerInformation`
- `QueryStreamInformation`
- `QueryStreamType`

Not supported:

- `QueryActiveStreams`
- `QueryStreamName`
- `SetActiveStream`

#### VideoDecoderCapabilities

This interface on the engine object reports video decoder capabilities without interpretation, exactly as claimed by the underlying OpenMAX IL implementation.

These fields in `XAVideoCodecDescriptor` are filled:

- `codecId`
- `profileSetting`
- `levelSetting`

The other fields are not filled and should be ignored.

Applications should rely on the capabilities documented at Android Supported Media Formats, not the information reported by this interface.

### Data structures

Android API level 14 supports these OpenMAX AL 1.0.1 data structures:

- XADataFormat_MIME
- XADataLocator_NativeDisplay
- XADataLocator_OutputMix
- XADataSink
- XADataSource
- XAEngineOption
- XAInterfaceID
- XAMediaContainerInformation
- XANativeHandle
- XA*StreamInformation
- XAVideoCodecDescriptor

#### XADataLocator_NativeDisplay

The native display data locator is used to specify the video sink:

    typedef struct XADataLocator_NativeDisplay_ {
        XAuint32 locatorType;      // XA_DATALOCATOR_NATIVEDISPLAY
        XANativeHandle hWindow;    // ANativeWindow *
        XANativeHandle hDisplay;   // NULL
    } XADataLocator_NativeDisplay;

Set the `hWindow` field to an `ANativeWindow *` and set `hDisplay` to `NULL`. You can get a `ANativeWindow *` handle from an `android.view.Surface`, using this NDK function:

    #include <android/native_window_jni.h>

    ANativeWindow* ANativeWindow_fromSurface(JNIEnv* env, jobject surface);

Don't forget to free this handle in your shutdown code with `ANativeWindow_release`.

### Platform configuration

OpenMAX AL for Android is designed for multi-threaded applications, and is thread-safe.

OpenMAX AL for Android supports a single engine per application, and up to 32 objects. Available device memory and CPU may further restrict the usable number of objects.

`xaCreateEngine` recognizes, but ignores, these engine options:

- `XA_ENGINEOPTION_THREADSAFE`
- `XA_ENGINEOPTION_LOSSOFCONTROL`

OpenMAX AL and OpenSL ES may be used together in the same application. In this case, there is internally a single shared engine object, and the 32 object limit is shared between OpenMAX AL and OpenSL ES. The application should first create both engines, then use both engines, and finally destroy both engines. The implementation maintains a reference count on the shared engine, so that it is correctly destroyed at the second destroy.

## Planning for future versions of OpenMAX AL

The Android native multimedia APIs at level 14 are based on Khronos Group OpenMAX AL 1.0.1 (see section "References" below). As of the time of this writing, Khronos has recently released a revised version 1.1 of the standard. The revised version includes new features, clarifications, correction of typographical errors, and some incompatibilities. Most of the incompatibilities are relatively minor, or are in areas of OpenMAX AL not supported by Android. However, even a small change can be significant for an application developer, so it is important to prepare for this.

The Android team is committed to preserving future API binary compatibility for developers to the extent feasible. It is our intention to continue to support future binary compatibility of the 1.0.1-based API, even as we add support for later versions of the standard. An application developed with this version should work on future versions of the Android platform, provided that you follow the guidelines listed in section "Planning for binary compatibility" below.

Note that future source compatibility will not be a goal. That is, if you upgrade to a newer version of the NDK, you may need to modify your application source code to conform to the new API. We expect that most such changes will be minor; see details below.

### Planning for binary compatibility

We recommend that your application follow these guidelines, to improve future binary compatibility:

- Use only the documented subset of Android-supported features from OpenMAX AL 1.0.1.
- Do not depend on a particular result code for an unsuccessful operation; be prepared to deal with a different result code.
- Application callback handlers generally run in a restricted context, and should be written to perform their work quickly and then return as soon as possible. Do not do complex operations within a callback handler. For example, within a buffer queue completion callback, you can enqueue another buffer, but do not create a media player.
- Callback handlers should be prepared to be called more or less frequently, to receive additional event types, and should ignore event types that they do not recognize. Callbacks that are configured with an event mask of enabled event types should be prepared to be called with multiple event type bits set simultaneously. Use "&" to test for each event bit rather than a switch case.
- Use prefetch status and callbacks as a general indication of progress, but do not depend on specific hard-coded fill levels or callback sequence. The meaning of the prefetch status fill level, and the behavior for errors that are detected during prefetch, may change.

### Planning for source compatibility

As mentioned, source code incompatibilities are expected in the next version of OpenMAX AL from Khronos Group. Likely areas of change include:

- Addition of `const` to input parameters passed by reference, and to `XAchar *` struct fields used as input values. This should not require any changes to your code.
- Substitution of unsigned types for some parameters that are currently signed. You may need to change a parameter type from `XAint32` to `XAuint32` or similar, or add a cast.
- Additional fields in struct types. For output parameters, these new fields can be ignored, but for input parameters the new fields will need to be initialized. Fortunately, these are expected to all be in areas not supported by Android.
- Interface GUIDs will change. Refer to interfaces by symbolic name rather than GUID to avoid a dependency.
- `XAchar` will change from `unsigned char` to `char`. This primarily affects the MIME data format.
- `XADataFormat_MIME.mimeType` will be renamed to `pMimeType`. We recommend that you initialize the `XADataFormat_MIME` data structure using a brace-enclosed comma-separated list of values, rather than by field name, to isolate your code from this change. In the example code we have used this technique.

## Android extensions

The API for Android extensions is defined in `OMXAL/OpenMAXAL_Android.h` . Consult that file for details on these extensions. Unless otherwise noted, all interfaces are "explicit".

Note that use these extensions will limit your application's portability to other OpenMAX AL implementations. If this is a concern, we advise that you avoid using them, or isolate your use of these with `#ifdef` etc.

The following figure shows which Android-specific interfaces and data locators are available for each object type.

### Android buffer queue data locator and interface

#### Comparison with OpenSL ES buffer queue

The Android buffer queue data locator and interface are based on similar concepts from OpenSL ES 1.0.1, with these differences:

- Though currently usable with only a media player and MPEG-2 transport stream data, the Android buffer queue API is designed for flexibility so that the API can also apply to other use cases in the future.
- Commands may be optionally specified by the application at time of `Enqueue`. Each command consists of an item key and optional item value. Command key/value pairs are carried alongside the corresponding buffer in the queue, and thus are processed in synchrony with the buffer.
- To enqueue command(s) without associated data, specify a buffer address of NULL and buffer size of zero, along with at least one command.
- Status may be provided by the implementation during a completion callback. Each status consists of an item key and optional item value. Status key/value pairs are carried alongside the corresponding buffer in the queue, and thus are received by the application in synchrony with the completion callback.
- The completion callback receives additional parameters: buffer address, buffer maximum data size, buffer actual size consumed (or filled by a future recorder object), and a `void *` for application.
- The callback returns a value, which must be `XA_RESULT_SUCCESS`.

The data locator type code is `XA_DATALOCATOR_ANDROIDBUFFERQUEUE` and the associated structure is `XADataLocator_AndroidBufferQueue`.

The interface ID is `XA_IID_ANDROIDBUFFERQUEUESOURCE`.

#### Usage

A typical buffer queue configuration is 8 buffers of 1880 bytes each.

The application enqueues filled buffers of data in MPEG-2 transport stream format. The buffer size must be a multiple of 188 bytes, the size of an MPEG-2 transport stream packet. The buffer data must be properly aligned on a packet boundary, and formatted per the MPEG-2 Part 1 specification.

An application may supply zero or one of these item codes (command key/value pairs) at `Enqueue`:

XA_ANDROID_ITEMKEY_EOS
  End of stream. Informs the decode and rendering components that playback is complete. The application must not call `Enqueue` again. There is no associated value, so `itemSize` must be zero. There must be no data buffer alongside the EOS command.
XA_ANDROID_ITEMKEY_DISCONTINUITY
  Discontinuity. This and following buffers have a new presentation time. The new presentation time may be optionally specified as a parameter, expressed in `itemData` as a 64-bit unsigned integer in units of 90 kHz clock ticks. The `itemSize` should be either zero or 8. The discontinuity command is intended for seeking to a new point in the stream. The application should flush its internal data, then send the discontinuity command prior to, or alongside of, the first buffer corresponding to the new stream position. The initial packets in the video elementary stream should describe an IDR (Instantaneous Decoding Refresh) frame. Note that the discontinuity command is not intended for stream configuration / format changes; for these use `XA_ANDROID_ITEMKEY_FORMAT_CHANGE`.
XA_ANDROID_ITEMKEY_FORMAT_CHANGE
  Format change. This and following buffers have a new format, for example for MBR (Multiple Bit Rate) or resolution switching.

Upon notification of completion via a registered callback, the now consumed buffer is available for the application to re-fill.

The implementation may supply zero or more of these item codes (status key/value pairs) to the callback handler:

XA_ANDROID_ITEMKEY_BUFFERQUEUEEVENT
  Buffer queue event mask. The `itemSize` is 4, and `itemData` contains the bit-wise "or" of zero or more of the `XA_ANDROIDBUFFERQUEUEEVENT_*` symbols. This event mask explains the reason(s) why the callback handler was called.

### Reporting of extensions

`Engine::QueryNumSupportedExtensions`, `Engine::QuerySupportedExtension`, `Engine::IsExtensionSupported` report these extensions:

- `ANDROID_SDK_LEVEL_#` where # is the platform API level, 14 or higher

## Programming notes

These notes supplement the OpenMAX AL 1.0.1 specification, available in the "References" section below.

### Objects and interface initialization

Two aspects of the OpenMAX AL programming model that may be unfamiliar to new developers are the distinction between objects and interfaces, and the initialization sequence.

Briefly, an OpenMAX AL object is similar to the object concept in programming languages such as Java and C++, except an OpenMAX AL object is only visible via its associated interfaces. This includes the initial interface for all objects, called `XAObjectItf`. There is no handle for an object itself, only a handle to the `XAObjectItf` interface of the object.

An OpenMAX AL object is first "created", which returns an `XAObjectItf`, then "realized". This is similar to the common programming pattern of first constructing an object (which should never fail other than for lack of memory or invalid parameters), and then completing initialization (which may fail due to lack of resources). The realize step gives the implementation a logical place to allocate additional resources if needed.

As part of the API to create an object, an application specifies an array of desired interfaces that it plans to acquire later. Note that this array does not automatically acquire the interfaces; it merely indicates a future intention to acquire them. Interfaces are distinguished as "implicit" or "explicit". An explicit interface must be listed in the array if it will be acquired later. An implicit interface need not be listed in the object create array, but there is no harm in listing it there. OpenMAX AL has one more kind of interface called "dynamic", which does not need to be specified in the object create array, and can be added later after the object is created. The Android implementation provides a convenience feature to avoid this complexity; see section "Dynamic interfaces at object creation" above.

After the object is created and realized, the application should acquire interfaces for each feature it needs, using `GetInterface` on the initial `XAObjectItf`.

Finally, the object is available for use via its interfaces, though note that some objects require further setup. Some use cases needs a bit more preparation in order to detect connection errors. See the next section "Media player prefetch" for details.

After your application is done with the object, you should explicitly destroy it; see section "Destroy" below.

### Media player prefetch

For a media player, `Object::Realize` allocates resources but does not connect to the data source (i.e. "prepare") or begin pre-fetching data. These occur once the player state is set to either `XA_PLAYSTATE_PAUSED` or `XA_PLAYSTATE_PLAYING`.

The prefetch status interface is useful for detecting errors. Register a callback and enable at least the `XA_PREFETCHEVENT_FILLLEVELCHANGE` and `XA_PREFETCHEVENT_STATUSCHANGE` events. If both of these events are delivered simultaneously, and `PrefetchStatus::GetFillLevel` reports a zero level, and `PrefetchStatus::GetPrefetchStatus` reports `XA_PREFETCHSTATUS_UNDERFLOW`, then this indicates a non-recoverable error in the data source or in rendering to the video sink.

The next version of OpenMAX AL is expected to add more explicit support for handling errors in the data source. However, for future binary compatibility, we intend to continue to support the current method for reporting a non-recoverable error.

In summary, a recommended code sequence is:

- Engine::CreateMediaPlayer
- Object:Realize
- Object::GetInterface for XA_IID_PREFETCHSTATUS
- PrefetchStatus::SetCallbackEventsMask
- PrefetchStatus::RegisterCallback
- Object::GetInterface for XA_IID_PLAY
- Play::SetPlayState to XA_PLAYSTATE_PAUSED or XA_PLAYSTATE_PLAYING
- preparation and prefetching occur here; during this time your callback will be called with periodic status updates and error notifications

### Destroy

Be sure to destroy all objects on exit from your application. Objects should be destroyed in reverse order of their creation, as it is not safe to destroy an object that has any dependent objects. For example, destroy in this order: audio players and recorders, output mix, then finally the engine.

OpenMAX AL does not support automatic garbage collection or reference counting of interfaces. After you call `Object::Destroy`, all extant interfaces derived from the associated object become undefined.

The Android OpenMAX AL implementation does not detect the incorrect use of such interfaces. Continuing to use such interfaces after the object is destroyed will cause your application to crash or behave in unpredictable ways.

We recommend that you explicitly set both the primary object interface and all associated interfaces to NULL as part of your object destruction sequence, to prevent the accidental misuse of a stale interface handle.

### Callbacks and threads

Callback handlers are generally called synchronously with respect to the event, that is, at the moment and location where the event is detected by the implementation. But this point is asynchronous with respect to the application. Thus you should use a mutex or other synchronization mechanism to control access to any variables shared between the application and the callback handler. In the example code, such as for buffer queues, we have omitted this synchronization in the interest of simplicity. However, proper mutual exclusion would be critical for any production code.

Callback handlers are called from internal non-application thread(s) which are not attached to the Dalvik virtual machine and thus are ineligible to use JNI. Because these internal threads are critical to the integrity of the OpenMAX AL implementation, a callback handler should also not block or perform excessive work. Therefore, if your callback handler needs to use JNI or do anything significant (e.g. beyond an `Enqueue` or something else simple such as the "Get" family), the handler should instead post an event for another thread to process.

Note that the converse is safe: a Dalvik application thread which has entered JNI is allowed to directly call OpenMAX AL APIs, including those which block. However, blocking calls are not recommended from the main thread, as they may result in the dreaded "Application Not Responding" (ANR).

### Performance

As OpenMAX AL is a native C API, non-Dalvik application threads which call OpenMAX AL have no Dalvik-related overhead such as garbage collection pauses. However, there is no additional performance benefit to the use of OpenMAX AL other than this. In particular, use of OpenMAX AL does not result in lower audio or video latency, higher scheduling priority, etc. than what the platform generally provides. On the other hand, as the Android platform and specific device implementations continue to evolve, an OpenMAX AL application can expect to benefit from any future system performance improvements.

### Security and permissions

As far as who can do what, security in Android is done at the process level. Java programming language code can't do anything more than native code, nor can native code do anything more than Java programming language code. The only differences between them are what APIs are available that provide functionality that the platform promises to support in the future and across different devices.

Applications using OpenMAX AL must request whatever permissions they would need for similar non-native APIs. Applications that play network resources need `android.permission.NETWORK`. Note that the current Android implementation does not directly access the network, but many applications that play multimedia receive their data via the network.

## Platform issues

This section describes known issues in the initial platform release which supports these APIs.

### Dynamic interface management

`DynamicInterfaceManagement::AddInterface` does not work. Instead, specify the interface in the array passed to Create.

## References and resources

Android:

- Android developer resources
- Android developers discussion group
- Android NDK
- Android NDK discussion group (for developers of native code, including OpenMAX AL)
- Android open source bug database

Khronos Group:

- Khronos Group OpenMAX AL Overview
- Khronos Group OpenMAX AL 1.0.1 specification
- Khronos Group public message board for OpenMAX AL (please limit to non-Android questions)

For convenience, we have included a copy of the OpenMAX AL 1.0.1 specification with the NDK in `docs/openmaxal/OpenMAX_AL_1_0_1_Specification.pdf`.

Miscellaneous:

- JNI
- MPEG-2
- MPEG-2 transport stream
- H.264
- AAC