README.md
1# Benchmark tests
2
3Benchmark app analyses the time taken by MediaCodec, MediaExtractor and MediaMuxer for given set of inputs. It is used to benchmark these modules on android devices.
4Benchmark results are published as a CSV report.
5
6This page describes steps to run the NDK and SDK layer test.
7
8Run the following steps to build the test suite:
9```
10mmm frameworks/av/media/tests/benchmark/
11```
12
13# Resources
14The resource file for the tests is taken from [here](https://storage.googleapis.com/android_media/frameworks/av/media/tests/benchmark/MediaBenchmark.zip)
15
16Download the MediaBenchmark.zip file, unzip and push it to /data/local/tmp/ on the device.
17
18```
19unzip MediaBenchmark.zip
20adb push MediaBenchmark /data/local/tmp/MediaBenchmark/res/
21```
22
23The resource files are assumed to be at /data/local/tmp/MediaBenchmark/res/. You can use a different location, but you have to modify the rest of the instructions to replace /data/local/tmp/MediaBenchmark/res/ with wherever you chose to put the files.
24
25# NDK CLI Tests
26Note: [Benchmark Application](#BenchmarkApplication) now supports profiling both SDK and NDK APIs and that is the preferred way to benchmark codecs
27
28To run the test suite for measuring performance of the native layer, follow the following steps:
29
30The 64-bit binaries will be created in the following path : ${OUT}/data/nativetest64/
31
32To test 64-bit binary push binaries from nativetest64.
33
34adb push $OUT/data/nativetest64/* /data/local/tmp/. For example
35
36```
37adb push $OUT/data/nativetest64/extractorTest/extractorTest /data/local/tmp/
38```
39
40The 32-bit binaries will be created in the following path : ${OUT}/data/nativetest/
41
42To test 32-bit binary push binaries from nativetest.
43
44adb push $OUT/data/nativetest/* /data/local/tmp/. For example
45
46```
47adb push $OUT/data/nativetest/extractorTest/extractorTest /data/local/tmp/
48```
49
50To run the binary, follow the commands mentioned below under each module.
51
52## Extractor
53
54The test extracts elementary stream and benchmarks the extractors available in NDK.
55
56```
57adb shell /data/local/tmp/extractorTest -P /data/local/tmp/MediaBenchmark/res/
58```
59
60## Decoder
61
62The test decodes input stream and benchmarks the decoders available in NDK.
63
64```
65adb shell /data/local/tmp/decoderTest -P /data/local/tmp/MediaBenchmark/res/
66```
67
68## Muxer
69
70The test muxes elementary stream and benchmarks the muxers available in NDK.
71
72```
73adb shell /data/local/tmp/muxerTest -P /data/local/tmp/MediaBenchmark/res/
74```
75
76## Encoder
77
78The test encodes input stream and benchmarks the encoders available in NDK.
79
80```
81adb shell /data/local/tmp/encoderTest -P /data/local/tmp/MediaBenchmark/res/
82```
83
84# <a name="BenchmarkApplication"></a> Benchmark Application
85To run the test suite for measuring performance of the SDK and NDK APIs, follow the following steps:
86Benchmark Application can be run in two ways.
87
88## Steps to run with atest
89Note that atest command will install Benchmark application and push the required test files to the device as well.
90
91For running all the tests, run the following command
92```
93atest com.android.media.benchmark.tests -- --enable-module-dynamic-download=true
94```
95
96For running the tests individually, run the following atest commands:
97
98```
99atest com.android.media.benchmark.tests.ExtractorTest -- --enable-module-dynamic-download=true
100atest com.android.media.benchmark.tests.DecoderTest -- --enable-module-dynamic-download=true
101atest com.android.media.benchmark.tests.MuxerTest -- --enable-module-dynamic-download=true
102atest com.android.media.benchmark.tests.EncoderTest -- --enable-module-dynamic-download=true
103```
104
105## Steps to run without atest
106
107The apk will be created at the following path:
108
109The 64-bit apk will be created in the following path :
110$OUT/testcases/MediaBenchmarkTest/arm64/
111
112For installing the apk, run the command:
113```
114adb install -f -r $OUT/testcases/MediaBenchmarkTest/arm64/MediaBenchmarkTest.apk
115```
116
117The 32-bit apk will be created in the following path :
118$OUT/testcases/MediaBenchmarkTest/arm/
119
120For installing the apk, run the command:
121```
122adb install -f -r $OUT/testcases/MediaBenchmarkTest/arm/MediaBenchmarkTest.apk
123```
124
125To get the resource files for the test follow instructions given in [Resources](#Resources)
126
127For running all the tests, run the following command
128```
129adb shell am instrument -w -r -e package com.android.media.benchmark.tests com.android.media.benchmark/androidx.test.runner.AndroidJUnitRunner
130```
131
132## Extractor
133
134The test extracts elementary stream and benchmarks the extractors available in SDK and NDK.
135```
136adb shell am instrument -w -r -e class 'com.android.media.benchmark.tests.ExtractorTest' com.android.media.benchmark/androidx.test.runner.AndroidJUnitRunner
137```
138
139## Decoder
140
141The test decodes input stream and benchmarks the decoders available in SDK and NDK.
142```
143adb shell am instrument -w -r -e class 'com.android.media.benchmark.tests.DecoderTest' com.android.media.benchmark/androidx.test.runner.AndroidJUnitRunner
144```
145
146## Muxer
147
148The test muxes elementary stream and benchmarks different writers available in SDK and NDK.
149```
150adb shell am instrument -w -r -e class 'com.android.media.benchmark.tests.MuxerTest' com.android.media.benchmark/androidx.test.runner.AndroidJUnitRunner
151```
152
153## Encoder
154
155The test encodes input stream and benchmarks the encoders available in SDK and NDK.
156```
157adb shell am instrument -w -r -e class 'com.android.media.benchmark.tests.EncoderTest' com.android.media.benchmark/androidx.test.runner.AndroidJUnitRunner
158```
159
160# Codec2
161To run the test suite for measuring performance of the codec2 layer, follow the following steps:
162
163The 32-bit binaries will be created in the following path : ${OUT}/data/nativetest/
164
165The 64-bit binaries will be created in the following path : ${OUT}/data/nativetest64/
166
167To test 64-bit binary push binaries from nativetest64.
168adb push $(OUT)/data/nativetest64/* /data/local/tmp/
169```
170adb push $(OUT)/data/nativetest64/C2DecoderTest/C2DecoderTest /data/local/tmp/
171```
172
173To test 32-bit binary push binaries from nativetest.
174adb push $(OUT)/data/nativetest/* /data/local/tmp/
175```
176adb push $(OUT)/data/nativetest/C2DecoderTest/C2DecoderTest /data/local/tmp/
177```
178
179To get the resource files for the test follow instructions given in [Resources](#Resources)
180
181## C2 Decoder
182
183The test decodes input stream and benchmarks the codec2 decoders available in device.
184
185```
186adb shell /data/local/tmp/C2DecoderTest -P /data/local/tmp/MediaBenchmark/res/
187```
188## C2 Encoder
189
190The test encodes input stream and benchmarks the codec2 encoders available in device.
191
192```
193adb shell /data/local/tmp/C2EncoderTest -P /data/local/tmp/MediaBenchmark/res/
194```
195
196# Analysis
197
198The benchmark results are stored in a CSV file which can be used for analysis. These results are stored in following format:
199<app directory>/<module_name>.<timestamp>.csv
200
201Note: This timestamp is in nano seconds and will change based on current system time.
202
203To find the location of the CSV file, look for the path in logs. Example log below -
204
205```
206com.android.media.benchmark D/DecoderTest: Saving Benchmark results in: /storage/emulated/0/Android/data/com.android.media.benchmark/files/Decoder.1587732395387.csv
207```
208
209This file can be pulled from the device using "adb pull" command.
210```
211adb pull /storage/emulated/0/Android/data/com.android.media.benchmark/files/Decoder.1587732395387.csv ./Decoder.1587732395387.csv
212```
213
214## CSV Columns
215
216Following columns are available in CSV.
217
218Note: All time values are in nano seconds
219
2201. **currentTime** : The time recorded at the creation of the stats. This may be used to estimate time between consecutive test clips.
221
2222. **fileName**: The file being used as an input for the benchmark test.
223
2243. **operation**: The current operation on the input test vector i.e. Extract/Mux/Encode/Decode.
225
2264. **NDK/SDK**: The target APIs i.e. AMedia vs Media calls for the operation being performed.
227
2285. **sync/async**: This is specific to MediaCodec objects (i.e. Encoder and Decoder). It specifies the mode in which MediaCodec APIs are working. For async mode, callbacks are set. For sync mode, we have to poll the dequeueBuffer APIs to queue and dequeue input output buffers respectively.
229
2306. **setupTime**: The time taken to set up the MediaExtractor/Muxer/Codec instance.
231
232 * MediaCodec: includes setting async/sync mode, configuring with a format and codec.start
233
234 * MediaExtractor: includes AMediaExtractor_new and setDataSource.
235
236 * MediaMuxer: includes creating the object, adding track, and starting the muxer.
237
2387. **destroyTime**: The time taken to stop and close MediaExtractor/Muxer/Codec instance.
239
2408. **minimumTime**: The minimum time taken to extract/mux/encode/decode a frame.
241
2429. **maximumTime**: The maximum time taken to extract/mux/encode/decode a frame.
243
24410. **averageTime**: Average time taken to extract/mux/encode/decode per frame.
245
246 * MediaCodec: computed as the total time taken to encode/decode all frames divided by the number of frames encoded/decoded.
247
248 * MediaExtractor: computed as the total time taken to extract all frames divided by the number of frames extracted.
249
250 * MediaMuxer: computed as the total time taken to mux all frames divided by the number of frames muxed.
251
25211. **timeToProcess1SecContent**: The time required to process one second worth input data.
253
25412. **totalBytesProcessedPerSec**: The number of bytes extracted/muxed/decoded/encoded per second.
255
25613. **timeToFirstFrame**: The time taken to receive the first output frame.
257
25814. **totalSizeInBytes**: The total output size of the operation (in bytes).
259
26015. **totalTime**: The time taken to perform the complete operation (i.e. Extract/Mux/Decode/Encode) for respective test vector.
261
262
263## Muxer
2641. **componentName**: The format of the output Media file. Following muxers are currently supported:
265 * Ogg, Webm, 3gpp, and mp4.
266
267## Decoder
2681. **componentName**: Includes all supported codecs on the device. Aliased components are skipped.
269 * Video: H263, H264, H265, VPx, Mpeg4, Mpeg2, AV1
270 * Audio: AAC, Flac, Opus, MP3, Vorbis, GSM, AMR-NB/WB
271
272## Encoder
2731. **componentName**: Includes all supported codecs on the device. Aliased components are skipped.
274 * Video: H263, H264, H265, VPx, Mpeg4
275 * Audio: AAC, Flac, Opus, AMR-NB/WB
276
277## Common Failures
278On some devices, if a codec isn't supported some tests may report a failure like "codec not found for"
279
280For example: On mobile devices without support for mpeg2 decoder, following failure is observed:
281```
282Unable to create codec by mime: video/mpeg2
283```
284