Integrating Machine Learning Models into Android Apps

The complete pipeline for shipping ML on Android - from converting a model to running inference at 35 ms on-device, with the code, the pitfalls, and the performance rules. Three years ago, a health-tech client wanted a screening feature in their Android app: point the camera at a skin image, and get a risk score locally, with no network call. The reason was not convenience - it was privacy. Medical images leaving the device would have triggered compliance, consent, and a GDPR conversation the startup was not ready for. They needed the model to run on the phone. The first attempt failed because the team treated it like a normal feature. They wrapped a TensorFlow SavedModel in a thin service, ran inference on the main thread, and shipped a model that had not been quantized. The result: app start times ballooned, inference froze the UI, the APK gained 45 MB, and the battery graph looked like a cliff. The feature was turned off two weeks after launch. I have rebuilt that pipeline four times since, for a dozen clients across different industries. This article is the exact process I now use - the same sequence every time, with the code and the pitfalls that taught me the rules. Step 1 - Decide Where the Model Runs The first decision is not technical; it is architectural. You have three options: - On-device inference (TensorFlow Lite). Fast, private, offline, zero cost per call. But the model is fixed at install time (or downloaded later), and you are limited by phone hardware. - Cloud inference. Unlimited model size, easy to update, but every call costs money, adds latency, and sends user data off-device. - Hybrid. A small on-device model for the common case, a bigger cloud model for edge cases. My default for anything with private data, or anything that needs a response in under a second, is on-device. The health app went on-device and the risk score came back in under 100 ms, which changed the whole product feel. Step 2 - Convert and Quantize the Model You train in TensorFlow or PyTorch, but Android runs TFLite. The conversion step is where most people first stumble. From a trained model: import tensorflow as tf # Load your trained Keras/SavedModel model = tf.saved_model.load("path/to/saved_model") # Convert to a TFLite flatbuffer converter = tf.lite.TFLiteConverter.from_saved_model("path/to/saved_model") tflite_model = converter.convert() with open("model.tflite", "wb") as f: f.write(tflite_model) If you convert this way, you get a float32 model - and it will be big and slow on-device. The important part is quantization. Converting weights from float32 to int8 shrinks the model by 75 percent and can make inference several times faster on devices with the right hardware, at a small accuracy cost: converter = tf.lite.TFLiteConverter.from_saved_model("path/to/saved_model") converter.optimizations = [tf.lite.Optimize.DEFAULT] converter.representative_dataset = representative_dataset # sample inputs tflite_quant = converter.convert() The representative_dataset is a small set of typical inputs - the quantizer uses them to calibrate the range of values. Without it, integer-only quantization will not work. This is the most common conversion failure I see: people skip the calibration set and then wonder why the model converts wrong or loses accuracy. For PyTorch models, export to ONNX first, then convert via onnx2tf or the official converter. Keep the pipeline in CI so the model file is a build artifact, not a manual download - I have seen three versions of "model.tflite" in a repo because someone forgot to replace the file. Step 3 - Add Dependencies and Package the Model In the app module, add the TFLite runtime. Keep it small: the base interpreter is around 1.5 MB; the full tensorflow-lite-support adds utilities and ML Kit glue but also weight. Start minimal: dependencies { implementation("org.tensorflow:tensorflow-lite:2.16.1") // optional, for GPU delegation: implementation("org.tensorflow:tensorflow-lite-gpu:2.16.1") } Put the model in src/main/assets/ : app/src/main/assets/model.tflite Do not paste the model into res/raw and do not load it from a network path at startup. Assets are packed into the APK and load via the interpreter natively. Step 4 - Load the Interpreter Off the Main Thread This is the rule that saves your UI. Loading the model and running inference both do I/O and compute - doing either on the main thread gives you an ANR. The correct pattern is a small wrapper class that loads once and keeps the interpreter alive for reuse: class Classifier(context: Context) { private val interpreter: Interpreter private val inputBuffer: TensorBuffer private val outputBuffer: TensorBuffer init { val options = Interpreter.Options() options.numThreads = 4 val modelBytes = context.assets.open("model.tflite").use { it.readBytes() } val buffer = ByteBuffer.allocateDirect(modelBytes.size) buffer.put(modelBytes) buffer.rewind() interpreter = Interpreter(buffer, options) val inputShape = intArrayOf(1, 224, 224, 3) val outputShape = intArrayOf(1, NUM_CLASSES) inputBuffer = TensorBuffer.createFixedSize(inputShape, DataType.FLOAT32) outputBuffer = TensorBuffer.createFixedSize(outputShape, DataType.FLOAT32) } fun predict(pixels: FloatArray): FloatArray { inputBuffer.loadArray(pixels) interpreter.run(inputBuffer.buffer, outputBuffer.buffer) return outputBuffer.floatArray } } Two details worth calling out: - Load once, reuse forever. Instantiating an Interpreter per call is the number one performance bug in ML-on-Android code I review. Model loading can take hundreds of milliseconds; keep one interpreter alive. - Do inference on a background thread. Use a coroutine on Dispatchers.Default or an executor. Never block the main thread withinterpreter.run . suspend fun predictAsync(pixels: FloatArray): FloatArray = withContext(Dispatchers.Default) { classifier.predict(pixels) } Step 5 - Preprocess the Input Correctly Models do not accept raw images or raw text. They accept tensors with a specific shape, range, and ordering. The two mistakes that dominate: Normalization. Most image models expect input in the range [-1, 1] or [0, 1]. A raw Bitmap gives you 0-255 per channel. Forgetting to normalize produces a model that outputs garbage with total confidence - and it is maddening because it works in Python and breaks on Android. private fun bitmapToFloatArray(bitmap: Bitmap): FloatArray { val scaled = Bitmap.createScaledBitmap(bitmap, 224, 224, true) val pixels = IntArray(224 * 224) scaled.getPixels(pixels, 0, 224, 0, 0, 224, 224) val input = FloatArray(224 * 224 * 3) for (i in pixels.indices) { val color = pixels[i] val r = ((color shr 16) and 0xFF) / 255.0f val g = ((color shr 8) and 0xFF) / 255.0f val b = (color and 0xFF) / 255.0f input[i * 3] = r * 2f - 1f input[i * 3 + 1] = g * 2f - 1f input[i * 3 + 2] = b * 2f - 1f } return input } Channel order. TFLite models are usually NHWC: batch, height, width, channels (RGB). Android's Bitmap gives you ARGB. If you build the float array in the wrong order, your classifier "works" in tests with synthetic data and fails on real photos. Write the preprocessing once, and test it against a known input from the training script - if the Python notebook and the app produce different numbers for the same pixel, one of them is wrong. Step 6 - Run Inference and Postprocess With the interpreter loaded and the input prepared, running it is one call. The output handling depends on the task: - Classification: output is a probability vector. Take argmax for the label, and read the confidence - and decide a confidence threshold below which the app should say "uncertain" instead of guessing. val probs = classifier.predict(input) val maxIdx = probs.indices.maxByOrNull { probs[it] } ?: -1 val confidence = probs[maxIdx] if (confidence val text = visionText.text // text is plain String; feed it to your pipeline } .addOnFailureListener { e -> // handle failure; never leave the user with a spinner } ML Kit's on-device models are ready-made, sized sensibly, and frequently updated. The rule I use: if ML Kit covers your task, use it. Hand-roll TFLite only for custom models or unusual tasks. The first time you integrate a custom model, you will understand why this rule exists. Step 8 - Performance: Delegates, Threading, and Warm-Up On-device inference has three performance levers, and teams usually reach for the wrong one first. GPU and NNAPI delegates. The default CPU interpreter is slow for big models. The GPU delegate is a drop-in accelerator on most devices, and NNAPI delegates to vendor hardware (Qualcomm Hexagon, etc.): val options = Interpreter.Options() options.addDelegate(GpuDelegate()) The honest caveat: delegates are hardware-dependent and occasionally produce subtly different outputs than the CPU path. The right pattern is fallback - try GPU, catch the failure, retry with CPU: class FallbackInterpreter(model: MappedByteBuffer) { val interpreter: Interpreter init { val gpu = Interpreter.Options().apply { addDelegate(GpuDelegate()) } interpreter = try { Interpreter(model, gpu) } catch (e: RuntimeException) { Interpreter(model, Interpreter.Options()) // CPU fallback } } } Threading. options.numThreads = 4 helps on multi-core phones but can be slower on single-core devices. Benchmark on both classes of device before shipping the setting as a constant. Warm-up. The first inference call is always the slowest - model operators get JIT-tuned. Run a dummy inference once at startup (or lazily, on a background thread) and discard the result. My health app's first call was 250 ms; after warm-up, steady-state dropped to around 35 ms. That is the difference between a feature that feels instant and one that feels broken. Model size discipline. Every MB of model is MB of APK, and 200 MB+ APKs get rejected or see install drops. My default budget is under 15 MB for the model in most apps. Quantize to int8, and if the model is still too big, question the architecture - you do not need a 200 MB transformer on-device for a scre