Real-Time Image Processing using Android NDK and OpenCV

In the previous recipes, we covered how to setup an NDK project and created an app to test the performance of using native code vs Kotlin. This is a fantastic start, but not particularly exciting. The best part of app development is that we get to build tools that give magic powers to our users. Let’s take a look at how Android NDK can help us do things that are not possible with standard Kotlin.

In this recipe, we’ll use OpenCV (full form Open Computer Vision) to apply a real-time image filter to a camera feed. This will introduce you to a popular third-party native library, show you how to use it in your Android app, and give you glimpse of how awesome it is to bring the power of the native C++ ecosystem to Android.

🛠 What You’ll Learn

✅ How to integrate OpenCV in an Android app

✅ How to process images on Android in real-time

📌 Step 1: Setup the Project

1. Download OpenCV Android SDK (e.g. opencv-4.11.0-android-sdk.zip) from OpenCV Releases and unzip it to a folder, such as ~/OpenCV-android-sdk.

2. Create a New Android App in Android Studio by selecting New Project → Empty Activity.

3. Name the project CartoonifyMe, the package name to com.example.cartoonifyme, set the Minimum SDK to 21, and click Finish.

📌 Step 2: Add OpenCV to Your Android App

1. Download OpenCV Android SDK from OpenCV Releases.

2. Extract it and import the extracted OpenCV-android-sdk/sdk folder into your project as a new module by clicking File → New → Import Module. Name the module opencv and finish the import.

3. Add the OpenCV dependency to your app module by opening the app/build.gradle.kts file and adding the following to the dependencies block.

implementation(project(":opencv"))

4. (optional) To make things a bit easier when it comes to working with the layout, let’s enable View Bindings for the app module by adding the following to the bottom of the android block:

buildFeatures {
    viewBinding = true
}

📌 Step 3: Add Permissions and Base Layout

1. Setup permissions by opening AndroidManifest.xml and adding the following:

<uses-permission android:name="android.permission.CAMERA" />
<uses-feature android:name="android.hardware.camera" android:required="true" />

2. Create the layout by opening res/layout/activity_main.xml and replacing its contents with

<?xml version="1.0" encoding="utf-8"?>
<RelativeLayout xmlns:android="http://schemas.android.com/apk/res/android"
    android:layout_width="match_parent"
    android:layout_height="match_parent">

    <org.opencv.android.JavaCamera2View
        android:id="@+id/camera_view"
        android:layout_width="match_parent"
        android:layout_height="match_parent" />

    <Button
        android:id="@+id/toggle_button"
        android:layout_width="wrap_content"
        android:layout_height="wrap_content"
        android:text="Toggle Cartoon"
        android:layout_marginBottom="60dp"
        android:layout_alignParentBottom="true"
        android:layout_centerHorizontal="true" />

</RelativeLayout>

📌 Step 4: Code the Main Activity

1. Open MainActivity.java and replace its contents with the following

package com.example.cartoonifyme

import android.os.Bundle
import android.util.Log
import androidx.appcompat.app.AppCompatActivity
import com.example.cartoonifyme.databinding.ActivityMainBinding
import org.opencv.android.CameraBridgeViewBase
import org.opencv.android.CameraBridgeViewBase.CvCameraViewListener2
import org.opencv.android.OpenCVLoader
import org.opencv.core.Core
import org.opencv.core.CvType
import org.opencv.core.Mat
import org.opencv.core.Size
import org.opencv.imgproc.Imgproc

class MainActivity : AppCompatActivity(), CvCameraViewListener2 {

    private lateinit var binding: ActivityMainBinding
    private var isCartoonOn = false
    private lateinit var lastFrame: Mat

    // Pre-allocate matrices for the cartoonify effect to avoid repeated allocation
    private lateinit var gray: Mat
    private lateinit var edges: Mat
    private lateinit var smoothed: Mat
    private lateinit var bgrFrame: Mat
    private lateinit var result: Mat

    private var processingScale = 0.75 // Process at 75% resolution for better performance

    override fun onCreate(savedInstanceState: Bundle?) {
        super.onCreate(savedInstanceState)
        binding = ActivityMainBinding.inflate(layoutInflater)
        setContentView(binding.root)

        binding.cameraView.setCvCameraViewListener(this)

        binding.toggleButton.setOnClickListener {
            isCartoonOn = !isCartoonOn
        }

        // Load OpenCV
        if (OpenCVLoader.initLocal()) {
            binding.cameraView.apply {
                setCameraPermissionGranted()
                enableView()
            }
            Log.i("NDKTest", "OpenCV initialized correctly")
        } else {
            Log.e("NDKTest", "Unable to initialize OpenCV")
        }
    }

    override fun onCameraViewStarted(width: Int, height: Int) {
        // Initialize all matrices with the camera resolution
        lastFrame = Mat()

        // Initialize matrices for cartoonify function
        val scaledWidth = (width * processingScale).toInt()
        val scaledHeight = (height * processingScale).toInt()

        gray = Mat(scaledHeight, scaledWidth, CvType.CV_8UC1)
        edges = Mat(scaledHeight, scaledWidth, CvType.CV_8UC1)
        bgrFrame = Mat(scaledHeight, scaledWidth, CvType.CV_8UC3)
        smoothed = Mat(scaledHeight, scaledWidth, CvType.CV_8UC3)
        result = Mat(height, width, CvType.CV_8UC4)
    }

    override fun onCameraViewStopped() {
        // Release all matrices
        lastFrame.release()
        gray.release()
        edges.release()
        bgrFrame.release()
        smoothed.release()
        result.release()
    }

    override fun onCameraFrame(inputFrame: CameraBridgeViewBase.CvCameraViewFrame): Mat {
        lastFrame = inputFrame.rgba()
        return if (isCartoonOn) cartoonify(lastFrame) else lastFrame
    }

    override fun onResume() {
        super.onResume()
        binding.cameraView.enableView()
    }

    override fun onPause() {
        super.onPause()
        binding.cameraView.disableView()
    }

    override fun onDestroy() {
        super.onDestroy()
        binding.cameraView.disableView()
    }
}

🚀 What This Code Does

• Implements a simple toggle button UI that lets users switch between normal and cartoon mode.

• Optimizes performance by processing frames at 75% resolution to reduce computational load.

• Pre-allocated all matrices (gray, edges, smoothed) to avoid expensive memory operations during processing.

• Properly manages resources by releasing all matrices when the camera view stops

📌 Step 5: Implement the Cartoon Effect in Kotlin

1. Open MainActivity.kt and add the cartoon processing logic below the onCameraFrame method using the following

private fun cartoonify(frame: Mat): Mat {
    val scaledInput = Mat()

    // Downscale for processing
    if (processingScale < 1.0) {
        Imgproc.resize(
            frame,
            scaledInput,
            Size(frame.width() * processingScale, frame.height() * processingScale),
            0.0, 0.0, Imgproc.INTER_LINEAR
        )
    } else {
        frame.copyTo(scaledInput)
    }

    // Convert to BGR (3-channel) for bilateral filter
    Imgproc.cvtColor(scaledInput, bgrFrame, Imgproc.COLOR_RGBA2BGR)

    // Convert to grayscale
    Imgproc.cvtColor(scaledInput, gray, Imgproc.COLOR_RGBA2GRAY)

    // Detect edges - use smaller kernel sizes for better performance
    Imgproc.medianBlur(gray, gray, 5) // Reduced from 7 to 5
    Imgproc.adaptiveThreshold(
        gray, edges, 255.0, Imgproc.ADAPTIVE_THRESH_MEAN_C,
        Imgproc.THRESH_BINARY, 7, 2.0  // Reduced from 9 to 7
    )

    // Smooth colors - use faster parameters for bilateral filter
    // Reduced d from 9 to 7, and sigma values from 75.0 to 50.0
    Imgproc.bilateralFilter(bgrFrame, smoothed, 7, 50.0, 50.0)

    // Convert back to RGBA
    val edgesRgba = Mat()
    Imgproc.cvtColor(edges, edgesRgba, Imgproc.COLOR_GRAY2RGBA)
    Imgproc.cvtColor(smoothed, smoothed, Imgproc.COLOR_BGR2RGBA)

    // Combine edges and smoothed image
    val tempResult = Mat()
    Core.bitwise_and(smoothed, edgesRgba, tempResult)

    // Upscale result to original size if needed
    if (processingScale < 1.0) {
        Imgproc.resize(
            tempResult,
            result,
            frame.size(),
            0.0, 0.0, Imgproc.INTER_LINEAR
        )
    } else {
        tempResult.copyTo(result)
    }

    // Clean up temp matrices
    scaledInput.release()
    edgesRgba.release()
    tempResult.release()

    return result
}

🚀 What This Code Does

• Downscales input frames to 75% resolution using Imgproc.resize for faster processing.
• Processes the image through color conversions, median blur, and adaptive thresholding to create cartoon-style edges.
• Applies bilateral filtering with optimized parameters to smooth colors while preserving edges.
• Combines the smoothed colors with edge detection using Core.bitwise_and to create the final cartoon effect.
• Upscales the result back to original size and properly releases all temporary matrices to prevent memory leaks.

📌 Step 6: Run the App

1. Install the app on a real device (emulators don’t support camera access).

2. Allow camera permissions for our app via Android Settings

3. Open the app

3. See the live edge-detection filter applied in real-time!

📌 Expected Output

When you point the camera at objects, you should see images from the camera feed filtered just like a cartoon sketch!

Closing Remarks

The source code for this recipe can be found at https://github.com/advaitsaravade/Android-NDK-3-OpenCV.

Happy coding!