Getting Frames of Live camera footage as Bitmaps in Android using Camera2 API Kotlin

7 min readApr 13, 2021

Learn the use of machine learning and computer vision in Android, Flutter & React Native with our Mobile Machine Learning courses. You can avail discount on following Android Machine learning courses

Face Recognition in Android — Build Attendance Systems

Train Object Detection Models & build Android Applications

ChatGPT & Android — Build Chatbots & Smart Apps for Android

Android & Regression: Train Prediction ML models for Android

Note: we will be using Kotlin in this story for Java click here.You can also watch video lecture of story here on youtube

There are always some things that we think are difficult to understand but in reality, we are not looking at those things from the right angle.

One such thing for Android developers is to display live camera footage inside their applications and getting frames of live footage one by one and using them for particular purposes like passing them to machine learning models and stuff like that.

So here I will teach you to add that live camera footage in your application in a really simple way.

So firstly create a new Android studio Application using either Kotlin.

Permissions

After that inside the manifest file paste these lines to get Camera permissions because we want to access the camera.

<uses-permission android:name="android.permission.CAMERA" />
<uses-feature android:name="android.hardware.camera" />
<uses-feature android:name="android.hardware.camera.autofocus" />

3: Then inside the onCreate method of the Activity class ask for permission dynamically

//TODO ask for permission of camera upon first launch of application
if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.M) {
    if (checkSelfPermission(Manifest.permission.CAMERA) == PackageManager.PERMISSION_DENIED
    ) {
        val permission = arrayOf(
                Manifest.permission.CAMERA
              
        )
        requestPermissions(permission, 1122)
    } else {
        //TODO show live camera footage
        
    }
}else{
    //TODO show live camera footage
    
}

And paste this method below onCreate

override fun onRequestPermissionsResult(requestCode: Int, permissions: Array<String?>, grantResults: IntArray) {
    super.onRequestPermissionsResult(requestCode, permissions, grantResults)
    //TODO show live camera footage
    if (grantResults.isNotEmpty() && grantResults[0] == PackageManager.PERMISSION_GRANTED) {
        //TODO show live camera footage
    } else {
        finish()
    }
}

After that, you need to copy three code files to your Android studio project

CameraConnectionFragment: Contain the code related to Camera2 API

AutoFitTextureView: A class extending TextureView which s used to render camera preview

ImageUtils: A utility class that will help us to convert frames of live camera footage to bitmaps.

Layout

Now in the layout folder create a file named camera_fragment.xml and paste this code inside it

<RelativeLayout xmlns:android="http://schemas.android.com/apk/res/android"
    android:layout_width="match_parent"
    android:layout_height="match_parent">    <com.example.imageclassificationlivefeed.AutoFitTextureView
        android:id="@+id/texture"
        android:layout_width="wrap_content"
        android:layout_height="wrap_content"
        android:layout_alignParentTop="true" />
</RelativeLayout>

Make sure you change com.example.imageclassificationlivefeed with your application package name.

There you can see we are using that AutoFitTextureView instead of TextureView because it can adapt to different aspect ratios.

Then inside the layout file of your activity where you want to display the live camera footage add a FrameLayout which will be replaced with CameraConnectionFragment later. So in our case inside activity_main.xml add paste below code

<FrameLayout
    android:id="@+id/container"
    android:layout_width="match_parent"
    android:layout_height="wrap_content"
    android:background="@android:color/black"
    app:layout_constraintEnd_toEndOf="parent"
    app:layout_constraintHorizontal_bias="0.0"
    app:layout_constraintStart_toStartOf="parent"
    app:layout_constraintTop_toTopOf="parent"
    ></FrameLayout>

Showing live camera footage

Now inside our MainActivity.kt file, we need to add the code to replace Framelayout with CameraConnectionFragment.

So just below onCreate method of activity class paste this setFrament method.

var previewHeight = 0;
var previewWidth = 0
var sensorOrientation = 0;
//TODO fragment which show llive footage from camera
protected fun setFragment() {
    val manager =
            getSystemService(Context.CAMERA_SERVICE) as CameraManager
    var cameraId: String? = null
    try {
        cameraId = manager.cameraIdList[0]
    } catch (e: CameraAccessException) {
        e.printStackTrace()
    }
    val fragment: Fragment
    val camera2Fragment = CameraConnectionFragment.newInstance(
            object :
                    CameraConnectionFragment.ConnectionCallback {
                override fun onPreviewSizeChosen(size: Size?, rotation: Int) {
                    previewHeight = size!!.height
                    previewWidth = size.width
                    sensorOrientation = rotation - getScreenOrientation()
                }
            },
            this,
            R.layout.camera_fragment,
            Size(640, 480)
    )
    camera2Fragment.setCamera(cameraId)
    fragment = camera2Fragment
    fragmentManager.beginTransaction().replace(R.id.container, fragment).commit()
}
protected fun getScreenOrientation(): Int {
    return when (windowManager.defaultDisplay.rotation) {
        Surface.ROTATION_270 -> 270
        Surface.ROTATION_180 -> 180
        Surface.ROTATION_90 -> 90
        else -> 0
    }
}

Inside this method, you can see that was replacing Framelayout(with id container) with the object of CameraConnectionFragment class.

Now inside the onCreate method and inside onRequestPermissionsResult method just call the setFragment function.

So onCreate method should look like that

override fun onCreate(savedInstanceState: Bundle?) {
    super.onCreate(savedInstanceState)
    setContentView(R.layout.activity_main)

    //TODO ask for permission of camera upon first launch of application
    if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.M) {
        if (checkSelfPermission(Manifest.permission.CAMERA) == PackageManager.PERMISSION_DENIED
        ) {
            val permission = arrayOf(
                    Manifest.permission.CAMERA
            )
            requestPermissions(permission, 1122)
        } else {
            //TODO show live camera footage
            setFragment()
        }
    }else{
        //TODO show live camera footage
        setFragment()
    }


}

And onRequestPermissionsResult will look like that

override fun onRequestPermissionsResult(
        requestCode: Int,
        permissions: Array<String?>,
        grantResults: IntArray
) {
    super.onRequestPermissionsResult(requestCode, permissions, grantResults)
    //TODO show live camera footage
    if (grantResults.isNotEmpty() && grantResults[0] == PackageManager.PERMISSION_GRANTED) {
        //TODO show live camera footage
        setFragment()
    } else {
        finish()
    }
}

Now inside MainActivity.kt class implement an Interface ImageReader.OnImageAvailableListener .That interface contains a method onImageAvailable. So add this method aswell.

//TODO getting frames of live camera footage and passing them to model
override fun onImageAvailable(reader: ImageReader) {
    reader.acquireLatestImage().close()
}

We are implementing that interface because inside CameraConnectionFragment we are passing it to get Frames of live footage. So for each frame of live camera footage, onImageAvailable method will be called and we can get that frame and use it for a variety of purposes.

So after doing that your main activity will look like that.

import android.Manifest
import android.app.Fragment
import android.content.Context
import android.content.pm.PackageManager
import android.hardware.camera2.CameraAccessException
import android.hardware.camera2.CameraManager
import android.media.ImageReader
import android.os.Build
import androidx.appcompat.app.AppCompatActivity
import android.os.Bundle
import android.util.Size
import android.view.Surface
import com.example.imageclassificationlivefeed.CameraConnectionFragment

class MainActivity : AppCompatActivity(), ImageReader.OnImageAvailableListener {
    override fun onCreate(savedInstanceState: Bundle?) {
        super.onCreate(savedInstanceState)
        setContentView(R.layout.activity_main)

        //TODO ask for permission of camera upon first launch of application
        if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.M) {
            if (checkSelfPermission(Manifest.permission.CAMERA) == PackageManager.PERMISSION_DENIED
            ) {
                val permission = arrayOf(
                        Manifest.permission.CAMERA,
                        Manifest.permission.WRITE_EXTERNAL_STORAGE
                )
                requestPermissions(permission, 1122)
            } else {
                //TODO show live camera footage
                setFragment()
            }
        }else{
            //TODO show live camera footage
            setFragment()
        }


    }

    override fun onRequestPermissionsResult(
            requestCode: Int,
            permissions: Array<String?>,
            grantResults: IntArray
    ) {
        super.onRequestPermissionsResult(requestCode, permissions, grantResults)
        //TODO show live camera footage
        if (grantResults.isNotEmpty() && grantResults[0] == PackageManager.PERMISSION_GRANTED) {
            //TODO show live camera footage
            setFragment()
        } else {
            finish()
        }
    }
    var previewHeight = 0;
    var previewWidth = 0
    var sensorOrientation = 0;
    //TODO fragment which show llive footage from camera
    protected fun setFragment() {
        val manager =
                getSystemService(Context.CAMERA_SERVICE) as CameraManager
        var cameraId: String? = null
        try {
            cameraId = manager.cameraIdList[0]
        } catch (e: CameraAccessException) {
            e.printStackTrace()
        }
        val fragment: Fragment
        val camera2Fragment = CameraConnectionFragment.newInstance(
                object :
                        CameraConnectionFragment.ConnectionCallback {
                    override fun onPreviewSizeChosen(size: Size?, rotation: Int) {
                        previewHeight = size!!.height
                        previewWidth = size.width
                        sensorOrientation = rotation - getScreenOrientation()
                    }
                },
                this,
                R.layout.camera_fragment,
                Size(640, 480)
        )
        camera2Fragment.setCamera(cameraId)
        fragment = camera2Fragment
        fragmentManager.beginTransaction().replace(R.id.container, fragment).commit()
    }

    protected fun getScreenOrientation(): Int {
        return when (windowManager.defaultDisplay.rotation) {
            Surface.ROTATION_270 -> 270
            Surface.ROTATION_180 -> 180
            Surface.ROTATION_90 -> 90
            else -> 0
        }
    }

    //TODO getting frames of live camera footage and passing them to model
    override fun onImageAvailable(reader: ImageReader) {
        reader.acquireLatestImage().close()
    }
}

And that’s it. Now when you will run your Application Live camera footage will be displayed inside the main activity..

Converting frames into Bitmaps

Now we displayed live camera footage inside our Android application and we are getting frames of that live camera footage. But mostly we deal with images in Bitmap format. So let’s convert those frames to bitmap.

So replace onImageAvailable method with this code

//TODO getting frames of live camera footage and passing them to model
private var isProcessingFrame = false
private val yuvBytes = arrayOfNulls<ByteArray>(3)
private var rgbBytes: IntArray? = null
private var yRowStride = 0
private var postInferenceCallback: Runnable? = null
private var imageConverter: Runnable? = null
private var rgbFrameBitmap: Bitmap? = null
override fun onImageAvailable(reader: ImageReader) {
    // We need wait until we have some size from onPreviewSizeChosen
    if (previewWidth == 0 || previewHeight == 0) {
        return
    }
    if (rgbBytes == null) {
        rgbBytes = IntArray(previewWidth * previewHeight)
    }
    try {
        val image = reader.acquireLatestImage() ?: return
        if (isProcessingFrame) {
            image.close()
            return
        }
        isProcessingFrame = true
        val planes = image.planes
        fillBytes(planes, yuvBytes)
        yRowStride = planes[0].rowStride
        val uvRowStride = planes[1].rowStride
        val uvPixelStride = planes[1].pixelStride
        imageConverter = Runnable {
            ImageUtils.convertYUV420ToARGB8888(
                    yuvBytes[0]!!,
                    yuvBytes[1]!!,
                    yuvBytes[2]!!,
                    previewWidth,
                    previewHeight,
                    yRowStride,
                    uvRowStride,
                    uvPixelStride,
                    rgbBytes!!
            )
        }
        postInferenceCallback = Runnable {
            image.close()
            isProcessingFrame = false
        }
        processImage()
    } catch (e: Exception) {
        return
    }
}


private fun processImage() {
    imageConverter!!.run()
    rgbFrameBitmap = Bitmap.createBitmap(previewWidth, previewHeight, Bitmap.Config.ARGB_8888)
    rgbFrameBitmap?.setPixels(rgbBytes, 0, previewWidth, 0, 0, previewWidth, previewHeight)
    postInferenceCallback!!.run()
}

protected fun fillBytes(
        planes: Array<Image.Plane>,
        yuvBytes: Array<ByteArray?>
) {
    // Because of the variable row stride it's not possible to know in
    // advance the actual necessary dimensions of the yuv planes.
    for (i in planes.indices) {
        val buffer = planes[i].buffer
        if (yuvBytes[i] == null) {
            yuvBytes[i] = ByteArray(buffer.capacity())
        }
        buffer[yuvBytes[i]]
    }
}

So now when a particular frame onImageAvailable method is called we will check if the isProcessingFrame variable is true or not. If that variable is true its means that processing of the previous frame is not completed yet so we will return from there. Otherwise, we will process that particular frame.

So to convert that frame into a bitmap we will firstly get planes of that frame then we will get the bytes. Then finally we will convert those bytes into Bitmap format. So you can see the code related to that in processImage method.

So now we have the frame from live camera footage in Bitmap format stored in a variable name rgbFrameBitmap. So you can perform any operation on that bitmap like you can pass it to machine learning model or perform other operations. But once you will finish the processing of that particular frame you need to call postInferenceCallback.run() method so that the isProcessingFrame variable will be set to false and the next frame will be passed for processing.

So above code will not just give you frames of live camera footage but you will get frames one by one. So once the processing of one frame will be completed the next frame will be passed for processing.

Mobile Machine Learning

Learn the use of machine learning and computer vision in Android, Flutter & React Native with our Mobile Machine Learning courses. You can avail discount on the following Mobile Machine learning courses