Face Landmark#

On-device AI Face Landmark App with ZETIC.MLange

Github Repository#

We provide Face Landmark demo application source code for both Android and iOS. repository

What is Face Landmark#

The Face Landmark model in Google’s MediaPipe is a highly efficient machine learning model used for real-time face detection and landmark extraction.
Face Landmark Google AI Document : link

Model pipelining#

For accurate use of the face landmark model, it is necessary to pass an image of the correct facial area to the model. To accomplish this, we construct a pipeline with the face detection Model.

Face Detection: we use the Face Detection Model to accurately detect the face regions in the image. Using the information from the detected face region, we extract that part of the original image.
Face Landmark: Input the extracted face image into the Face Landmark model to analyze facial landmarks.

Step-by-step implementation#

0. Prerequisites#

Prepare the model Face Detection and Face Landmark from github.

Face Detection: Convert the Tensorflow model to the TorchScript model.

$ pip install tf2onnx
$ python -m tf2onnx.convert --tflite face_detection_short_range.tflite --output face_detection_short_range.onnx --opset 13

Face Landmark: Convert the Tensorflow model to the TorchScript model.

$ pip install tf2onnx
$ python -m tf2onnx.convert --tflite face_landmark.tflite --output face_landmark.onnx --opset 13

1. Generate ZETIC.MLange model#

Get your own MLange model key from the model

If you want to get your own model key, please get your own model key as below.

    # (1) Get mlange_gen
    $ wget https://github.com/zetic-ai/ZETIC_MLange_document/raw/main/bin/mlange_gen && chmod 755 mlange_gen

    # (2) Run mlange_gen for two models
    #    - Face detection model
    $ ./mlange_gen -m face_detection_short_range.onnx -i input.npy

    #    - Face landmark model
    $ ./mlange_gen -m face_landmark.onnx -i input.npy

Expected output

    ...
    MLange Model Key : {YOUR_FACE_DETECTION_MODEL_KEY}
    ...

    ...
    MLange Model Key : {YOUR_FACE_LANDMARK_MODEL_KEY}
    ...

2. Implement ZeticMLangeModel with your model key#

We prepared a model key for the demo app: face_detection and face_landmark. You can use the model key to try the Zetic.MLange Application.

Anroid app

For the detailed application setup, please follow deploy to Android Studio page
ZETIC.MLange usage in Kotlin

  val faceLandmarkModel = ZeticMLangeModel(this, 'face_landmark')

  faceLandmarkModel.run(inputs)

  val outputs = faceLandmarkModel.outputBuffers

iOS app

For the detailed application setup, please follow deploy to XCode page
ZETIC.MLange usage in Swift

  let faceLandmarkModel = ZeticMLangeModel('face_landmark')

  faceLandmarkModel.run(inputs)

  let outputs = faceLandmarkModel.getOutputDataArray()

3. Prepare Face Landmark image feature extractor for Android and iOS#

We provide a Face Landmark feature extractor as an Android and iOS module.
- (The Face Landmark feature extractor extension will be exposed as an open-source repository soon.)
- You can use your own feature extractor if you have one for Face Landmark usage

For Android

// (0) Initialize ZeticMLangeFeatureFaceLandmark
val feature = ZeticMLangeFeatureFaceLandmark()

// (1) Preprocess bitmap and get processed float array
val inputs = feature.preprocess(bitmap)

...

// (2) Postprocess to bitmap
val resultBitmap = feature.postprocess(outputs)

For iOS

import ZeticMLange

// (0) Initialize ZeticMLangeFeatureFaceLandmark
let feature = ZeticMLangeFeatureFaceLandmark()

// (1) Preprocess UIImage and get processed float array
let inputs = feature.preprocess(image)

...

// (2) Postprocess to UIImage
let resultBitmap = feature.postprocess(&outputs)

Total Face Landmark Process implementation#

Pipelining two models.

For Android

Kotlin

Face Detection Model

// (0) Initialization Models
val faceDetectionModel = ZeticMLangeModel(this, 'face_detection')

// (1) Initialization Feature
val faceDetectionFeature = ZeticMLangeFeatureFaceDetection()

// (2) Preprocess Image
val faceDetectionInputs = faceDetectionFeature.preprocess(bitmap)

// (3) Process Model
faceDetectionModel.run(faceDetectionInputs)
val faceDetectionOutputs = faceDetectionModel.outputBuffers

// (4) Postprocess model run result
val faceDetectionPostprocessed = faceDetectionFeature.postprocess(faceDetectionOutputs)

Face Landmark Model Pass the result of face detection model as a input.

// (0) Initialization Models
val faceLandmarkModel = ZeticMLangeModel(this, 'face_landmark')

// (1) Initialization Feature
val faceLandmarkFeature = ZeticMLangeFeatureFaceLandmark()

// (2) Preprocess Image
val faceLandmarkInputs = faceLandmarkFeature.preprocess(bitmap, faceDetectionPostprocessed)

// (3) Process Model
faceLandmarkModel.run(faceLandmarkInputs)
val faceLandmarkOutputs = faceLandmarkModel.outputBuffers

// (4) Postprocess model run result
val faceLandmarkPostprocessed = faceLandmarkFeature.postprocess(faceLandmarkOutputs)