Skip to content

About

Real-time sign language recognition on Android: MediaPipe hand landmarks + a PyTorch classifier running on-device with ONNX Runtime. 3rd place worldwide, Huawei ICT Competition.

Topics

Resources

Stars

1 star

Watchers

0 watching

Forks

Latest commit

 

History

1 Commit

Folders and files

NameName
Last commit message
Last commit date
 
 
 
 

Repository files navigation

Conversign — Real-time sign language recognition. A hand skeleton made of 21 landmarks with red joints.

3rd place worldwide, Huawei ICT Competition · Teknofest 2025 finalist

Conversign is an Android app that translates sign language gestures into text in real time, using the phone's camera. Hand tracking and classification both run on the device, so recognition works without sending video to a server.

Available on Huawei AppGallery.

This is a showcase repository. The source code is kept in a private repository; this page describes what the app does and how it is built.

What it does

  • Reads the live camera feed and finds the hand in each frame.
  • Extracts 21 hand landmarks with MediaPipe.
  • Classifies the gesture with a model trained in PyTorch and exported to ONNX.
  • Shows the recognized sign as text in the UI.

How it works

flowchart LR
    A[Camera frame] --> B[MediaPipe<br/>21 hand landmarks]
    B --> C[ONNX Runtime<br/>gesture classifier]
    C --> D[ViewModel<br/>UI state]
    D --> E[Jetpack Compose<br/>text on screen]
Loading

The model is trained offline in PyTorch on collected gesture data, exported to ONNX, and bundled with the app. At runtime, ONNX Runtime runs inference on the landmark coordinates rather than raw pixels, which keeps the model small enough for a phone.

Architecture

The app follows Clean Architecture with MVVM:

Layer Responsibility
Presentation Jetpack Compose screens and ViewModels that expose UI state
Domain Use cases for recognizing a gesture from a set of landmarks
Data Camera input, MediaPipe hand landmark detection, ONNX Runtime inference

The first version was written in Flutter. It was rewritten in native Kotlin and Jetpack Compose to fix compatibility issues on Huawei devices.

Tech stack

Kotlin · Jetpack Compose · MVVM · Clean Architecture · MediaPipe · PyTorch · ONNX Runtime

Running it

The source code is not public, so the app cannot be built from this repository. To try Conversign, install it from Huawei AppGallery.

Contact

Ulaş Uçan · LinkedIn · uls.ucn@gmail.com

About

Real-time sign language recognition on Android: MediaPipe hand landmarks + a PyTorch classifier running on-device with ONNX Runtime. 3rd place worldwide, Huawei ICT Competition.

Topics

Resources

Stars

1 star

Watchers

0 watching

Forks

Releases

Packages

Contributors