Vision-based Therapeutic System Involving Finger Exercise

Abstract

This work aims to create a system that will allow patients to receive physiotherapy at home. We focus on two key parts, a vision based approach, and rehabilitation therapy primarily for stroke patients with upper limb disability through finger exercises. Our hardware consists of only using a HD webcam or a camera. For our hand detection, we are using three deep learning models. The first model is trained using Google’s Mediapipe hands to find the hand landmark from an image, and using the landmarks to recognize the therapeutic hand gestures. To train our model, we also collected an extensive dataset for the pinching finger exercises with different environmental conditions, distances from camera and orientations. The data we collected have been used to create 3 deep learning models capable of detecting therapeutic hand gestures from real-time video feeds. In this report, we have discussed the methodologies we used for data collection, the types of models we built and their architecture along with their strong points and shortcomings. Lastly, we elaborated on the future works that we intend to perform.