Hand Activity Recognition: Literature Review

I am planning on starting my own project in Hand Activity Recognition. However before starting on this project I have conducted a literature review that is shown below.

Please note that most of the information given below are not mine. The source of the information is linked with the respective title.

Background Study

Activity recognition is one of the very popular projects that are done by different enthusiast. Activity recognition can be done using on-body sensors or using an external visual sensors (Microsoft Kinect). For this project we will be focusing mainly on the On-body sensors.

Current Market

1. SenseGlove 

• This glove offers Motion Tracking, Force Feedback and Tactile Feedback.
• Applications: Virtual Reality training, Augmented Process Simulation and Design Verification & Simulation

2. CyberGlove 2

• The 18-sensor data glove features two bend sensors on each finger, four abduction sensors, plus sensors measuring thumb crossover, palm arch, wrist flexion, and wrist abduction.
• The 22-sensor data glove has three flexion sensors per finger, four abduction sensors, a palm-arch sensor, and sensors to measure wrist flexion and abduction.
• Each sensor is extremely thin and flexible being virtually undetectable in the lightweight elastic glove.

3. Gest
Gest lets you work with your hands in a more intuitive way.

• Switch between apps just by twitching your finger.
• Point at your screen to move the mouse around.
• Twist your palm to adjust sliders in Photoshop.
• Rotate a 3D object by literally grabbing it and rotating your hand.

Available Projects

1. Intuitive Palm Controller - Roman Moskvichev

• The main idea is to create an intuitive controller that would replace modern-day input devices (mouse and keyboard) as well as console controllers and other devices.
• The palm built in controller would serve as a seamless and effortless way for the user to input data and interact with majority of devices through ­­­­­hand gesture and gest-commands.

2. A Sliding Window Approach to Natural Hand Gesture Recognition using a Custom Data Glove

Custom data glove, and scheme of sensor positions. 

(Luzhnica, Granit & Simon, Joerg & Lex, Elisabeth & Pammer, Viktoria. (2016). A Sliding Window Approach to Natural Hand Gesture Recognition using a Custom Data Glove. 10.1109/3DUI.2016.7460035.)

• An IMU & a pressure sensor are placed on each finger tip.
• Two bend sensors cover the two main joints of each finger.
• The thumb is special as it has 3 bend sensors.
• An IMU and a Magnetometer is on the top of the hand.
• An IMU is placed on the wrist & one bend sensor at the top and bottom of the wrist
• All sensors are connected to an Arduino board

3. Development of Artificial Hand Gripper for Rehabilitation Process

(Mohd Ali, Abdul & Y Ismail, M & Abdul Jamil, Muhammad Mahadi. (2011). Development of Artificial Hand Gripper for Rehabilitation Process. 10.1007/978-3-642-21729-6_192.)

• Development of a robotic hand that imitates the movement of a human hand. The basic movement of the surgeon hand was limited from a wrist, elbow and shoulder degree of freedom during an operation.
• The artificial hand gripper system requires sensors for a smooth and accurate movement. This allows large movement from the surgeon hand to be corrected on a small scale with a perfect incision and without any vibration.

4. Smart glove translates sign language gestures into text

• "The Language of Glove," a Bluetooth-enabled, sensor-packed glove that reads the sign language hand gestures and translates them into text.

5. Silver nanowire strain sensors for wearable body motion tracking

(Yao, Shanshan & Seok Lee, Jeong & James, K'Ehleyr & Miller, Jace & Narasimhan, Venkataramana & Joseph Dickerson, Andrew & Zhu, Xu & Zhu, Yong. (2015). Silver nanowire strain sensors for wearable body motion tracking. 1-4. 10.1109/ICSENS.2015.7370650.)

• demonstrates a wearable body motion tracking technology in the form of data glove to measure the instantaneous bending positions of individual finger knuckles.
• Attached to the glove is a highly stretchable and flexible silver nanowire (AgNW) based capacitive strain sensor which can adapt to curvilinear surfaces.
• The sensor shows a linear response to large tensile strain up to 60% with less than 5 msec response time.
• Applications: Virtual Reality, Gaming, Robot control

6. Novel smart sensor glove for arthritis rehabilitation

(O'Flynn, Brendan & Torres Sanchez, Javier & Connolly, James & Condell, Joan & Curran, Kevin & Gardiner, Philip. (2013). Novel smart sensor glove for arthritis rehabiliation. 1-6. 10.1109/BSN.2013.6575529.)

• Development of a smart glove to facilitate this rehabilitative process through the integration of sensors, processors and wireless technology to empirically measure ROM.
• The Tyndall/University of Ulster glove uses a combination of 20 bend sensors, 16 tri-axial accelerometers and 11 force sensors to detect joint movement.
• All sensors are placed on a flexible PCB to provide high levels of flexibility and sensor stability.
• The system operation means that the glove does not require calibration for each glove wearer

7. A Modular Data Glove System for Finger and Hand Motion Capture Based on Inertial Sensors

• This study proposes a modular data glove system to accurately and reliably capture hand kinematics.
• This data glove system’s modular design enhances its flexibility.
• It can provide the hand’s angular velocities, accelerations, and joint angles to physicians for adjusting rehabilitation treatments.
• The errors of the finger ROMs obtained from the fusion algorithm were less than 2°, proving that the fusion algorithm can measure the wearer’s range of motion accurately.

8. Sign Language Glove

• We designed and built a glove to be worn on the right hand that uses a Machine Learning (ML) algorithm to translate sign language into spoken English.
• Our device uses five Spectra Symbol Flex-Sensors that we use to quantify how much each finger is bent, and the MPU-6050 (a three-axis accelerometer and gyroscope) is able to detect the orientation and rotational movement of the hand.

9. Design of an Inertial-Sensor-Based Data Glove for Hand Function Evaluation

(Lin, Bor-Shing & Lee, I-Jung & Yang, Shu-Yu & Lo, Yi-Chiang & Lee, Junghsi & Chen, Jean-Lon. (2018). Design of an Inertial-Sensor-Based Data Glove for Hand Function Evaluation. Sensors. 18. 1545. 10.3390/s18051545.)

• Capturing hand motions for hand function evaluations is essential in the medical field.
• Various data gloves have been developed for rehabilitation and manual dexterity assessments.
• This study proposed a modular data glove with 9-axis inertial measurement units (IMUs) to obtain static and dynamic parameters during hand function evaluation.
• A sensor fusion algorithm is used to calculate the range of motion of joints.
• The data glove is designed to have low cost, easy wearability, and high reliability.

10. Modular finger and hand motion capturing system based on inertial and magnetic sensors

• A modular, ambulatory measurement system for the assessment of the remaining hand function and for closed-loop controlled therapy.
• The device is based on inertial sensors and utilizes up to five interchangeable sensor strips to achieve modularity and to simplify the sensor attachment.
• We introduce the modular hardware design and describe algorithms used to calculate the joint angles.
• Measurements with two experimental setups demonstrate the feasibility and the potential of such a tracking device

More Sources

• Activity Recognition. (https://en.wikipedia.org/wiki/Activity_recognition)
• A Review on Systems-Based Sensory Gloves for Sign Language Recognition State of the Art between 2007 and 2017. (https://www.ncbi.nlm.nih.gov/pubmed/29987266)
• Recognition of Human Activities using Machine Learning Methods with Wearable Sensors. (https://ieeexplore.ieee.org/document/7868369/)
• Human Activity Recognition Based on Wearable Sensor Data: A Standardization of the State-of-the-Art. (https://arxiv.org/abs/1806.05226)
• A Study on Multiple Wearable Sensors for Activity Recognition. (https://ieeexplore.ieee.org/document/8073827)
• Robust Human Activity Recognition Using Lesser Number of Wearable Sensors. (https://ieeexplore.ieee.org/document/8304292/)
• Design of Activity Recognition Systems with Wearable Sensors. (https://ieeexplore.ieee.org/document/8336752)
• Survey of Motion Tracking Methods Based on Inertial Sensors: A Focus on Upper Limb Human Motion. (https://www.researchgate.net/publication/317280180_Survey_of_Motion_Tracking_Methods_Based_on_Inertial_Sensors_A_Focus_on_Upper_Limb_Human_Motion)
• Physical Human Activity Recognition Using Wearable Sensors. (https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4721778/)
• Weakly Supervised Recognition of Daily Life Activities with Wearable Sensors (https://ieeexplore.ieee.org/document/5719639/)
• Co-Recognition of Human Activity and Sensor Location via Compressed Sensing in Wearable Body Sensor Networks. (https://ieeexplore.ieee.org/document/6200552)
• Human Daily and Sport Activity Recognition Using a Wearable Inertial Sensor Network (https://ieeexplore.ieee.org/document/8362913/all-figures)
• Optimal Low-power Workout State Classification for Multiple Configurations. (https://ieeexplore.ieee.org/document/8539521)