Module src.recognition.gesture_classifier

Classes

class GestureClassifier (hand_landmarks: mediapipe.framework.formats.landmark_pb2.NormalizedLandmarkList)

Gesture Classifier

Gesture Classifier initializes with a list of 21 landmarks belonging to one hand. It predicts a gesture from these landmarks by labeling each finger as either open (True) or closed (False).

Finger 'openness' is estimated the following way: A given finger has three joints. The algorithm sums up the angles for these joints, and compares them to a threshold. If the accumulated angles are greater then the threshold, the finger is considered closed. Otherwise it is open.

Thumb 'openness' is a special case where both positive and negative values are considered. The sign of an angle is determined by estimating a normal vector for the palm, and using this as an orientation vector. The angle of the joint is then calculated through:

$$\sin\theta={(\boldsymbol{n}\times\boldsymbol{u})\cdot\boldsymbol{v}\over |u||v|}.$$ $$\theta = \begin{cases} \arccos{\frac{u\cdot v}{|u||v|}}, & \text{if $\sin\theta\ge0$;}\\ 2\pi-\arccos{\frac{u \cdot v}{|u||v|}}, & \text{if $\sin\theta<0$.}\\ \end{cases}$$

Where $\theta$ is the joint angle, $\boldsymbol{n}$ is the estimated palm normal, and $\boldsymbol{u}$ and $\boldsymbol{v}$ are the vectors extending from the joint. The thumb has its separate threshold for 'openness', defined in GestureClassifier_THUMB_ANG_TRESHOLD

Methods

def estimate(self) ‑> str

Estimate name of gesture

Checks finger states and tries to find a matching gesture in the list of defined gestures. If a matching gesture is found it returns that gestures name. If not it returns a fallback string representing the state of each finger.

Returns

Label of predicted gesture