Module src.streaming.frame_processor

Module for processing individual frames

Classes

class FrameProcessor (num_hands: int, res: Tuple[int, int], annotate: bool, confidence_settings: Tuple[int, int], volume_mixer=None)

FrameProcessor manages state and processes frame in the class context.

A FrameProcessor object starts in the Idle state. From this it can enter several different states, which effect what gestures will be detected, and how they in turn affect the camera session.

Camera settings (crop, zoom, exposure) is managed with a CamController class member. These settings are manipulated through interacting with gestures.

For more info on how to interact with this class through gestures, see the user manual (README.md).

Args

num_hands
Max number of hands for inference.
res
Resolution (width, height).
annotate
Whether to draw annotations over images.
volume_mixer
Alsaaudio volume mixer object.

Methods

def process_frame(self, frame: numpy.ndarray) ‑> numpy.ndarray

Process a video frame in the current object context.

Processes a frame in accordance with the current state. This involves running inference for gestures and acting correspondingly. The class also manages current camera settings, i.e. zoom, cropping, volume settings, etc. See the user manual (README.md) for more info.

Args

frame
Image as a numpy array.

Returns

A new processed image as a numpy array. Can for instance be zoomed, cropped, contain drawn annotations etc.

class FrameProcessorState

Abstract base class for FrameProcessor State.

The states that inherit this class specify how the program responds to gesture interaction. Different states allow for different gestures and responses. We recommend reading the user manual, (README.md) for a better grasp of how this works.

Note: For all inherited objects, the user can go to the idle state by performing the corresponding gesture.

Subclasses

Methods

def process(self, fp: FrameProcessor, frame: numpy.ndarray) ‑> NoneType

Processes a video frame in the object state.

Args

fp
The parent FrameProcessor object that controls the current session.
frame
Image as a numpy array.
class Idle

Idle state

In this state, the program looks for gestures indicating that the user intends to enter either Gesture or Dynamic state.

Ancestors

Inherited members

class Gesture

Gesture state

In this state, the program is open for interaction through gestures. The user can zoom, pan or crop the camera, reset the settings or enter any of the other states.

Ancestors

Inherited members

class RangeGesture (point, filled_to, orientation)

RangeGesture state

In this state, the program renders a interactive slider that can used to set session settings. The slider orientation is dependant on where it was created, left side of the screen -> horizontal, right side -> vertical. The user can exit this state by releasing the fist gesture, which keeps the value setting. Alternatively, they can use the reset gesture to reset the session. Both options enter Gesture state.

Ancestors

Inherited members

class DynamicGesture

EXPERIMENTAL: DynamicGesture state

In this state, the program looks for dynamic gestures. Currently, the only support is for (somewhat slow) hand waving. A recognized wave will render "Left/Right wave!" to the screen. Does not currently control any functionality.

The user can exit this state either by entering Gesture or Idle state, with their corresponding gestures.

Ancestors

Inherited members