There are many advantages of gesture control over capacitive touch control and traditional control devices such as buttons and knobs. When wearing gloves, capacitive touch panels will not work, but Infra-Red (IR) gesture control will. While both sensing technologies work in two dimensions on a flat plane, gesture control adds a third dimension: users can move their hand towards and away from the sensor.
When safety might be compromised by requiring a user to find a knob or push a specific button, gesture control allows a user to make these changes in a wide active field in which gross movements can replace finer movements. In addition, simple, more natural hand movements can replace less intuitive control technology. For example, a radio channel can be changed with a simple left or right swipe of a hand. Volume can be controlled by an up or down swipe or a push in or out. Finally, no stylus or input device is required other than the hand.
Devices such as the Microsoft® Kinect®, the Nintendo® Wii and the Leap Motion gesture device are the most sophisticated gesture-control devices available today. But because they use lasers, cameras and complex photo-detectors, these devices tend to be very expensive and software-intense compared to systems that use reflected IR light to detect the movement of the user’s hand.
For designers wishing to experiment with IR-based gesture control, Vishay has introduced its Gesture Control Sensor Board, as shown in Figure 1, featuring the VCNL4020 IR light sensor. The board enables users to quickly prove a design concept and to start initial hardware and software gesture-control development. This Design Note explains the technology’s operation.
The heart of Vishay’s Gesture Control Sensor Board is the VCNL4020 proximity and ambient light sensor. Two of Vishay’s VSMF2890RGX01 IR diodes are mounted on either side of the sensor. They are mounted on the opposite side of the PCB from the VCNL4020 sensor, to minimise cross-talk between the diodes and the sensor.
The emitters’ high radiant intensity of typically 80mW/sr at 200mA allows for the detection of hand gestures as much as 25cm above the sensor board, as shown in see Figure 2.
Operating principles of IR gesture control
The detection of gestures using the Gesture Control Sensor Board is accomplished by comparing the IR light signal coming from each of the emitters. The emitted IR light is reflected from an object, for example a hand, and then detected by the VCNL4020 proximity sensor. In order to differentiate between the signals coming from each emitter, the emitters are multiplexed, meaning they are pulsed one after the other in quick succession.
The proximity signal is then read out between each pulse via an I2C bus interface. When a hand is in the proximity of the board, it will reflect more signal from the emitter it is directly located over. If the hand is then moved across the board, the signal from one emitter will increase before the other. This time difference of signal strength is analysed to determine if a swipe gesture was made and in which direction.
How a gesture is recognised
The detection of a gesture is performed in two stages: the acquisition and preparation of the raw data to be analysed; and the interpretation of the raw data by the detection algorithm. If a gesture is recognised, a corresponding event is triggered.
The data used for gesture recognition are acquired in separate data streams, one per emitter. As the emitters are multiplexed, the measurements occur one after the other. One measurement cycle consists of a set of data from each channel. The measurements are made in quick succession so that the time required to complete a full measurement cycle is much greater than the time between the individual proximity measurements of each emitter. In this way the signals can be compared with one another in close to real time.
If the jumps between measurements are too erratic, for example if the sensor is in a noisy environment, several measurements per emitter and per measurement cycle can be made and averaged. This will slow down the measurement speed, but will result in a cleaner signal.
To be able to analyse and compare the data streams of each emitter channel, the data of each stream are split into frames of ‘n’ measurements. Here, each frame contains the latest n measurements, with each iteration of the measurement cycle moving the frame over by one. The recognition algorithm analyses each frame, and once a frame is seen to contain a gesture, the next few frames are ignored, in order to avoid detecting the same gesture twice, as shown in Figure 3.
Furthermore, by ignoring several frames after a valid gesture has been detected, it is possible to reduce false positives from hand movements in the sensor’s field of view that were not intended as a gesture.
Each frame of the acquired signal is analysed for two parameters: the standard deviation of each signal, and the time delay between signals.
By comparing the results of this analysis to user-defined thresholds, the algorithm can tell whether a gesture was made, and what kind of a gesture it was.
The standard deviation is a measure of the spread of the data within the frame under analysis. It is calculated using the following formula:
x = the mean of the current frame.
n = the number of samples under analysis.
A low standard deviation implies there is no change in the signal and there is either no hand in the sensor’s detection area or the hand is held steady over the sensor and no swipe or push gesture is being made. A high standard deviation implies a large change in the signal, suggesting the movement of a hand across or towards the sensor. The detection algorithm only analyses the frame for further parameters if the signal is above a set standard-deviation threshold.
The presence of a sufficient time delay between the signals signifies that a swipe gesture has been made. In the detection algorithm implemented in the demo kit, the time delay is found by measuring the cross-correlation between the two signals. The cross correlation can be calculated by the following formula:
f[x] and g[x] = individual data streams from either emitter.
f* = the conjugate of f.
The cross-correlation effectively shifts one of the signals over the other and computes the percentage overlay of their integrals. By taking note of the point at which there is a maximum cross-correlation between the two signals, an estimation of the time delay between the two signals can be found. As the cross-correlation of the signals is always computed in the same order, regardless of which of the signals comes first, the time delay will either be positive or negative. The sign of the time delay denotes the direction of the swipe gesture. The decision on whether a right or left swipe was detected is only made when this time delay threshold is exceeded.
Both the time delay threshold and the standard deviation threshold can be adjusted in the Vishay sensor board’s software, allowing designers to fine-tune the software’s operation for their application.
The gesture-control demo software is supplied free by Vishay with the VCNL4020 Gesture Control Sensor Board.