Controlling television with a gesture

Research by Sabrina Bilocca

Advancements in IT have made it possible to integrate computer systems into everyday life. However, cumbersome traditional devices like keyboards, mice, or the TV remote control, are inconvenient and unnatural to use. Hand gestures are a natural means of communication amongst humans. They are used to show feelings, emphasise points, and express thoughts and emotions. The use of hand gestures for humans to interact with computers should come as second nature to us.

Using this approach, Sabrina Bilocca (supervised by Prof. Adrian Muscat) developed a computer vision system which automatically recognises simple human hand gestures to control a TV. Bilocca’s research involved the recognition of isolated gestures as well as multiple gestures in continuous sequences. The greatest challenge was to recognise gestures in a series, since another procedure was needed to spot the key hand gestures while discarding unrelated movements between the key gestures.

The system was developed on a real-life dataset of isolated and a chain of hand gestures. The video samples were pre-processed using two algorithms (computer programmes that do a specific task, like separate the hand from the background) called ‘hand segmentation’ and ‘feature extraction’. The resulting images were input into a machine learning technique for training and testing that allows the computer to learn how to identify the hand gestures in a better way. Bilocca used the Hidden Markov Model as a machine learning. To efficiently spot and recognise key gestures in continuous sequences, Bilocca proposed and implemented an orientation backtracking technique. Using these techniques she achieved a 100% recognition rate for isolated gestures and an 87.8% recognition rate for gestures in a sequence. This compares with existing state-of-the-art techniques.

The project could be further developed as a real-time application to recognise hand gestures as they happen, which would help assist people with disability and the elderly. The skills Bilocca learnt in this project help her in her current position as a network engineer.

This research was performed as part of a Master of Science in Information and Communication Technology (Telecommunications) at the Faculty of Information and Communication Technology, University of Malta. . It was partially funded by STEPS (the Strategic Educational Pathways Scholarship—Malta). This scholarship was part-financed by the European Union—European Social Fund (ESF) under Operational Programme II—Cohesion Policy 2007–2013, ‘Empowering People for More Jobs and a Better Quality of Life’.