Can You Enhance That?

Cop shows have taught us that grainy photos of a crime scene usually contain clues to a killer in action. By zooming in or running a program, investigators are able to catch critical clues to help in their investigation. In reality, if one zooms in on a poor-quality image, one simply gets pixels. A team of researchers at the University of Malta (UM) are developing software which can enhance images to extract critical details and clues using artificial intelligence. Dr Ing. Christian Galea, one of the researchers in the Deep-FIR project, talks us through his journey.

One can look at the challenge of reading low-quality images captured from CCTV, mobile phones, and other means as a puzzle. Hidden in those pixels could be clues to solving a crime. However, if one is not cautious, image enhancement software could implicate the wrong person. To understand why this is the case, one has to understand how Dr Ing. Christian Galea (Research Support Officer, Faculty of ICT) and the team at UM tried to solve these puzzles. What is the artificial intelligence software they programmed trying to do?

The Deep-FIR software works by feeding a large dataset of images through an artificial intelligence, training it to recognise patterns and to thus make an incredibly accurate guess wherever there is information missing. In other words, the software makes an educated guess about what should be in an image, based on different examples it has been exposed to. In reality, the Deep-FIR project does not simply enhance an image or zoom in. Instead, it increases the image’s resolution by filling in the gaps wherever there is missing information. Of course, therein lies the risk and the need for ever greater accuracy. Galea elaborates: ‘If certain details in an image are very small, it is very difficult to get a true prediction. Eye colour is one example. I had an image of David Beckham. I degraded the quality and then super resolved it using the software. The result I got was a David Beckham with blue eyes. In reality, his eyes are green. However, because in the publicly available datasets we use celebrities tend to have blue eyes, then the software used probability to estimate that his eyes would be blue based on the information left in the degraded image.’

WHAT CAN IT BE USED FOR?

Fighting crime is not the only application of Deep-FIR. There are many applications for this sort of software, even in the entertainment industry. Historical footage can be improved and made suitable for the higher definition screens we have today. This is significantly easier for the software to do, as there is already a lot of information to work with. In some cases, all that is needed is to make the images sharper and remove blurriness or any visible compression artefacts.

In deciding the focus of the project, however, the team decided to address a major research gap. While software had been designed before to improve poor quality images, the novelty of the project lies in the ability to apply it to the real world. Currently, software applied to real images such as CCTV snapshots do not work very well because the software is not trained on sufficiently realistic images. Deep-FIR takes the technology a few steps further and uses supplementary information to train the artificial intelligence models.

The software is not limited to faces. Even when a face is not visible due to a mask, for example, there could still be things to note such as tattoos or other identifiable features. By gathering feedback from the local police force, the project team was able to help focus on certain applications and decide which data to feed the software to train it accordingly.

‘We spoke to the police to see what their actual requirements are and how we could make the software as useful as possible. It was then that we realised the scope of the challenge. One of the results of this feedback was to work on making number plates more visible. It can be difficult to distinguish between a 0 and a 9 in images, for example. We can also make the model of a vehicle more recognisable. The goal then is not to get all of the information on the spot, but to help the police to narrow down the possibilities,’ Galea explains.

ETHICAL CONSIDERATIONS?

Facial recognition software is a tool to be used with caution. The project team kept the need for accuracy at the forefront to reduce the risk of misidentifying someone or mistaking crucial details. By its nature, however, the software is not meant to be used to prosecute suspects. In a court of law, the uncertainty around software which works on probability means that the evidence might not be admissible. However, the point of the software is to help police with their
investigation before prosecution. It is there to help them gather crucial data in their investigation, directing them towards potential new leads, for example. It is from these subsequent leads that they would get what they need to prosecute.

In developing Deep-FIR, Galea and the team worked only with publicly available datasets. In other words, no private images of people were used to train the software. Generally, they would use images of celebrities. However, there are still risks that this choice of dataset will skew the data. For example, racial and gender biases are problems which currently affect face-recognition systems.
While the team has conducted limited research on what is called ‘face super-resolution’, Galea expresses the importance of tackling this and other demographic considerations in any follow up to the project: ‘Celebrities do not tend to represent the majority of the population.

Furthermore, they wear makeup and are always dressed in a certain glamorous way. So the software models must be tuned accordingly, and we have to be very careful in how we choose the models. If most people in the dataset are of a particular demographic, then the algorithm will likely work well on that demographic at the expense of other demographics. To counteract this, the available data is not used solely for training but is split into a validation set and a test set so that the tuning of certain parameters and evaluation can be made on data that is not seen during training. This helps to provide a better picture of what would happen when the algorithm is used in the real world.’

WHAT DOES THE FUTURE HOLD?

Asking Galea what might be next, he points to a number of ways in which the software might be improved: ‘There are still a lot of things that could be explored.
We want to work on videos, for example, instead of just images. Currently, we can super resolve and enhance videos frame by frame, but that is not quite the same thing.’

Asking Galea what he is most excited about regarding Deep-FIR, he highlights the fact that it can be applied in the real world unlike other research in the field. Furthermore, the software could be provided with additional information in future, such as the contents in a scene itself. In the case of face super-resolution, information about, say, race and gender could be provided to help guide the software into producing more faithful and less biased results that are tuned to the characteristics of the person. This information could be derived from eyewitnesses’ or victims’ descriptions of criminals when interviewed by law enforcement agents.

Here is facial recognition software for the real world based on upscaling poor quality images – technology from fiction which is no longer just sci-fi. The future, it seems, is finally just around the corner, and Galea and the rest of the Deep-FIR team have made sure that it has been programmed in an ethical way.

Deep-FIR (R&I-2017-002T) was financed by the Malta Council for Science and Technology through the Fusion: R&I Technology Development Program.