Future Cockpit

In the near future, commercial airline pilots will be able to speak and use hand gestures to interact with an aircraft. But there is still a way to go before this technology becomes available. Dr Jason Gauci writes about the research that is being done by the Institute of Aerospace Technologies (University of Malta) to meet these challenges.

In 2016, 3.8 billion people travelled by air, according to estimates from the  International Air Transport Association (IATA). The figure is expected to nearly double by 2035. With so many people airborne, safety cannot be underestimated.

Airline pilots are vital in keeping planes safe. But with the continuous increase in aircraft automation, the role of pilots is changing from flying the aircraft to managing and supervising its systems. For this to be effective, pilots need to be aware of how the aircraft is behaving and monitor what is going on around them, a concept called situational awareness.

Improving the cockpit interface is one way to increase pilot situational awareness and reduce crew workload. Current cockpits can be made simpler to use and more intuitive. One of the challenges associated with conventional cockpits is the vast number of controls (such as switches and buttons) and indicators (such as instruments and displays) that are physically separate from each other instead of being integrated. For instance, to change the aircraft’s speed, the pilot needs to access a control unit in one cockpit area and then check the speed on another display in a different cockpit area. The pilots end up looking like choreographed hand puppets stretching in different directions to access the right controls.

“The aircraft systems monitoring interface integrates a plane’s vital stats (such as temperature and engine RPM) in the same interface.”

New cockpit design solutions have been proposed by leading avionics manufacturers. Most concepts revolve around replacing conventional displays with multiple large touchscreen displays in front of the pilots.

Our team from the Institute of Aerospace Technologies (University of Malta), made up of myself, Prof. Ing. David Zammit Mangion, Matthew Xuereb, and Kevin Theuma, in collaboration with QuAero Ltd., have developed a novel touchscreen interface concept which we believe has several advantages over competing solutions. This concept was developed as part of the avionics TOUCH-FLIGHT project.

One touchscreen to rule them all

Current touchscreen cockpit concepts use multiple touchscreens to access the various kinds of information a pilot needs. The TOUCH-FLIGHT concept instead uses a single touchscreen to seamlessly meld all tools in one place. The device can be placed on a table in front of each pilot with side-sticks. The setup allows pilots to comfortably fly the plane, even in mild turbulence. With this device, the pilot can monitor and control any avionic system using touch.

To show that the concept works, we built a custom prototype in the Engineering Building on campus. The prototype touch-screen interface focuses on four key areas: autopilot control, flight management, aircraft systems monitoring, and aircraft performance. The autopilot control interface allows the pilots to both make tactical changes to the aircraft’s flight path and monitor those changes from the same system. The pilots could use a keypad or a speed tape as they adjust to using their fingers, an interface that merges the Flight Control Unit (FCU) and the Primary Flight Display (PFD) of conventional aircraft.

Pilots use the flight management interface to adjust a plane’s flight plan. Normally, they modify waypoints along the route or, as in the case of a delayed flight, select a different runway at a destination airport. In our system, the flight plan can be changed as normal through a keypad or through touch by swivelling an on-screen representation of the flight plan. Our studies have shown that this is much more intuitive than the current system of modifying the flight plan through buttons on the Control Display Unit (CDU) and viewing changes on the Navigation Display (ND).   

The aircraft systems monitoring interface integrates a plane’s vital stats (such as temperature and engine RPM) in the same interface. In a conventional cockpit, pilots generally monitor systems via an Engine Indicating and Crew Alerting System (EICAS) display and control them through switches on an overhead panel. TOUCH-FLIGHT’s fourth interface is dedicated to aircraft performance. It allows pilots to view and adjust various performance parameters related to different phases of flight, from take-off to landing. Our system allows pilots to control and monitor a plane using four interface devices on one screen instead of the many systems current cockpits have, streamlining the process.

Having one screen also lets us overcome the challenge of using touchscreen devices in turbulent conditions. This common problem was addressed by placing the touchscreen device right in front of the pilot, where it is easier to grip the device and touch the screen more accurately. However, this solution is not sufficient, and our team is currently working on ways to interact with the cockpit without the need to physically touch it. 

The real deal

We tested the above cockpit on ten commercial airline pilots. We wanted to see how they reacted and whether they enjoyed using our system. The pilots virtually flew from Malta to Rome with issues arising to test all of the  functions of the touchscreen interface.

Six of these pilots saw a clear benefit of the touchscreen idea, with the other pilots finding TOUCH-FLIGHT equally good. Considering that new technologies need time and effort to get used to, we saw this as an important success. All pilots thought it was great for normal plane flights, being intuitive and easy to use.

The pilots finding the system intuitive and useful was a key challenge we wanted to overcome. This was the reason we teamed up with QuAero Ltd., who brought in the commercial pilots we tested the prototype on. We involved these pilots throughout the research project from developing the concept to testing the product. This proved to be particularly useful when defining the requirements of the project and when testing out some early design ideas for the interface. By having QuAero directly involved in the project, we could get quick feedback
on our designs and make alterations as necessary. This was essential, as multiple design iterations were required before the interface was sufficiently mature for pilots to like it.

Despite involving the pilots from the start, however, a psychological challenge reared its head. Convincing some pilots of the benefits of using touchscreen technology was, and still is, an issue. Experienced pilots were the toughest to convince, with the main issues being discomfort with the technology and safety concerns—why fix something that is not broken? As expected, younger pilots pick this technology up very rapidly. Pilot training will be key to implementing touchscreen technology.

The concept developed in TOUCH-FLIGHT has been protected by a number of international patents and we are now working with the Knowledge Transfer Office at the University of Malta to commercialise the technology, all the while implementing the feedback we got from pilots to make TOUCH-FLIGHT even better.

The future is bright

The TOUCH-FLIGHT project is now being taken to the next level: TOUCH-FLIGHT 2. This project adds voice recognition technology to the touchscreen interface, allowing pilots some redundancy for which systems they prefer using.

The use of voice recognition in the cockpit has several potential benefits. For instance, since pilots do not need to use their hands or look at a particular display to interact with a particular system, they can focus on other tasks and spend more time with their heads up. Also, certain tasks which would require the pilot to navigate through multiple menus on a touchscreen display could be executed with a single voice command. Furthermore, the use of voice commands can overcome turbulence issues described previously.

Touchscreen and voice recognition technologies will inevitably become part of the cockpit. The advances being made cannot be stopped. Pilots can look forward to the challenge of ‘reprogramming’ themselves. No more waving hands all over the cockpit, a simple touch will be enough.   

The TOUCH-FLIGHT R&I project was financed by the Malta Council for Science and Technology through the National Research and Innovation Programme 2012 (Grant Agreement R&I-2012-065); TOUCH FLIGHT 2 is being financed by the FUSION R&I programme (Grant Agreement R&I-2015-015-V).