Augmented Reality, a useful application for aircraft maintenance?

With the Microsoft HoloLens and the success of Pokémon Go, Augmented Reality (AR) is being developed more and more, the use of augmented reality for industrial applications is coming closer. Aviation is an industry with long certification processes for new technologies like this, but AR applications are being developed as we speak. The use of a Heads Up Display (HUD) is already widely applied to military aircraft and even is used in the newest civil aircraft. But what are the applications of AR in the area of aircraft Maintenance, Repair and Overhaul (MRO)?

(Image source: Netherlands Aerospace Centre)

Safety and cost savings are the number one and two drivers in Aviation (up to you to decide which comes first). Among others, because of extensive (and mostly expensive) maintenance activities, the level of safety remains sky high and the number of casualties as low as possible. However, due to human factors, enough incidents occur that could have expensive effects. Proper training and experience minimizes the risk of these incidents to occur, and AR could help minimize these risks even further. I found some interesting applications of AR that have been developed already in MRO and will discuss them in the next section.

Firstly, Microsoft has demonstrated the application of getting familiar with for instance a turbofan engine (link). From my own experience I could add to this that seeing something in real does add up to the theory you learn in books. Adding AR to this equation could combine these two and even go further. Secondly, Operose has developed an application for the HoloLens, in cooperation with Magnetic MRO, which displays liveries on aircraft (link). It is not an application that could be useful on a daily basis for every MRO, but it demonstrates the way AR could be used to see the application on 1:1 scale. And lastly, the Netherlands Aerospace Centre (NLR), in collaboration with KLM Royal Dutch Airlines, developed a demonstrator for using AR in maintenance training (link).

The use of AR could be applied even further, to the maintenance activities itself. Kingsee demonstrates in this video (link) the application of AR during maintenance tasks. The way certain things are displayed could be improved, but it shows the basic application of AR within the maintenance activities. I expect that steps will be taken to reach a way of working like this, however it has its pros and cons.

Starting with the pros, the major improvement that comes with using AR is a more efficient maintenance and more fault free system. The tonnes of paperwork that came with the old aircraft will become obsolete and all documentation could be done digital which saves a lot of money. With the development of more and more ‘Ditigal Twins’ and 'Big Data', these buzzwords could even be combined. Imagine a line maintenance manager doing a workaround during a turnaround with all the information he or she should have displayed on the focus spots itself? Countless applications could be imagined.

Digital Twins and AR, a golden combination?

Back to using AR in the maintenance process, it will also come with a lot of cons: when looking at the dirty dozen in human factors (link), some came up by imagining the application of AR. Development of a Lack of Assertiveness is one of the first Dirty Dozen that crossed my mind: When personnel gets familiar with using the HoloLens and, by a technical error, information is displayed wrongly, personnel could mindless follow the wrong instructions. Using physical manuals, this problem could be mitigated for the biggest part. Stress is a second Dirty Dozen that crossed my mind: due to being exposed to more inputs than in ‘normal reality’, the stress level could be enhanced that could lead to hazardous situations. When a mechanic has not slept well and a lot of noise is present in the hangar, the stress level could rise to an unwanted level. The last dirty dozen is not the most obvious, but personnel will experience fatigue problems faster when using an application like the HoloLens. The lens works with projecting the augmented reality around you by using a screen right in front of your eyes. Unconsciously, your eyes will switch from focussing to the screen to focussing on the surface where the object is projected on very fast which results in eye sore and higher fatigue for the personnel using the AR application. The research by NLR also mentioned this issue (link).

Due to all the cons I just mentioned, I cannot imagine a future of Aviation MRO without the application of AR. We, as Aviation professionals, should be aware of the Dirty Dozen and the new challenges in the Human Factor area that are linked with applying AR in MRO. In the meantime, developments like MRO AIR are a major stepping stone to the actual application of AR.