Walk the plank, please

It's supposed to be straightforward – we smile at our conversation partner, and the other person smiles back. But what if our smile is not understood, or not interpreted as friendly?

Those diagnosed with autism struggle with exactly that. Autism is a neurodevelopmental disorder that affects social interaction, communication and behaviour. Autistic people often have to learn which facial expressions belong to which emotion in order to "read" other people and react in the way that is socially expected. Fear and uncertainty make this a constant struggle for those with autism. Researchers at Furtwangen University (HFU) are now seeking to alleviate this.

The goal of the "KomPASS" research project is to generate computer games that make it easier for autistic people - especially young people - to "read faces". "The idea is to develop a form of training, or even a permanent therapy, for children and young people that they do not find strenuous," explains project leader Prof. Dr. Knut Möller, Head of the Institute for Applied Research at HFU. Games are fun – even if you do have to identify and categorise an emotion to reach the next level.

"The aim of the project is to develop a protected system in which the game players can complete long-term training, even at home," says Möller. The computer games are not only suitable for autistic people – they could also feasibly be used by those whose personnel responsibilities require high levels of empathy.

Capturing emotions through virtual reality

In order to create such game-based training, emotions must initially be captured so that they can later be reproduced via avatar. "In order for a computer to recognise an emotion, we first have to successfully trigger this specific emotion," says Prof. Dr. Verena Wagner-Hartl, who is also part of the project team. She is in charge of the measuring station in the "Engineering Psychology (IP) / Human Factors (HF)" laboratory on the Tuttlingen campus of HFU where tests have already been carried out to determine the extent to which images can trigger measurable feelings. "But the results were not clear enough," says Wagner-Hartl. In the meantime, the team uses VR goggles with which the test persons – mostly students – experience particular situations. "In the IP/HF lab, for example, we make them walk on a plank across the space between two high-rise buildings," reports the professor of engineering psychology. The thrill is clearly measurable – sweaty palms, increased pulse, and an eye-tracking system in the goggles measures eye movements. The experiment is also extremely popular with visitors. "For example, we have already had children as guests at the Open Day whose experience was so real that they landed on all fours when they jumped off the plank," reports Wagner-Hartl.

Test persons also help to record emotions by recalling moments in which the emotion in question was felt most intensely.

The KomPASS team is interdisciplinary and not only within the HFU, where the Faculties of Industrial Technologies and Medical and Life Sciences collaborate. International partners are also involved in KomPASS and other planned follow-up projects: software specialists are being brought in from Hungary; the gamification of the project is being developed at Graz University of Technology; and Germany's largest autism centre, the University Hospital of Freiburg, is also involved. There, for example, the concept of the games is evaluated by employees who are themselves autistic. "This is how we ensure that we simulate situations in the games that are really helpful," explains Prof. Möller.

Also of great interest – and therefore being researched in a separate project – is the question of the extent to which our social behaviour patterns depend on our cultural background. Prof. Möller and his team are therefore collaborating with researchers from New Zealand who are investigating with Maoris how ethnic identity influences forms of expression.

The KomPASS project, funded by the state of Baden-Württemberg, is scheduled to run for two years – the follow-up projects are supported by the Federal Ministry of Education and Research, and one of them by the EU.