LUCY: A drone with ears

Floods in Libya, Greece and Slovenia, fires in Hawaii and Tenerife, earthquakes in Turkey and Morocco,... When a region is hit by a natural disaster, every minute counts when trying to rescue victims. But searching for survivors is complex work. This is because buildings and roads may have been damaged and large areas may have been cut off. The use of drones with daylight cameras and thermal imaging cameras is therefore becoming increasingly popular. They can quickly fly over large areas of devastated infrastructure. They can locate victims and speed up the response of rescue teams.

The problem, however, is that victims trapped under debris are not visible to these image sensors. Factors such as thick smoke, fog or darkness also limit the effectiveness of the cameras. For such scenarios, researchers at Fraunhofer Institute for Communication, Information Processing and Ergonomics (FKIE) are working on a solution that allows acoustic sensors to be added to the cameras. FKIE scientist Macarena Varela, in collaboration with colleagues and research group leader Dr Marc Oispuu, developed LUCY, which stands for Listening system Using a Crow's nest arraY. It is a piece of technology to save the lives of people buried under rubble or trapped in fires.

LUCY is an array of MEMS microphones – known as a crow’s nest array. The aim is to mount them on drones to determine the direction from which sounds such as cries for help, clapping or tapping signals are coming. The small, robust MEMS microphones are inexpensive. They are used in applications such as smartphones. What makes this system unique is that the microphones are mounted on the underside of the drone in a special geometric configuration. This allows them to detect sound from all directions. “The highest vantage point on ships from which you can look in all directions is known as the crow’s nest. The same applies to LUCY. Our system can hear in virtually all directions without restriction,” explains Macarena Varela.

LUCY works in the same way as the human ear. It captures sound information and transmits it to the brain where analysis follows. In the case of the array system, the microphones are the ears and a signal processing unit forms the brain. This system determines the direction from which sounds come.

LUCY currently has 48 microphones. This makes it possible to determine the direction of the sound source very accurately. "Spatial hearing clearly works better with 48 or more microphones than with two acoustic sensors. And both focused hearing in a certain direction and the ability to ignore certain sounds are also improved," says Dr Oispuu. Moreover, the system is able to perceive frequencies that the human ear cannot register. In the future, the number of microphones will be increased to 256 sensors that can process signals in real time.

The system blocks out distracting ambient noise. Examples include wind or birds, but also the spinning rotors of the drone itself. Artificial intelligence (AI) methods and adaptive filters are used to filter out these signals. At the same time, the system is trained to detect sound patterns such as shouting, banging or clapping. People in distress often use these sound patterns to attract attention.

To enable the system to do this, it uses a database of different sounds or signatures on which the AI has been trained in advance. Combined with signal processing techniques, this makes it possible to detect and classify sounds and accurately determine their direction of origin. Furthermore, a compact processing unit ensures that signals can be processed very quickly. When a disaster occurs, the location data received is forwarded to the rescue teams. They can then use tablets, for example, to determine the exact position of the victims.

Thanks to their scalability, the sensor modules and microphone arrays can be used on many commercially available drones. Because both MEMS technology and drones are relatively inexpensive, multiple unmanned aerial vehicles can be used to effectively survey the disaster area. Due to its low weight, emergency responders can carry the LUCY system with them to use it on the ground as well. And it can be mounted on vehicles or used as stationary equipment. FKIE researchers are currently working on further improvements to the experimental system.

Opening photo: In the event of disasters, drones are increasingly being used to fly quickly over large areas of devastated infrastructure, locate victims and speed up the response of rescue teams. (photo: Fraunhofer FKIE)

Also read: Drones protect wind turbines from ice