Liza Alert search-and-rescue team has existed for eight years. It’s a volunteer organization, the fellowship of the ones who care, that searches for missing people effectively collaborating with the Ministry of Emergency Situations and the Ministry of Internal Affairs. Liza Alert collects the entries on missing people; they conduct various educational events, search management and search operations themselves. The team isn’t involved in any business activities, doesn’t have a checking account and doesn’t accept monetary donations.

Mail.Ru has recently provided Liza Alert with a free platform for services, extensively used for search-and-rescue operations. We decided to talk to Sergey Chumak — the head of Liza Alert IT branch — about the work of the volunteer emergency response group and how high-tech solutions aid them.

How do we search for people?

— Let’s imagine a typical situation — a person went into the woods and disappeared. It’s fall and, therefore, elderly men and women are busy picking mushrooms. That’s how they get lost. Family members start looking for them; they call Liza Alert hotline.

— Initiating the search, on the team forum, we create a new thread with the missing person description (“lookout”) and the info on the time and place for the volunteers willing to participate to meet.

While the group is getting ready to depart, map makers are quickly creating a set of area maps: these maps with square nets would be uploaded to the GPS devices handed out to the search groups.

At the entrance to the woods, we build a radio mast. Arriving volunteers are divided into small groups (“foxes”). Every group is given a task by the search coordinator. When “foxes” return, their movement tracks are copied from GPS to the staff laptop and the coordinator then analyzes the input data. For example, a group let them know about some landmarks not depicted on the map.

All information collected during the search is saved. If someone else gets lost in this area again, the search group will have the map sets available right away along with all the information on the area.

Maps

It’s extremely hard to look for a person in the woods without detailed maps. We are using satellite images and topographic maps (Soviet-issued and contemporary, like OpenStreetMap).

The map sets for a search are created in two formats: for Garmin GPS used by the groups in the woods, and for the field headquarters. Such maps take lots of space and require editing; they need to be stored somewhere. Some old and low-capacity private server belonging to one of the volunteer participants was used for it before. However, the number of active searches increases from year to year, the team activity spreads out more and more geographically — new local branches are created and that’s why the pressing need of a resource superior in both quality and quantity manner emerged. Then we asked Mail.Ru Group for assistance.

The company provided Liza Alert with cloud resources as part of the Cloud Servers project; so, we established a main and replicated servers where we moved our map storage in April 2018. It’s quite significant support since we now have much more space available for our current activity. Furthermore, Cloud Servers work much faster, not to mention how easy it is to manage and work with the storage. With the help of a special tool, the people responsible for the storage regulate the rights within the group: every new map maker quickly gains access to the server.

Besides the replication, we now regularly create backups that have their own object storage provided by Mail.Ru. It used to be impossible to create them: when anyone lost any data, it disappeared for good.

We’ve been given Cloud Servers and Cloud Storage for all our products that are already used in the searches and for the ideas of products that our volunteers are going to develop. These resources are enough to carry out some large-capacity operations.

The Eye of the Sky

One of our widely used search tools is a quadcopter drone. Automatically operated, they use previously uploaded routes and fly above the area where someone got lost taking shots of the place.

What is this for? The lost person might come out to the open space — a glade, a clearing, a field. Or find himself amongst windfall timber: in the place covered by fallen trees that’s well-seen from the air. The flying drone would spot him and significantly save the recourses of walking groups.

Up until recently, the drone pilots had to review all the images by themselves. It was a job from hell: every flight results in about 1000 images and they must be very carefully reviewed — you need to study them, zoom in unclear fragments. From my own experience, you can’t look through more than a hundred images at once: after that, you start missing the details, you lose concentration and become less perceptive. That’s why we created a page for collective reviewing of images.

Crowdsourcing to the rescue

We are currently teaching a group of volunteers that’ll oversee photogrammetry on “a professional level”. The thing is: some common objects — a tree stump, a post, a garbage bag — might look different or unclear from the above. And one need specific experience to be able to tell what you see right away.

The reviewing system developed for the “observants” present the shots in such a way, so that to allow a group to review as many images as possible. First, very user reviews his own selection. Then they look at the images of the others; and if there’s time — check the images they’ve already seen again. Cross review and double checking lessen the possibility of missing the important details.

Even an untrained viewer can identify a lying man. That’s why we have a simultaneously operating open service watcher.lizaalert.ru: a crowdsourcing tool of sort where anyone willing to help the team can do just that.

As soon as the notification on the images from the recent flight appears on forum and VKontakte group, up to 150 people simultaneously start reviewing them.

The images here are strictly depersonalized. To make the research easier, every picture is divided into 32 small fragments that fit the screen without scaling. The internet users need to answer “yes” or “no” to only one question: is there a person in the image?

The system memorizes the number of positive and negative responses for every fragment and calculates an average in order to range the fragments; the results are sent to the professional observants who review the most “promising” images in detail.

The team of trained photogrammetrists reviewing the images has detailed information on the missing person: the way he looks, the way he was dressed, what he was carrying on him. That’s why they are looking not only for people but for artifacts also — the person could’ve dropped or lost something of his (a basket, shoes, a hat). The notification on such findings is immediately received by the search coordinator who then decides whether to send one of the search groups to check it out. When we find a specific object, we can try to figure out the route of the person we’re looking for.

These days the maximum reviewing speed on our site is 10 images per second. As a result, all the data collected by a drone can be processed pretty quickly — within a few hours. While the search groups are walking in the woods, the observants are using computers and mobile devices to “investigate” open areas.

— I bet an idea of using neural networks to solve the images reviewing task have crossed your mind?

— Yes, we’ve considered this option; even though the experts’ opinions on machine learning applicability in this case differ. Anyways, to teach a neural network we’d need to collect a data set — a large set of images, pictures made by a drone that would either have or don’t have any people on them. The bigger the set we’d be able to collect, the more reliable model we’d have. In any case, the neural network experiments won’t begin until winter — it’s a busy search-and-rescue season now, the team members are exhausted due to the constant runs. But we’ll definitely get back to it later. I hope that neural network will at least help speed up the elimination process of images with no people and prioritize the pictures that need to be reviewed first.

From plane images to 3D

Finally, our latest idea is generating of 3D maps based on photos. In order to do so, the pictures must be made from a higher altitude and they should overlap each other better. Liza Alert IT branch already has an experience of 3D maps creating in expensive software. But there’s also an open source OpenDroneMap application; web version is available as well. If the results of their work are compatible, we’ll start using OpenDroneMap in our search operations.

3D map provides a more precise description of the area being searched. Sometimes the missing people go to the woods with a phone and stay in touch for some time. As long as the phone battery isn’t dead, we can ask them for some details, ask them to describe their surroundings. For instance, a person is sitting at the clearing or got into a natural trap — windfall or deadfall timber. Such objects aren’t marked on the maps, they are hard to identify in images. However, we can see them on 3D maps and immediately send the groups out for a check.

A drone needs about 30-40 minutes for shooting of 2 x 0,7 km area. It takes too long to generate a 3D map on PC but doing that on a cloud server with powerful configuration would take acceptable for a prompt response time. Therefore, during the first hours of search, we’d be able to get important information about the area and adjust the tasks for the search groups. Of course, such information will be valuable the next day also, however, the sooner we acquire it, the better are our chances to find and save the missing person.

Liza Alert: volunteers, who save lives