Hi! I am Aleksey Rybakov, Product Owner at Sber ARVR Lab.

And before telling you what we have achieved, let me explain the meaning of this article’s heading. Or more so, explain the word “metauniverse”. This term was well-described by Mark Zuckerburg: “Metauniverse is internet implemented, where instead of looking at internet, you are inside of it.” Like a parallel universe, which you can enter yourself with AR/VR gadgets or through a 3D-avatar.

In other words, a metauniverse is simultaneously a virtual space and a way of engaging with its objects without the common “screen + keybord/mouse/sensor” medium.

And yes, metauniverses are here. But to get into one, you’ll need a few things. First, you need to know where to look. We will talk about that later when depicting a curious case in Moscow. Second, you need a technology that will help connect the virtual world with the real one. We will talk about that as well.

Now, let me introduce: VPS.

What is a VPS?

VPS (Visual Positioning System) is a technology of visual positioning which detects the user’s position and the angle of your phone’s camera view when you point it at the world in a known location. Its characteristics are vital for “connecting” the virtual world with the real one. VPS allows estimating the user’s position with a 0.5m accuracy, works both outside and inside and, most importantly, not only gives information on where you are standing, but on where you are looking as well. All this allows depicting an augmented reality with a high level of “attachment” to the real world.

Right now, the main technology for positioning is GPS. It has a large bonus – you don’t have to condition the space for it. But it has a significant drawback as well – GPS data is not sufficient for metauniverse needs, because GPS provides only the latitude and longitude of a user’s location. And for precise positioning of an AR layer, you need to understand where the user is currently looking from a certain spot. Additionally, GPS is extremely inaccurate inside buildings which deprives the developers from creating AR layers for them.

VPS works only in familiar spaces meaning the location has to be scanned and uploaded in the service beforehand. At the same time, there is no special equipment needed for scanning. Basically, it’s just video shooting. We shoot it with a simple amateur 360-camera. It’s affordable and convenient because it “looks” all ways simultaneously and the shooting takes less time. Of course, we didn’t think of it at first. We started with laser scanning which was extremely expensive and took a tremendous amount of time. Then, we tried shooting with an iPad through LiDar with attached sensors, but this wasn’t much better. Nowadays, getting to a location takes more time for our scouts than the scanning itself, which is rather pleasing. But if the location is small, you can even shoot it with your phone.

You have probably never heard of this technology and “VPS” for you means virtual private server and nothing else. I won’t be surprised if so. The VPS market is, fairly speaking, empty. The only more or less major cases of it are the AR map routes in iOS 15 and the Local Lenses project by Snap. But these companies do not provide their solutions for external developers. We, on the other hand, are offering any developers to use our service of visual positioning in their projects.

Right now you can fill out a form on SmartMarket and tell us that you want access to test usage of VPS for your project. We have scanned several popular locations in Moscow and are ready to provide visual locating on them for free, so you can become familiar with the technology.

Specifically for an easy integration of VPS into your project we have developed SDK for all major platforms: iOS, Android and Unity, which you can already get acquainted with in our public GitHub repository. Be sure to write, ask questions and give feedback after testing!

How the technology works for users

VPS has numerous facilities. Let’s look at the most obvious one — building precise routes, street navigation and, what was almost impossible till now, navigation in buildings.

Imagine: you exit the subway into a large square and want to have a coffee. You open an app on your phone. A virtual character in augmented reality shows you the route to the nearest coffee shop and invites you to follow him. A similar case has already been implemented in Apple Maps on iOS 15 but for now it only works in major US and UK cities.

As a user, you don’t just receive information in a “turn left in 200m” format, like with GPS. The service, used by VPS shows you: “Go here. Move to the right. There is a bike line, be careful. Now take these stairs. No, not the central stairs, the side ones. You need the second door on the right”.

Or you may get the information through a game with a virtual character, which appears only at a certain location. VPS can also improve popular existing AR-games, like Pokémon Go. Imagine: hiding a pokemon not just in a certain area, but in a particular spot. Like behind a monument or in front of a restaurant entrance. This expands both conceptual and commercial potential of AR-games. Speaking of which, Pokémon Go developer, Niantic, is actively working on their own VPS right now and has launched the Lightship project just this May.

On top of that, we are anxious for AR-devices to become cheaper, more accessible, and convenient in the form-factor. Metaverse and similar platforms will give an extra push to VPS. And with the current speed of the AR industry development, this is a matter of not decades, as we used to think, but mere years.

Where can we see VPS in action? Where is the giant robot?

We finally get to the robot! Long story short, you can see it in the center of Moscow, above the Polytechnic Museum, if you have an iPhone. And here is its story.

It was a match. The Museum wanted to engage its audience while their main building at the “Novaya ploschad” was closed for reconstruction. And we were looking for an opportunity to apply VPS to something bigger than just an inside app for employees. Thus, the stars aligned. Now, while the building is still being reconstructed, the Museum decided to create a teaser of the new space with our help and show it through a new mobile app. You still can’t get inside the museum building but with the “Polytechnic Museum” app you can already have some fun. On a sidenote though, for now there’s only an iOS version, the Android one is in process. The concept of the project was created collectively: the Museum worked on the app and our lab was responsible for 3D-graphics and VPS. Here is some content from the app:

What will it look like?

You download the "Politekhnicheski Muzei" app.

↓

Launch the app and point your camera at the building. The algorithm works on any side of it, by the way. It may not work only from the top – we haven’t tested it on flying users yet. We also suggest testing it in different lighting and weather conditions. Even in the rain at night, VPS recognizes the Museum and adds the robot.

↓

We take the shot from your camera and using GPS, simultaneously confirm that you are next to the museum and not just showing a photo to your phone. If someone gets inspired to test this: “What if I use a photo and fake GPS?” – make sure to tell us in the comments if you were able to cheat the system.

↓

Everything’s OK and the geolocation is correct? Then we pre-edit the shot on your device and get a unique set of dots from it.

↓

We send the dots to our cloud server.

↓

We match these dots with a digital scan of the surroundings and locate the position and angle of the user’s camera. Then, we transfer the received coordinates into your app where we transform them into coordinates for your 3D engine using SDK.

↓

You see a giant robot, Sepulka, about the Museum building, their mascot. And if you catch one of the objects flying around him, you can interact with the app’s main content – see the AR halls and some of the display items.

What are the current limitations of VPS

The algorithm requires certain features to work, meaning unique objects and their combinations it can grasp. If you show an empty wall to the algorithm, it starts complaining about life and demands to see more. A regular white hallway with no doors turns into a “death hall” for it. Although such design is now rare even in industrial and warehouse buildings.

The VPS technology is now on the stage of testing which is why it’s not always precise. We are currently studying different approaches to localization and constantly improving it.

And why are we telling you all this?

VPS is the future of mobile AR-technologies. And we would like to confer with you on said future. Maybe you’ll get ideas and suggestions on how to make life simpler and more exciting with VPS. Any ideas – from business to absurd ones. We’ll be happy to hear your thoughts! You can contact our VPS team through the form on our page on SmartMarket.

Real world metaverses are here. You just have to know where to look. One of Sber AR/VR Lab projects of 2021