Danny Krastev, Mirabbos Umarov, Ekaterina Menshenina, ITMO University, Info communication Systems, Computer Science. 2019

Abstract

Due to the never-ending increase in volume of vehicles surrounding our daily lives, Automatic Number Plate Recognition (ANPR), has become an evolving solution for managing and monitoring vehicles worldwide to enforce rules and prevent criminal activities, such as parking violation, red light violation, speeding, and vehicle theft. Although there is already a variety of public and private methods and libraries that have been developed and are used to achieve the automatic recognition of car license plate numbers around the world, there has not been much focus on making advancements toward a cross platform ANPR solution that supports all vehicle license plates worldwide. This paper introduces the Plate Vision project, a web and mobile application built on Ruby on Rails and React Native, which aims to serve as an alternative ANPR platform that supports detection of all license plates worldwide by utilizing various open source optical character recognition (OCR) libraries and making efficiency optimizations.

Key words and phrases: ruby, rails, react native, license plate recognition, plate region extraction, optical character recognition (OCR), ANPR.

1. Introduction

Since the core functionality and performance of Automatic Number Plate Recognition (ANPR) is based on years of research that uses distinct combinations of optical character recognition (OCR) techniques on images in order to find and read the license plates on the vehicles, and a variety of solutions already exist, there is no need to go through the extraneous development and training required to entirely recreate the OCR functionality itself, therefore the objective of this paper is to present an example of how to combine and optimize the existing OCR and ANPR solutions in order to develop Plate Vision, a unique ANPR platform that can work on any device using any camera, vehicle, location.

The rest of the paper proceeds as follows. Section 2 presents some existing solutions and dependencies. Section 3 describes the Rails backend, frontend and the React native integration. Section 4 explains the ANPR optimization and testing as well as how Plate Vision was built, hosted, and maintained online, while Section 5 concludes the paper.

2. OCR Dependencies

Although many of the corporate tech giants such as Google, Amazon, Microsoft, and IBM provide their own cloud vision APIs which have the ability to analyze images and tag certain objects and features using algorithms that have taken their researchers years to develop and train, they are paid solutions, which can quickly become expensive when trying to scale or even demonstrate a platform such as ANPR to any amount of users, therefore using open source frameworks and libraries is not only a better solution economically but also one that provides the necessary ability to modify and control the ANPR detection algorithms.

From all of the various solutions that currently exist, OpenALPR is the only one which is open source and provides the ability to modify and control it’s ANPR detection algorithms for specific countries, which is why it is was chosen to be used for Plate Vision.

OpenALPR also has 2 primary dependencies, OpenCV and Tesseract OCR, which serve as the foundation for performing the ANPR. It is best to keep the amount of dependencies a platform has to a minimum for the best performance, however with any project that integrates custom OCR there are bound to be multiple large dependencies [1].

3. Platform

The Plate Vision platform is built using the Ruby on Rails and React Native frameworks, and is therefore composed of 2 individual applications, the React Native mobile application, and the Rails application which serves as the main website and backend for the app.

Rails is a model–view–controller (MVC) and a server-side web application framework written in Ruby, which provides default structures for a database, a web service, and web pages. It encourages and facilitates the use of web standards such as JSON or XML for data transfer, and HTML, CSS and JavaScript for display and user interfacing [2].

React Native is an open-source mobile application framework created by Facebook which enables writing JavaScript code that can be converted and interpreted to native code, thus allowing the easy compiling of working cross platform applications for Android, iOS, and UWP using one single language [3].

OpenALPR is an open source Automatic License Plate Recognition library written in C++ with bindings in C#, Java, Node.js, and Python. The library analyzes images and video streams to identify license plates. The output is the text representation of any license plate characters [1].

3.1 Backend

The backend of the platform is made up of various gems (libraries) and rails controllers which handle the requests from the website and mobile application respectively.

The following lists and describes the gems that are used in the backend:

ALPR – used for calling openALPR commands
Devise — is a flexible authentication solution for Rails based on Warden
Devise token auth — Simple, multi-client and secure token-based authentication for Rails.
Rmagick — an interface between the Ruby programming language and the ImageMagick image processing library
Rolify — simple Roles library without any authorization enforcement supporting scope on resource object.

The following lists and describes the website application controllers:

Application controller – is the main controller that contains the methods, which can be executed throughout the website.
Public controller – is inherited controller (form Application controller) that handles requests from the public pages on the website.
Users controller – handles requests from user account page.
Sessions controller – is a user authentication controller.
Vehicles controller – handles the creating, editing and deleting vehicles.

The following lists and describes the application controllers that serve as a restful JSON API and handle the React Native Mobile Application requests:

Alpr controller – contains methods for analyzing the image in order to find a license plate number.
Devices controller – contains methods for saving the device information of the user to the Database.
Locations controller – saves the user device location data to the database.
Users controller – handles the user authorization/authentication.
Vehicles controller – handles vehicle (create, edit, delete) object methods and saves vehicle data into the Database.

3.2 Frontend (Website)

The frontend of the platform is developed using a few rails gems as well as Bootstrap, jQuery and custom JavaScript where necessary.

Bootstrap is a free and open-source front-end framework for designing websites and web applications. It contains HTML- and CSS-based design templates for typography, forms, buttons, navigation and other interface components, as well as optional JavaScript extensions. [4]
jQuery is a free and open source cross-platform JavaScript library designed to simplify the client-side scripting of HTML. [5]

The following lists and describes the gems that are used in the frontend:

Simple Form – improves upon the existing usage of standard rails forms [6]
Simple Navigation – helps easily create dynamic navigations [7]
Rails Admin – provides a dynamic admin interface for the application [8]

The following is a list of the main site pages and their description:
Home – Main landing page for the website
Vehicles Index – Page where all created vehicles are listed
Vehicles New/Edit – Page with form where vehicle is created and edited, and the license plate is analyzed
Vehicle Show – Page that shows the details about a specific saved vehicle.
Sign up – Page where any guest user can create an account for the platform
Sign in – Page where any guest user can sign on to the platform
User account – Page where user can see and edit details about their account
Forgot Password – Page where any user can request to reset their password using email

3.3 React Native Application (Mobile)

The app-industry today is ruled by two giants, namely Android and iOS. For companies and developers, it is essential to deliver a product to the majority of users, thus the necessity to adapt both platforms. The platforms involve their own way of developing applications and only have small resemblances between each other. This has been a problem that the app industry has faced and overcome by having to hire staff with knowledge in either Android or iOS, or both, to build two separate applications that in turn require parallel upkeep [9].

Cross-platform frameworks that bridge this gap have come and gone, by not succeeding in creating applications with the same visual or functional standard as the native frameworks deliver. However, React Native promises to deliver a fully native experience with the use of only one code base. [10]. React native requires and runs on the NodeJS platform. Node.js is an open-source, cross-platform JavaScript run-time environment that executes JavaScript code server-side. [11]

In order to understand a React Native application, its necessary to know about some of the basic React concepts, like JSX, state, props and components. JSX — is an XML-like syntax extension to ECMAScript without any defined semantics. Components – are the building blocks of any React native app and a typical React native app will have many of these. Simply put, a component is a JavaScript class or function that optionally accepts inputs i.e. properties (props) and returns a React element that describes how a section of the UI (User Interface) should appear.

There are two types of data that control a component:

Props: are immutable and are set by the parent and they are fixed throughout the lifetime of a component.
State: is mutable. This means that state can be updated in the future while props can’t. State can be initialized in the constructor, and then called by setState when it needs to be changed.

By default React native provides a number of built-in components, that can be used in different purposes. Furthermore, developers are not limited to the components and APIs bundled with React Native because there are a lot of different packages (libraries) that many other developers have created to speed up the development for others.

The following lists and describes the packages that are used with nodeJS to develop the React Native application:

React – A declarative, efficient, and flexible JavaScript library for building user interfaces. [12]
React Native – A framework for building native apps with React. [13]
Expo – A set of tools, libraries, and services that allow the building of native iOS and Android apps by writing JavaScript. [14]
Redux – A predictable state container for JavaScript apps. [15]
Axios – A Promise based HTTP client for the browser and node.js [16]
Native Base – Essential cross-platform UI components for React Native [17]
React Native Navigation – provides native platform navigation on both iOS and Android for React Native apps [18]
React Form Builder – Simple form generator library for react. [19]
Il8next — Internationalization framework for browser or any other javascript environment (eg. node.js) [20]

The following is a list of the app pages and their description:

Home – Main app landing page for the application
About – App page that describes what the platform is about
Sign up – App page where any guest user can create an account for the platform
Sign in – App Page where any guest user can sign on to the platform
Vehicles Index – App page where all created vehicles are listed
Vehicles New/Edit – App page with form where vehicle is created and edited, and the license plate is analyzed
Vehicle Show – App page that shows the details about a specific saved vehicle.

4. Testing & Hosting

While the initial tests and research for this platform are based on Russian license plates the ultimate goal of the Plate Vision platform is to be able to read license plates from around the world, and since OpenALPR had already been trained for popular countries like USA and Europe, it was initially assumed that it would also work well with Russian license plate numbers, since some of the European plates are similar to the Russian ones.

However, initial tests using the default European OpenAlpr configurations on Russian license plates, show that although OpenAlpr was able to identify and recognize some amount of the Russian license plates, further customization of OpenAlpr was needed in order to get reliable and consistent results. In order to improve the analyzing of the Russian plates without any extra training using OpenAlpr, a few custom configurations were created, derived and optimized from the European configurations with specific modifications based on a variety of test images that contain cars with Russian license plates.

Even though the platform has been optimized to work best Russian license plates, it already supports many images of cars and plates from around the world because it runs multiple modified configurations of the OpenAlpr library, which leads to many possible outcomes, and in the end determines the best result based on world region plate pattern matching and reading accuracy.

The entire platform is hosted using the cloud services of Heroku and is updated and maintained through the git version control system. However, it is important to note that, setting up specific dependencies like OpenAlpr to work on a cloud server environment like Heroku was difficult due to the restrictions, and challenges presented by Heroku’s cloud server environment. Therefore, the services of Docker were also used in combination with Heroku in order to overcome the difficulties of setting up and ensuring that the cloud OpenAlpr and all of its dependencies function the same as the ones in the local development environment [1].

5. Conclusion

Conclusively, the objective of this paper was to present the Plate Vision platform as well as describe it’s 3 main functioning components and how it plans to optimize ANPR while still supporting any device, camera, vehicle, location in the future. Although the current version of the platform only implements and uses OpenAlpr with multiple configurations in order to improve upon the Automatic Number Plate Recognition, there are still different ways to achieve ANPR, and with further research, testing, and experiments, the Plate Vision platform hopes to combine unique techniques and methods that make use of the best open source vision and OCR libraries available to create a truly optimal international ANPR solution.

References

References

1. OpenALPR Documentation – http://doc.openalpr.com/
2. Ruby on Rails Documentation – https://guides.rubyonrails.org/
3. React Native Documentation – https://facebook.github.io/react-native/docs/getting-started.html
4. Bootstrap Documentation – https://getbootstrap.com/
5. JQuery Documentation – https://jquery.com/
6. Simple Form Documentation – https://github.com/plataformatec/simple_form
7. Simple Navigation Documentation – https://github.com/codeplant/simple-navigation
8. Rails Admin Documentation – https://github.com/sferik/rails_admin
9. Axelsson Oscar, Carlstrom Fredrik [Evaluation Targeting React Native in Comparison to Native Mobile Development] – https://lup.lub.lu.se/student-papers/search/publication/8886469
10. Hansson Niclas, Vidhall Tomas [Effects on performance and usability for cross-platform application development using React Native] – http://www.diva-portal.org/smash/record.jsf?pid=diva2%3A946127&dswid=8626
11. NodeJS — https://nodejs.org
12. ReactJS — https://reactjs.org/
13. React Native — https://facebook.github.io/react-native/
14. Expo — https://expo.io/
15. Redux — http://redux.js.org/
16. Axios — https://github.com/axios/axios
17. Native Base — https://nativebase.io/
18. React Native Navigation – https://wix.github.io/react-native-navigation
19. React Form Builder – https://github.com/bietkul/react-native-form-builder
20. Il8next – https://www.i18next.com/

ANPR using RoR & React Native