High performance

Making a Tarantool-Based Investment Business Core for Alfa-Bank

• Mail.ru Group corporate blog,
• High performance,
• System Analysis and Design,
• Database Administration

A still from «Our Secret Universe: The Hidden Life of the Cell»

Investment business is one of the most complex domains in the banking world. It's about not just credits, loans, and deposits — there are also securities, currencies, commodities, derivatives, and all kinds of complex stuff like structured products.

Recently, people have become increasingly aware of their finances. More and more get involved in securities trading. Individual investment accounts have emerged not so long ago. They allow you to trade in securities and get tax credits or avoid taxes at the same time. All clients coming to us want to manage their portfolios and see their reporting on-line. Most frequently, these are multi-product portfolios, which means that people are clients of different business areas.

Moreover, the demands of regulators, both Russian and international, also grow.

To meet the current needs and lay a foundation for future upgrades, we've developed our Tarantool-based investment business core.

Read more →

dokshina April 13, 2020 at 02:34 PM

Deploying Tarantool Cartridge applications with zero effort (Part 2)

• Mail.ru Group corporate blog,
• High performance,
• IT Infrastructure,
• Database Administration,
• Distributed systems

• Tutorial

We have recently talked about how to deploy a Tarantool Cartridge application. However, an application's life doesn't end with deployment, so today we will update our application and figure out how to manage topology, sharding, and authorization, and change the role configuration.

Feeling interested? Please continue reading under the cut.

Read more →

Fault Tolerance Web Architecture for Our Cloud Solutions

• Mail.ru Group corporate blog,
• High performance,
• Virtualization,
• Cloud computing,
• Cloud services

Hi Habr,

I'm Artyom Karamyshev, a system administration team leader at Mail.Ru Cloud Solutions (MCS). We launched many products in 2019. We've aimed to make API services easily scalable, fault-tolerant, and ready to accommodate rapid growth. Our platform is running on OpenStack, and in this article, I describe all the component fault tolerance issues that we've resolved.

The overall fault tolerance of the platform is consists of its components fault tolerance. So, I'm going to show you step by step tutorial about all levels where we've found the risks.

Read more →

gen December 18, 2019 at 12:48 PM

Accelerating PHP connectors for Tarantool using Async, Swoole, and Parallel

• Mail.ru Group corporate blog,
• High performance,
• PHP,
• SQL,
• NoSQL

In the PHP ecosystem, there are currently two connectors for the Tarantool server: the official PECL extension tarantool/tarantool-php written in C, and tarantool-php/client written in PHP. I am the author of the latter one.

In this article I would like to share the results of performance testing of both these libraries and show how you can achieve 3x-5x performance improvement (on synthetic tests!) with minimal changes in code.

Read more →

dokshina December 16, 2019 at 12:41 PM

Deploying Tarantool Cartridge applications with zero effort (Part 1)

• Mail.ru Group corporate blog,
• High performance,
• IT Infrastructure,
• Database Administration,
• Distributed systems

We have already presented Tarantool Cartridge that allows you to develop and pack distributed applications. Now let's learn how to deploy and control these applications. No panic, it's all under control! We have brought together all the best practices of working with Tarantool Cartridge and wrote an Ansible role, which will deploy the package to servers, start and join instances into replica sets, configure authorization, bootstrap vshard, enable automatic failover and patch cluster configuration.

Interesting, huh? Dive in, check details under the cut.

Read more →

Tarantool Kubernetes Operator

• Mail.ru Group corporate blog,
• High performance,
• IT Infrastructure,
• Database Administration,
• Kubernetes

Kubernetes has already become a de-facto standard for running stateless applications, mainly because it can reduce time-to-market for new features. Launching stateful applications, such as databases or stateful microservices, is still a complex task, but companies have to meet the competition and maintain a high delivery rate. So they create a demand for such solutions.

We want to introduce our solution for launching stateful Tarantool Cartridge clusters: Tarantool Kubernetes Operator, more under the cut.

Read more →

Tarantool Data Grid: Architecture and Features

• Mail.ru Group corporate blog,
• High performance,
• Lua,
• Data storage,
• Distributed systems

In 2017, we won the competition for the development of the transaction core for Alfa-Bank's investment business and started working at once. (Vladimir Drynkin, Development Team Lead for Alfa-Bank's Investment Business Transaction Core, spoke about the investment business core at HighLoad++ 2018.) This system was supposed to aggregate transaction data in different formats from various sources, unify the data, save it, and provide access to it.

In the process of development, the system evolved and extended its functions. At some point, we realized that we created something much more than just application software designed for a well-defined scope of tasks: we created a system for building distributed applications with persistent storage. Our experience served as a basis for the new product, Tarantool Data Grid (TDG).

I want to talk about TDG architecture and the solutions that we worked out during the development. I will introduce the basic functions and show how our product could become the basis for building turnkey solutions.

Read more →

Rosik October 9, 2019 at 05:32 PM

Тarantool Cartridge: Sharding Lua Backend in Three Lines

• Mail.ru Group corporate blog,
• High performance,
• Server optimization,
• Lua,
• Distributed systems

In Mail.ru Group, we have Tarantool, a Lua-based application server and a database united. It's fast and classy, but the resources of a single server are always limited. Vertical scaling is also not the panacea. That is why Tarantool has some tools for horizontal scaling, or the vshard module [1]. It allows you to spread data across multiple servers, but you'll have to tinker with it for a while to configure it and bolt on the business logic.

Good news: we got our share of bumps (for example, [2], [3]) and created another framework, which significantly simplifies the solution to this problem.

Тarantool Cartridge is the new framework for developing complex distributed systems. It allows you to concentrate on writing business logic instead of solving infrastructure problems. Under the cut, I will tell you how this framework works and how it could help in writing distributed services.

Read more →

Enumerable: How to yield a business value

• High performance,
• .NET,
• C#,
• ООP,
• Microservices

12.3 million of concurrent WebSockets

• High performance,
• C++,
• *nix,
• Lua

• Recovery Mode

How to speed up LZ4 decompression in ClickHouse?

• Яндекс corporate blog,
• High performance,
• Open source,
• C++,
• Big Data

When you run queries in ClickHouse, you might notice that the profiler often shows the LZ_decompress_fast function near the top. What is going on? This question had us wondering how to choose the best compression algorithm.

ClickHouse stores data in compressed form. When running queries, ClickHouse tries to do as little as possible, in order to conserve CPU resources. In many cases, all the potentially time-consuming computations are already well optimized, plus the user wrote a well thought-out query. Then all that's left to do is to perform decompression.



So why does LZ4 decompression becomes a bottleneck? LZ4 seems like an extremely light algorithm: the data decompression rate is usually from 1 to 3 GB/s per processor core, depending on the data. This is much faster than the typical disk subsystem. Moreover, we use all available CPU cores, and decompression scales linearly across all physical cores.

Read more →

From High Ceph Latency to Kernel Patch with eBPF/BCC

• Selectel corporate blog,
• High performance,
• Configuring Linux,
• Server Administration

There are a lot of tools for debugging kernel and userspace programs in Linux. Most of them have performance impact and cannot easily be run in production environments. A few years ago, eBPF was developed, which provides the ability to trace the kernel and userspace with low overhead, without needing to recompile programs or load kernel modules.

There are now plenty of tools that use eBPF and in this article, we’ll explain how to write your own profiling tool using the PythonBCC library. This article is based on a real issue from the production environment. We’ll walk you through solving the problem and show how existing bcc tools could be used in some cases.

Read more →

danikin May 13, 2019 at 12:02 PM

Citymobil — a manual for improving availability amid business growth for startups. Part 5

• Mail.ru Group corporate blog,
• High performance,
• System Analysis and Design,
• Start-up development,
• Product Management

This is the final part of the series describing how we’re increasing our service availability in Citymobil (you can read the previous part here). Now I’m going to talk about one more type of outages and the conclusions we made about them, how we modified the development process, what automation we introduced.

Read more →

danikin May 7, 2019 at 12:43 PM

Citymobil — a manual for improving availability amid business growth for startups. Part 4

• Mail.ru Group corporate blog,
• High performance,
• System Analysis and Design,
• Start-up development,
• Product Management

This is the next article of the series describing how we’re increasing our service availability in Citymobil (you can read the previous parts here: part 1, part 2, part 3). In further parts, I’ll talk about the accidents and outages in detail.

1. Bad release: database overload

Let me begin with a specific example of this type of outage. We deployed an optimization: added USE INDEX in an SQL query; during testing as well as in production, it sped up short queries, but the long ones — slowed down. The long queries slowdown was only noticed in production. As a result, a lot of long parallel queries caused the database to be down for an hour. We thoroughly studied the way USE INDEX worked; we described it in the Do’s and Dont’s file and warned the engineers against the incorrect usage. We also analyzed the query and realized that it retrieves mostly historical data and, therefore, can be run on a separate replica for historical requests. Even if this replica goes down due to an overload, the business will keep running.

Read more →

danikin April 29, 2019 at 11:51 AM

Citymobil — a manual for improving availability amid business growth for startups. Part 3

• Mail.ru Group corporate blog,
• High performance,
• System Analysis and Design,
• Start-up development,
• Product Management

This is the next article of the series describing how we’re increasing our service availability in Citymobil (you can read the previous parts here and here). In further parts, I’ll talk about the accidents and outages in detail. But first let me highlight something I should’ve talked about in the first article but didn’t. I found out about it from my readers’ feedback. This article gives me a chance to fix this annoying shortcoming.

Read more →

morozovsk April 29, 2019 at 01:15 AM

Statistics and monitoring of PHP scripts in real time. ClickHouse and Grafana go to Pinba for help

• High performance,
• PHP

• Tutorial

In this article I will explain how to use pinba with clickhouse and grafana instead of pinba_engine and pinboard.

On the php project pinba is probably the only reliable way to understand what is happening with performance. But usually people start to use pinba only when problems are already observed and it isn't clear where to look in.

Often developers have no idea how many RPS each script has. So they begin to optimize starting from places that seem to have problem.

Someone is analyzing the nginx logs, and someone is slow queries in the database.

Of course pinba would not be superfluous, but there are several reasons why it is not on every project.

Read more →

danikin April 25, 2019 at 12:03 PM

Citymobil — a manual for improving availability amid business growth for startups. Part 2

• Mail.ru Group corporate blog,
• High performance,
• System Analysis and Design,
• Start-up development,
• Product Management

This is a second article out of a series «Citymobil — a manual for improving availability amid business growth for startups». You can read the first part here. Let’s continue to talk about the way we managed to improve the availability of Citymobil services. In the first article, we learned how to count the lost trips. Ok, we are counting them. What now? Now that we are equipped with an understandable tool to measure the lost trips, we can move to the most interesting part — how do we decrease losses? Without slowing down our current growth! Since it seemed to us that the lion’s share of technical problems causing the trips loss had something to do with the backend, we decided to turn our attention to the backend development process first. Jumping ahead of myself, I’m going to say that we were right — the backend became the main site of the battle for the lost trips.

Read more →

danikin April 22, 2019 at 06:52 PM

Citymobil — a manual for improving availability amid business growth for startups. Part 1

• Mail.ru Group corporate blog,
• High performance,
• System Analysis and Design,
• Start-up development,
• Product Management

In this first part of an article series «Citymobil — a manual for improving availability amid business growth for startups» I’m going to break down the way we managed to dramatically scale up the availability of Citymobil services. The article opens with the story about our business, our task, the reason for this task to increase the availability emerged and limitations. Citymobil is a rapid-growing taxi aggregator. In 2018, it increased by more than 15 times in terms of number of successfully completed trips. Some months showed 50% increase compared with the previous month.

The business grew like a weed in every direction (it still does): there was an increase in server load, team size and number of deployments. At the same time the new threats to service availability emerged. The company faced a task of the most importance — how to increase availability without compromising company growth. In this article, I’ll talk about the way we managed to solve this task in a relatively short time.

Read more →

t1ha = Fast Positive Hash

• High performance,
• Perfect code,
• C

• From sandbox

The authoritative guide to Blockchain Sharding

• High performance,
• Distributed systems,
• Cryptocurrencies