Pull to refresh
26.05
Rating

Distributed systems *

Nuances of designing distributed systems

Show first
  • New
  • Top
Rating limit
  • All
  • ≥0
  • ≥10
  • ≥25
  • ≥50
  • ≥100

Distributed Artificial Intelligence with InterSystems IRIS

InterSystems corporate blog Machine learning *Distributed systems *Artificial Intelligence

Author: Sergey Lukyanchikov, Sales Engineer at InterSystems

What is Distributed Artificial Intelligence (DAI)?

Attempts to find a “bullet-proof” definition have not produced result: it seems like the term is slightly “ahead of time”. Still, we can analyze semantically the term itself – deriving that distributed artificial intelligence is the same AI (see our effort to suggest an “applied” definition) though partitioned across several computers that are not clustered together (neither data-wise, nor via applications, not by providing access to particular computers in principle). I.e., ideally, distributed artificial intelligence should be arranged in such a way that none of the computers participating in that “distribution” have direct access to data nor applications of another computer: the only alternative becomes transmission of data samples and executable scripts via “transparent” messaging. Any deviations from that ideal should lead to an advent of “partially distributed artificial intelligence” – an example being distributed data with a central application server. Or its inverse. One way or the other, we obtain as a result a set of “federated” models (i.e., either models trained each on their own data sources, or each trained by their own algorithms, or “both at once”).

Distributed AI scenarios “for the masses”

We will not be discussing edge computations, confidential data operators, scattered mobile searches, or similar fascinating yet not the most consciously and wide-applied (not at this moment) scenarios. We will be much “closer to life” if, for instance, we consider the following scenario (its detailed demo can and should be watched here): a company runs a production-level AI/ML solution, the quality of its functioning is being systematically checked by an external data scientist (i.e., an expert that is not an employee of the company). For a number of reasons, the company cannot grant the data scientist access to the solution but it can send him a sample of records from a required table following a schedule or a particular event (for example, termination of a training session for one or several models by the solution). With that we assume, that the data scientist owns some version of the AI/ML mechanisms already integrated in the production-level solution that the company is running – and it is likely that they are being developed, improved, and adapted to concrete use cases of that concrete company, by the data scientist himself. Deployment of those mechanisms into the running solution, monitoring of their functioning, and other lifecycle aspects are being handled by a data engineer (the company employee).

Читать далее
Rating 0
Views 524
Comments 2

Decentralized Torrent storage in DHT

Hosting Decentralized networks Data storages *Distributed systems *

The DHT system has existed for many years now, and torrents along with it, which we successfully use to get any information we want.

Together with this system, there are commands to interact with it. There are not many of them, but only two are needed to create a decentralized database: put and get.

This is what will be discussed below...
Total votes 8: ↑5 and ↓3 +2
Views 692
Comments 0

VeChain Has Introduced Blockchain-Based Healthcare Data Management Platform At Cyprus Hospital

Distributed systems *Software Artificial Intelligence
image

Blockchain possesses incredible potential, that's why blockchain systems are considered as a paradise for data. In all these years, the discovery of Blockchain has been enjoyed for the introduction of distributed systems to secure data by cryptography.

From the creation of cryptocurrency to distributed ledger systems and mobile applications, this technology is being welcomed by every business vertical, and its adoption has become complimentary for companies. Moreover, its adoption can majorly be cherished by the healthcare industry.

A number of IT institutions are engaged in finding the most promising usage of blockchain technology in healthcare. Let's take a brief look at Blockchain adoption in 2020.

The Global Blockchain Adoption


In 2020, the worldwide spending on blockchain systems is USD 4.3 billion. As per Statista, the market of Blockchain will be worth 20 billion USD by the year 2025. Its most critical adoption can be seen in the healthcare industry, where this technology is being considered as the biggest game-changer.



The truth is that this technology has shown a path to distributed systems coupled with unmatched security measures That secure data in a chain of blocks infused with cryptographic locks. Top-notch level security and quality of not being tempered by any external entity boost its adoption in several instances.
Read more →
Total votes 3: ↑3 and ↓0 +3
Views 975
Comments 0

Deploying Tarantool Cartridge applications with zero effort (Part 2)

Mail.ru Group corporate blog High performance *Database Administration *Distributed systems *Tarantool *
Tutorial


We have recently talked about how to deploy a Tarantool Cartridge application. However, an application's life doesn't end with deployment, so today we will update our application and figure out how to manage topology, sharding, and authorization, and change the role configuration.

Feeling interested? Please continue reading under the cut.
Read more →
Total votes 15: ↑15 and ↓0 +15
Views 625
Comments 0

Blockchain Is Changing The Way Rail Industry Works

Distributed systems *IT-companies
Sandbox
image

Railways had made our transportation very easy since 1830 when the first railway began in England. From 1830 to 2020, the development in the railways has been quite significant. The concept of blockchain is expanding widely; hence the public interests are also growing on a vast scale. Major enthusiasts about blockchain are the investors and businessmen who wish for transparency and equity in the transaction. Now since blockchain is no more just a concept its application in railways is expected to smoothen the transportation.
Read more →
Total votes 3: ↑2 and ↓1 +1
Views 1.7K
Comments 0

Deploying Tarantool Cartridge applications with zero effort (Part 1)

Mail.ru Group corporate blog High performance *IT Infrastructure *Database Administration *Distributed systems *


We have already presented Tarantool Cartridge that allows you to develop and pack distributed applications. Now let's learn how to deploy and control these applications. No panic, it's all under control! We have brought together all the best practices of working with Tarantool Cartridge and wrote an Ansible role, which will deploy the package to servers, start and join instances into replica sets, configure authorization, bootstrap vshard, enable automatic failover and patch cluster configuration.

Interesting, huh? Dive in, check details under the cut.
Read more →
Total votes 29: ↑29 and ↓0 +29
Views 945
Comments 0

How to Write a Smart Contract with Python on Ontology? Part 4: Native API

Cryptography *Python *DevOps *Distributed systems *Cryptocurrencies
Tutorial
image

Earlier, I have introduced the Ontology Smart Contract in
Part 1: Blockchain & Block API and
Part 2: Storage API
Part 3: Runtime API
Today, let’s talk about how to invoke an Ontology native smart contract through the Native API. One of the most typical functions of invoking native contract is asset transfer.
Read more →
Total votes 5: ↑5 and ↓0 +5
Views 1.1K
Comments 0

Тarantool Cartridge: Sharding Lua Backend in Three Lines

Mail.ru Group corporate blog High performance *Lua *Distributed systems *Tarantool *

In Mail.ru Group, we have Tarantool, a Lua-based application server and a database united. It's fast and classy, but the resources of a single server are always limited. Vertical scaling is also not the panacea. That is why Tarantool has some tools for horizontal scaling, or the vshard module [1]. It allows you to spread data across multiple servers, but you'll have to tinker with it for a while to configure it and bolt on the business logic.

Good news: we got our share of bumps (for example, [2], [3]) and created another framework, which significantly simplifies the solution to this problem.

Тarantool Cartridge is the new framework for developing complex distributed systems. It allows you to concentrate on writing business logic instead of solving infrastructure problems. Under the cut, I will tell you how this framework works and how it could help in writing distributed services.
Read more →
Total votes 26: ↑25 and ↓1 +24
Views 1.5K
Comments 0

How to Write a Smart Contract with Python on Ontology? Part 2: Storage API

Python *Programming *DevOps *Distributed systems *Cryptocurrencies
Tutorial
image
This is an official tutorial published earlier on Ontology Medium blog
Excited to publish it for Habr readers. Feel free to ask any related questions and suggest a better format for tutorial materials

Foreword


Earlier, in Part 1, we introduced the Blockchain & Block API of Ontology’s smart contract. Today we will discuss how to use the second module: Storage API. The Storage API has five related APIs that enable addition, deletion, and changes to persistent storage in blockchain smart contracts. Here’s a brief description of the five APIs:
Read more →
Rating 0
Views 646
Comments 0

How to Write a Smart Contract with Python on Ontology? Part 1: the Blockchain & Block API

Python *Distributed systems *Cryptocurrencies
Sandbox
image

This is an official tutorial published earlier on Ontology Medium blog
Excited to publish it for Habr readers. Feel free to ask any related questions and suggest a better format for tutorial materials

Foreword


In this article, we will begin to introduce the smart contract API of Ontology. The Ontology’s smart contract API is divided into 7 modules:


In this article, we will introduce the Blockchain & Block API, which is the most basic part of the Ontology smart contract system. The Blockchain API supports basic blockchain query operations, such as obtaining the current block height, whereas the Block API supports basic block query operations, such as querying the number of transactions for a given block.

Let’s get started!

First, create a new contract in SmartX and then follow the instructions below.

1. How to Use Blockchain API


References to smart contract functions are identical to Python’s references. Developers can introduce the appropriate functions as needed. For example, the following statement introduces GetHeight, the function to get the current block height, and GetHeader, the function to get the block header.
Read more →
Total votes 20: ↑18 and ↓2 +16
Views 1.9K
Comments 0

Qrator filtering network configuration delivery system

Qrator Labs corporate blog System Analysis and Design *IT Infrastructure *Network technologies *Distributed systems *


TL;DR: Client-server architecture of our internal configuration management tool, QControl.
At its basement, there’s a two-layered transport protocol working with gzip-compressed messages without decompression between endpoints. Distributed routers and endpoints receive the configuration updates, and the protocol itself makes it possible to install intermediary localized relays. It is based on a differential backup (“recent-stable,” explained further) design and employs JMESpath query language and Jinja templating for configuration rendering.

Qrator Labs operates on and maintains a globally distributed mitigation network. Our network is anycast, based on announcing our subnets via BGP. Being a BGP anycast network physically located in several regions across the Earth makes it possible for us to process and filter illegitimate traffic closer to the Internet backbone — Tier-1 operators.

On the other hand, being a geographically distributed network bears its difficulties. Communication between the network points-of-presence (PoP) is essential for a security provider to have a coherent configuration for all network nodes and update it in a timely and cohesive manner. So to provide the best possible service for customers, we had to find a way to synchronize the configuration data between different continents reliably.
In the beginning, there was the Word… which quickly became communication protocol in need of an upgrade.
Read more →
Total votes 24: ↑23 and ↓1 +22
Views 917
Comments 0

The big interview with Martin Kleppmann: “Figuring out the future of distributed data systems”

JUG Ru Group corporate blog Programming *Conferences Distributed systems *Interview


Dr. Martin Kleppmann is a researcher in distributed systems at the University of Cambridge, and the author of the highly acclaimed «Designing Data-Intensive Applications» (O'Reilly Media, 2017). 

Kevin Scott, CTO at Microsoft once said: «This book should be required reading for software engineers. Designing Data-Intensive Applications is a rare resource that connects theory and practice to help developers make smart decisions as they design and implement data infrastructure and systems.»

Martin’s main research interests include collaboration software, CRDTs, and formal verification of distributed algorithms. Previously he was a software engineer and an entrepreneur at several Internet companies including LinkedIn and Rapportive, where he worked on large-scale data infrastructure.

Vadim Tsesko (@incubos) is a lead software engineer at Odnoklassniki who works in Core Platform team. Vadim’s scientific and engineering interests include distributed systems, data warehouses and verification of software systems.

Contents:


  • Moving from business to academic research;
  • Discussion of «Designing Data-Intensive Applications»;
  • Common sense against artificial hype and aggressive marketing;
  • Pitfalls of CAP theorem and other industry mistakes;
  • Benefits of decentralization;
  • Blockchains, Dat, IPFS, Filecoin, WebRTC;
  • New CRDTs. Formal verification with Isabelle;
  • Event sourcing. Low level approach. XA transactions; 
  • Apache Kafka, PostgreSQL, Memcached, Redis, Elasticsearch;
  • How to apply all that tools to real life;
  • Expected target audience of Martin’s talks and the Hydra conference.

Read more →
Total votes 13: ↑12 and ↓1 +11
Views 1.8K
Comments 0

Flightradar24 — how does it work? Part 2, ADS-B protocol

Python *IT Standards *Distributed systems *Popular science Systems engineering
I’m going to have a guess and say that everyone whose friends or family have ever flown on a plane, have used Flightradar24 — a free and convenient service for tracking flights in real time.

image

In the first part the basic ideas of operation were described. Now let's go further and figure out, what data is exactly transmitting and receiving between the aircraft and a ground station. We'll also decode this data using Python.
Read more →
Total votes 12: ↑12 and ↓0 +12
Views 4.6K
Comments 0

Crystal Blockchain Analytics: Investigating the Hacks and Theft Cases

Bitfury Group corporate blog Information Security *System Analysis and Design *Distributed systems *Cryptocurrencies
In this report, Bitfury shares analysis completed by its Crystal Blockchain Analytics engineering team on the movement of bitcoin from the Zaif exchange, Bithumb exchange and Electrum wallets.

Read more →
Total votes 15: ↑13 and ↓2 +11
Views 1.4K
Comments 0

Building a Private Currency Service Using Exonum

Bitfury Group corporate blog Development for e-commerce *Distributed systems *Finance in IT
Zero-knowledge proofs/arguments are an emerging cryptographic technology that promises to bring us closer to the Holy Grail of blockchain: providing data privacy and auditability.

Potential applications for zero-knowledge include, but are not limited to:


Another application for zero-knowledge proofs is helping blockchains scale. ZKPs allow for the “compressing” of computations for blockchain transactions without sacrificing security.

In this article, we describe how zero-knowledge (specifically, Bulletproofs) can be applied to build a privacy-focused service using Bitfury’s Exonum platform.

Read more →
Total votes 28: ↑28 and ↓0 +28
Views 950
Comments 1

Generic Methods in Rust: How Exonum Shifted from Iron to Actix-web

Bitfury Group corporate blog Programming *System Analysis and Design *Rust *Distributed systems *
The Rust ecosystem is still growing. As a result, new libraries with improved functionality are frequently released into the developer community, while older libraries become obsolete. When we initially designed Exonum, we used the Iron web-framework. In this article, we describe how we ported the Exonum framework to actix-web using generic programming.

Read more →
Total votes 28: ↑27 and ↓1 +26
Views 5.1K
Comments 0

The authoritative guide to Blockchain Sharding

High performance *Distributed systems *Cryptocurrencies

Hi, I'm one of the developers of the sharded blockchain Near Protocol, and in this article want to talk about what blockchain sharding is, how it is implemented, and what problems exist in blockchain sharding designs.


It is well-known that Ethereum, the most used general purpose blockchain at the time of this writing, can only process less than 20 transactions per second on the main chain. This limitation, coupled with the popularity of the network, leads to high gas prices (the cost of executing a transaction on the network) and long confirmation times; despite the fact that at the time of this writing a new block is produced approximately every 10–20 seconds the average time it actually takes for a transaction to be added to the blockchain is 1.2 minutes, according to ETH Gas Station. Low throughput, high prices, and high latency all make Ethereum not suitable to run services that need to scale with adoption.

Read more →
Total votes 15: ↑14 and ↓1 +13
Views 834
Comments 0

What to think during NALSD interview

Google Developers corporate blog High performance *System Analysis and Design *IT career Distributed systems *
There are a lot of posts about what a typical coding interview at Google looks like. But, while not as widely described and discussed, there is also quite often a system design interview. For an SRE position it’s NALSD: non-abstract large system design. The key difference between SWE and SRE interviews consists in these two letters: NA.

So, what is the difference? How to be prepared for this interview? Let’s be non-abstract, and use an example. To be more non-abstract, let’s take something from the material world, such that you won’t be asked the exact same thing at the real interview (at least, not at the Google interview) :)

So, let’s design a public library system. For the paper books, like you have seen everywhere around. The whole text below was written all at once within around one hour, to roughly show you the areas that you should be able to cover / touch during the interview. Please excuse some disorder, that’s how I think (therefore I am).
Read more →
Total votes 26: ↑24 and ↓2 +22
Views 6.3K
Comments 0

Authors' contribution