Pull to refresh

All streams

Show first
Period
Level of difficulty

The big interview with Martin Kleppmann: “Figuring out the future of distributed data systems”

Reading time 25 min
Views 2.7K


Dr. Martin Kleppmann is a researcher in distributed systems at the University of Cambridge, and the author of the highly acclaimed «Designing Data-Intensive Applications» (O'Reilly Media, 2017). 

Kevin Scott, CTO at Microsoft once said: «This book should be required reading for software engineers. Designing Data-Intensive Applications is a rare resource that connects theory and practice to help developers make smart decisions as they design and implement data infrastructure and systems.»

Martin’s main research interests include collaboration software, CRDTs, and formal verification of distributed algorithms. Previously he was a software engineer and an entrepreneur at several Internet companies including LinkedIn and Rapportive, where he worked on large-scale data infrastructure.

Vadim Tsesko (@incubos) is a lead software engineer at Odnoklassniki who works in Core Platform team. Vadim’s scientific and engineering interests include distributed systems, data warehouses and verification of software systems.

Contents:


  • Moving from business to academic research;
  • Discussion of «Designing Data-Intensive Applications»;
  • Common sense against artificial hype and aggressive marketing;
  • Pitfalls of CAP theorem and other industry mistakes;
  • Benefits of decentralization;
  • Blockchains, Dat, IPFS, Filecoin, WebRTC;
  • New CRDTs. Formal verification with Isabelle;
  • Event sourcing. Low level approach. XA transactions; 
  • Apache Kafka, PostgreSQL, Memcached, Redis, Elasticsearch;
  • How to apply all that tools to real life;
  • Expected target audience of Martin’s talks and the Hydra conference.

Read more →
Total votes 13: ↑12 and ↓1 +11
Comments 0

Simplify working with parallel tasks in C# (updated)

Reading time 7 min
Views 22K

image


No doubts that async/await pattern has significantly simplified working with asynchronous operations in C#. However, this simplification relates only to the situation when asynchronous operations are executed consequently. If we need to execute several asynchronous operations simultaneously (e.g. we need to call several micro-services) then we do not have many built-in capabilities and most probably Task.WhenAll will be used:


Task<SomeType1> someAsyncOp1 = SomeAsyncOperation1();
Task<SomeType2> someAsyncOp2 = SomeAsyncOperation2();
Task<SomeType3> someAsyncOp3 = SomeAsyncOperation3();
Task<SomeType4> someAsyncOp4 = SomeAsyncOperation4();
await Task.WhenAll(someAsyncOp1, someAsyncOp2, someAsyncOp4);
var result = new SomeContainer(
     someAsyncOp1.Result,someAsyncOp2.Result,someAsyncOp3.Result, someAsyncOp4.Result);

This is a working solution, but it is quite verbose and not very reliable (you can forget to add a new task to “WhenAll”). I would prefer something like that instead:


var result =  await 
    from r1 in SomeAsyncOperation1()
    from r2 in SomeAsyncOperation2()
    from r3 in SomeAsyncOperation3()
    from r4 in SomeAsyncOperation4()
    select new SomeContainer(r1, r2, r3, r4);

Further I will tell you what is necessary for this construction to work...

Read more →
Total votes 13: ↑12 and ↓1 +11
Comments 4

Protocol for communication between iframe and the main window

Reading time 4 min
Views 4.8K

From time to time, developers need to establish communication between several browser tabs to be able to send messages from one tab to another and receive responses. We have also faced this need at some point.


Some solutions already exist (like, for instance, BroadcastChannel API). However, its browser support leaves a lot to be desired, so we decided to use our own library. When the library was ready, that functionality was no longer required. Nevertheless, another task emerged: communication between an iframe and the main window.


On closer examination, it turned out that two-thirds of the library would not have to be changed — only some code refactoring was necessary. The library is a communication PROTOCOL that can work with text data. It can be applied in all cases in which text is transferred, such as iframes, window.open, worker, browser tabs or WebSocket.


How it works


Currently, the protocol has two functions: sending messages and subscription to events. Any message in the protocol is a data object. For us, the main field in that object is type, which tells us what kind of message it is. The type field is an enum with the following values:

Read more →
Total votes 13: ↑12 and ↓1 +11
Comments 2

How We Find Lambda Expressions in IntelliJ IDEA

Reading time 10 min
Views 7.1K

Type Hierarchy in IntelliJ IDEACode search and navigation are important features of any IDE. In Java, one of the commonly used search options is searching for all implementations of an interface. This feature is often called Type Hierarchy, and it looks just like the image on the right.


It's inefficient to iterate over all project classes when this feature is invoked. One option is to save the complete class hierarchy in the index during compilation since the compiler builds it anyway. We do this when the compilation is run by the IDE and not delegated, for example, to Gradle. But this works only if nothing has been changed in the module after the compilation. In general, the source code is the most up-to-date information provider, and indexes are based on the source code.


Finding immediate children is a simple task if we are not dealing with a functional interface. When searching for implementations of the Foo interface, we need to find all the classes that have implements Foo and interfaces that have extends Foo, as well as new Foo(...) {...} anonymous classes. To do this, it is enough to build a syntax tree of each project file in advance, find the corresponding constructs, and add them to an index.

Read more →
Total votes 11: ↑11 and ↓0 +11
Comments 0

Announcing ML.NET 1.0

Reading time 5 min
Views 1.6K

We are excited to announce the release of ML.NET 1.0 today.  ML.NET is a free, cross-platform and open source machine learning framework designed to bring the power of machine learning (ML) into .NET applications.


ML.NET Logo


https://github.com/dotnet/machinelearning 

Get Started: http://dot.net/ml

Read more →
Total votes 13: ↑12 and ↓1 +11
Comments 0

The Role Of Augmented Reality And Virtual Reality In The NBA

Reading time 3 min
Views 1.3K
image

App builders that are looking to achieve a greater level of success will often look to areas that have yet to be explored. Augmented reality and virtual reality are opening up whole new worlds to app builders. For example, there are a number of app builders who are already looking to the NBA in this regard.

The NBA has established a reputation for themselves as a progressive league from a technological standpoint. Teams are already participating in competitive gaming, as numerous franchises have created their own e-sports teams. Now, the league is partnering with app builders to find out more about how they can leverage augmented reality and virtual reality to their benefit.
Read more →
Total votes 11: ↑11 and ↓0 +11
Comments 0

Web and Azure Tool Updates in Visual Studio 2019

Reading time 2 min
Views 865

Hopefully by now you’ve seen that Visual Studio 2019 is now generally available. As you would expect, we’ve added improvements for web and Azure development. As a starting point, Visual Studio 2019 comes with a new experience for getting started with your code and we updated the experience for creating ASP.NET and ASP.NET Core projects to match:



If you are publishing your application to Azure, you can now configure Azure App Service to use Azure Storage and Azure SQL Database instances, right from the publish profile summary page, without leaving Visual Studio. This means that for any existing web application running in App Service, you can add SQL and Storage, it is no longer limited to creation time only.

Read more →
Total votes 11: ↑11 and ↓0 +11
Comments 0

Dog Breed Identifier: Full Cycle Development from Keras Program to Android App. on Play Market

Reading time 25 min
Views 16K
With the recent progress in Neural Networks in general and image Recognition particularly, it might seem that creating an NN-based application for image recognition is a simple routine operation. Well, to some extent it is true: if you can imagine an application of image recognition, then most likely someone have already did something similar. All you need to do is to Google it up and to repeat.

However, there are still countless little details that… they are not insolvable, no. They simply take too much of your time, especially if you are a beginner. What would be of help is a step-by-step project, done right in front of you, start to end. A project that does not contain «this part is obvious so let's skip it» statements. Well, almost :)

In this tutorial we are going to walk through a Dog Breed Identifier: we will create and teach a Neural Network, then we will port it to Java for Android and publish on Google Play.

For those of you who want to see a end result, here is the link to NeuroDog App on Google Play.

Web site with my robotics: robotics.snowcron.com.
Web site with: NeuroDog User Guide.

Here is a screenshot of the program:

image

Read more →
Total votes 11: ↑11 and ↓0 +11
Comments 6

Visual Studio 2019 .NET productivity

Reading time 2 min
Views 1.7K

Your friendly neighborhood .NET productivity team (aka. Roslyn) focuses a lot on improving the .NET coding experience. Sometimes it’s the little refactorings and code fixes that really improve your workflow. You may have seen many improvements in the previews, but for all of you who were eagerly awaiting the GA release here’s a few features you may enjoy!


Read more →
Total votes 11: ↑11 and ↓0 +11
Comments 1

Web application firewalls

Reading time 6 min
Views 3.9K

Web application firewall


Web application firewalls (WAFs) are a type of intrusion detection and prevention system and might be either a hardware or software solution. It is specifically designed to inspect HTTP(s) and analyse the GET and POST requests using the appalling detection logic explained below. Web application firewall software is generally available as a web server plugin.

WAF has become extremely popular and various companies offer a variety of solutions in different price categories, from small businesses to large corporations. Modern WAF is popular because it has a wide range of covered tasks, so web application developers can rely on it for various security issues, but with the assumption that this solution cannot guarantee absolute protection. A basic WAF workflow is shown below.



Its main function is the detection and blocking of queries in which, according to WAF analysis, there are some anomalies, or an attacking vector is traced. Such an analysis should not make it difficult for legitimate users to interact with a web application, but, at the same time, it must accurately and timely detect any attempted attack. In order to implement this functionality, WAF developers usually use regular expressions, tokens, behavioural analysis, reputation analysis and machine learning, and, often, all these technologies are used together.



In addition, WAF can also provide other functionality: protection from DDoS, blocking of IP-addresses of attackers, tracking of suspicious IP-addresses, adding an HTTP-only flag to the cookie, or adding the functionality of CSRF-tokens. Each WAF is individual and has a unique internal arrangement, but there are some typical methods used for analysis.
Read more →
Total votes 15: ↑13 and ↓2 +11
Comments 0

Dependency Injection in Flutter

Reading time 4 min
Views 14K

We’re currently experimenting with Flutter while developing our side project for step challenges with colleagues. This side project should also be considered as a playground, where we can check if we can use Flutter in more serious projects. That’s why we want to use some approaches there that can look like an over-engineering for such a small project.


So one of the first questions was what can we use for dependency injection. A quick search in the internet revealed 2 libraries with positive reviews: get_it and kiwi. As get_it turned out to be a Service Locator (and I’m not a fan of this pattern), I was going to play with kiwi, which looked more promising, but then I’ve found another one library: inject.dart. It is heavily inspired by Dagger library, and as we use the latest one in our other Android projects, I’ve decided to dig into it.

Read more →
Total votes 13: ↑12 and ↓1 +11
Comments 0

What's new in CUBA 7

Reading time 11 min
Views 1.7K

What's new in CUBA 7


Three years ago we announced the second publicly available major version of the framework. CUBA 6 was the game-changing version — the licensing was turned from proprietary to Apache 2.0. Those days we couldn't even guess where it was going to bring the framework in long term. CUBA community started to grow exponentially, so we have learned a lot of possible (and sometimes impossible) ways of how developers use the framework. Now we are happy to announce CUBA 7, which, we hope, will make development more coherent and joyful for all community members from those just starting their journey in CUBA and Java to skilled enterprise developers and Java experts.


cuba

Read more →
Total votes 13: ↑12 and ↓1 +11
Comments 0

Zotero hacks: unlimited synced storage and its smooth use with rmarkdown

Reading time 7 min
Views 22K
Here is a bit refreshed translation of my 2015 blog post. The post shows how to organize a personal academic library of unlimited size for free. This is a funny case of a self written manual which I came back to multiple times myself and many many more times referred my friends to it, even non-Russian speakers who had to use Google Translator and infer the rest from screenshots. Finally, I decided to translate it adding some basic information on how to use Zotero with rmarkdown.


A brief (and hopefully unnecessary for you) intro of bibliographic managers


Bibliographic manager is a life saver in everyday academic life. I suffer almost physical pain just thinking about colleagues who for some reason never started using one — all those excel spreadsheets with favorite citations, messy folders with PDFs, constant hours lost for the joy-killing task of manual reference list formatting. Once you start using a reference manager this all becomes a happily forgotten nightmare.

Read more →
Total votes 13: ↑12 and ↓1 +11
Comments 6

Crystal Blockchain Analytics: Investigating the Hacks and Theft Cases

Reading time 8 min
Views 2.7K
In this report, Bitfury shares analysis completed by its Crystal Blockchain Analytics engineering team on the movement of bitcoin from the Zaif exchange, Bithumb exchange and Electrum wallets.

Read more →
Total votes 15: ↑13 and ↓2 +11
Comments 0

Implementation of the digital mobile-only bank in Kazakhstan

Reading time 9 min
Views 3.9K
In today’s world, Kazakhstan carries out significant improvement work for the economic situation of the country. As part of a commission from Head of State, National Bank of Kazakhstan has revoked licenses of three commercial bank pursued high-risk policy (NBK, 2017). Furthermore, over the past 10 years, 7 second-tier banks have gone into liquidation as well as the liabilities of the commercial banks as a whole have increased by 70 per cent or 5.5 trillion Tenge (Hereinafter — KZT). NBK plans to continue treatment the real economy sector, withdrawal of financial institution, restructuring its assets to well-balanced entities.

On the other side of the world, the digital and mobile banks namely, Starling bank, Monzo, N26 are becoming increasingly popular among society in the United Kingdom. As a vivid illustration, Starling Bank has increased considerably the number of customers for 8 times (up to 400 000 people) at the end of the 2018 year (Starling Bank, 2019). At the same time, one million clients have joined and used Monzo’s services (Monzo). Such tendency establishes substantial competition for all players including high-street banks in the payments market.

In consequences, the main object of the paper is identification the advantages of digital bank and illumination capabilities to deploy analogous mobile bank in Kazakhstani real financial sector.
Read more →
Total votes 17: ↑14 and ↓3 +11
Comments 5