All streams

Great Visual Studio extensions share a few key features that sets them apart from the rest. They look and feel well crafted, are performant and reliable, do what they advertise to perfection, and blend in naturally among Visual Studio’s own features.

To make it easier to write great extensions, we’ve worked with the extensibility community to come up with a simple checklist to follow. There’s even a GitHub issue template you can use so you remember to go through the checklist.

Some time ago, in a discussion on one of SObjectizer's releases, we were asked: "Is it possible to make a DSL to describe a data-processing pipeline?" In other words, is it possible to write something like that:

A | B | C | D

and get a working pipeline where messages are going from A to B, and then to C, and then to D. With control that B receives exactly that type that A returns. And C receives exactly that type that B returns. And so on.

It was an interesting task with a surprisingly simple solution. For example, that's how the creation of a pipeline can look like:

auto pipeline = make_pipeline(env, stage(A) | stage(B) | stage(C) | stage(D));

Or, in a more complex case (that will be discussed below):

auto pipeline = make_pipeline( sobj.environment(),
        stage(validation) | stage(conversion) | broadcast(
            stage(archiving),
            stage(distribution),
            stage(range_checking) | stage(alarm_detector{}) | broadcast(
                stage(alarm_initiator),
                stage( []( const alarm_detected & v ) {
                        alarm_distribution( cerr, v );
                    } )
                )
            ) );

In this article, we'll speak about the implementation of such pipeline DSL. We'll discuss mostly parts related to stage(), broadcast() and operator|() functions with several examples of usage of C++ templates. So I hope it will be interesting even for readers who don't know about SObjectizer (if you never heard of SObjectizer here is an overview of this tool).

What is Swoft?

Swoft is a PHP microservices coroutine framework based on the Swoole extension. Like Go, Swoft has a built-in coroutine web server and a common coroutine client and is resident in memory, independent of traditional PHP-FPM. There are similar Go language operations, similar to the Spring Cloud framework flexible annotations, powerful global dependency injection container, comprehensive service governance, flexible and powerful AOP, standard PSR specification implementation and so on.

Through three years of accumulation and direction exploration, Swoft has made Swoft the Spring Cloud in the PHP world, which is the best choice for PHP's high-performance framework and microservices management.

Elegant service governance

Swoft officially recommends that developers use service mesh patterns, such as the Istio/Envoy framework, to separate business and service governance, but Swoft also provides a set of microservices components for small and medium-sized businesses to quickly build microservices.

Readers of our articles occasionally note that the PVS-Studio static code analyzer detects a large number of errors that are insignificant and don't affect the application. It is really so. For the most part, important bugs have already been fixed due to manual testing, user feedback, and other expensive methods. At the same time, many of these errors could have been found at the code writing stage and corrected with minimal loss of time, reputation and money. This article will provide several examples of real errors, which could have been immediately fixed, if project authors had used static code analysis.

Hi everybody! I’m a research engineer at the Mail.ru Group computer vision team. In this article, I’m going to tell a story of how we’ve created AI-based photo restoration project for old military photos. What is «photo restoration»? It consists of three steps:

• we find all the image defects: fractures, scuffs, holes;
  
• we inpaint the discovered defects, based on the pixel values around them;
  
• we colorize the image.
  

Further, I’ll describe every step of photo restoration and tell you how we got our data, what nets we trained, what we accomplished, and what mistakes we made.

The other day Google revealed the sources of the robots.txt parser. Why not give a run for the already far and wide checked project using PVS-Studio and possibly find a bug. So said so done. But I wish we could find something meaningful. Well, then let it be just a reason to give full marks for Google developers.

robots.txt — is an index file that contains rules for search robots. It works for https, http and FTP protocols. Google made the parser of the robots.txt file available for everyone. Read more about this news here: Google opens the source code of the robots.txt parser

Some time ago, the creator of Lua programming language, Roberto Ierusalimschy, visited our Moscow office. We asked him some questions that we prepared with the participation of Habr.com users also. And finally, we’d like to share full-text version of this interview.

For any webmaster interested to know where his users came from there are two methods available:

• compare users IP address to a database of longitude/latitude or use an online API for this purpose;
• or use a W3C HTML Geolocation API.

What's this about

After working on different projects, I've noticed that every one of them had some common problems, regardless of domain, architecture, code convention and so on. Those problems weren't challenging, just a tedious routine: making sure you didn't miss anything stupid and obvious. Instead of doing this routine on a daily basis I became obsessed with seeking solution: some development approach or code convention or whatever that will help me to design a project in a way that will prevent those problems from happening, so I can focus on interesting stuff. That's the goal of this article: to describe those problems and show you that mix of tools and approaches that I found to solve them.

It is a new selection of tips and tricks about Python and programming from my Telegram-channel @pythonetc.

← Previous publications


The \ symbol in regular string have special meaning. \t is tab character, \r is carriage return and so on.

You can use raw-strings to disable this behaviour. r'\t' is just backslash and t.

You obviously can’t use ' inside r'...'. However, it still can be escaped by \, but \ is preserved in the string:

inb4: This is not another "setting up" a new project with Vue and TypeScript tutorial. Let's do some deep dive into more complex topics!

typescript is awesome. Vue is awesome. No doubt, that a lot of people try to bundle them together. But, due to different reasons, it is hard to really type your Vue app. Let's find out what are the problems and what can be done to solve them (or at least minimize the impact).

A memory problem may arise when a large number of objects are active in RAM during the execution of a program, especially if there are restrictions on the total amount of available memory.

Below is an overview of some methods of reducing the size of objects, which can significantly reduce the amount of RAM needed for programs in pure Python.

Note: This is english version of my original post (in russian).

Hi All,

I’m in the process of implementing a new simple microservices-based project as an example of a step-by-step guide for those who had a hard time with a microservices architecture and are still looking for “another” good reference. Also, I would really appreciate thought through feedback and proposal to make this project a high-quality chunk of work.

There are tons of articles and source code examples. But, unfortunately, I could not find any reference with simple step-by-step instructions, without doing a deep dive into Docker, Event Store, a multitude of configurations, cloud deployment stuff, etc. I cloned several projects and tried to start playing with them, but you know, only God knows how to start them, which dependencies are missing and why all those scripts are failing with thousands of ERRORS.

For example, this eShop project from Microsoft contains all we need, but it is not so simple to figure out what is going on there, SQL database connection strings, Docker scripts fail, no How-Tos and I’m not sure it is super-simple architecture you need to start with.

At the moment, cloud CI systems are a highly-demanded service. In this article, we'll tell you how to integrate analysis of source code into a CI cloud platform with the tools that are already available in PVS-Studio. As an example we'll use the Travis CI service.

Many of our articles are focused on anything, but not the PVS-Studio tool itself. Whereas we do a lot to make its usage convenient for developers. Nevertheless, our efforts are often concealed behind the scenes. I decided to remedy this situation and tell you about the PVS-Studio plugin for Visual Studio. If you use Visual Studio, this article is for you.

Dr. Martin Kleppmann is a researcher in distributed systems at the University of Cambridge, and the author of the highly acclaimed «Designing Data-Intensive Applications» (O'Reilly Media, 2017). 

Kevin Scott, CTO at Microsoft once said: «This book should be required reading for software engineers. Designing Data-Intensive Applications is a rare resource that connects theory and practice to help developers make smart decisions as they design and implement data infrastructure and systems.»

Martin’s main research interests include collaboration software, CRDTs, and formal verification of distributed algorithms. Previously he was a software engineer and an entrepreneur at several Internet companies including LinkedIn and Rapportive, where he worked on large-scale data infrastructure.

Vadim Tsesko (@incubos) is a lead software engineer at Odnoklassniki who works in Core Platform team. Vadim’s scientific and engineering interests include distributed systems, data warehouses and verification of software systems.

Contents:

• Moving from business to academic research;
  
• Discussion of «Designing Data-Intensive Applications»;
  
• Common sense against artificial hype and aggressive marketing;
  
• Pitfalls of CAP theorem and other industry mistakes;
  
• Benefits of decentralization;
  
• Blockchains, Dat, IPFS, Filecoin, WebRTC;
  
• New CRDTs. Formal verification with Isabelle;
  
• Event sourcing. Low level approach. XA transactions; 
  
• Apache Kafka, PostgreSQL, Memcached, Redis, Elasticsearch;
  
• How to apply all that tools to real life;
  
• Expected target audience of Martin’s talks and the Hydra conference.
  

No doubts that async/await pattern has significantly simplified working with asynchronous operations in C#. However, this simplification relates only to the situation when asynchronous operations are executed consequently. If we need to execute several asynchronous operations simultaneously (e.g. we need to call several micro-services) then we do not have many built-in capabilities and most probably Task.WhenAll will be used:

Task<SomeType1> someAsyncOp1 = SomeAsyncOperation1();
Task<SomeType2> someAsyncOp2 = SomeAsyncOperation2();
Task<SomeType3> someAsyncOp3 = SomeAsyncOperation3();
Task<SomeType4> someAsyncOp4 = SomeAsyncOperation4();
await Task.WhenAll(someAsyncOp1, someAsyncOp2, someAsyncOp4);
var result = new SomeContainer(
     someAsyncOp1.Result,someAsyncOp2.Result,someAsyncOp3.Result, someAsyncOp4.Result);

This is a working solution, but it is quite verbose and not very reliable (you can forget to add a new task to “WhenAll”). I would prefer something like that instead:

var result =  await 
    from r1 in SomeAsyncOperation1()
    from r2 in SomeAsyncOperation2()
    from r3 in SomeAsyncOperation3()
    from r4 in SomeAsyncOperation4()
    select new SomeContainer(r1, r2, r3, r4);

Further I will tell you what is necessary for this construction to work...

During code analysis, PVS-Studio analyzes the data flow and operates variable values. Values are taken from constants or derived from conditional expressions. We call them virtual values. Recently, we have refined them in order to work with multi-character constants and this has become the reason to create a new diagnostic rule.

Introduction

Multi-character-literals are implementation-defined, so different compilers can encode them in different ways. For example, GCC and Clang set a value, based on the order of the symbols in the literal, while MSVC moves them depending on the symbol's type (regular or escape).

Hello Habr! Now that is summer vacation season. Many of you will take a flight to a place far from your everyday routine at home. Before this hot vacation season starts, we should discuss an interesting and useful method on how to save money using a VPN.

One of the easiest ways to see the value in this is looking at car rentals while on vacation.

When you run queries in ClickHouse, you might notice that the profiler often shows the LZ_decompress_fast function near the top. What is going on? This question had us wondering how to choose the best compression algorithm.

ClickHouse stores data in compressed form. When running queries, ClickHouse tries to do as little as possible, in order to conserve CPU resources. In many cases, all the potentially time-consuming computations are already well optimized, plus the user wrote a well thought-out query. Then all that's left to do is to perform decompression.



So why does LZ4 decompression becomes a bottleneck? LZ4 seems like an extremely light algorithm: the data decompression rate is usually from 1 to 3 GB/s per processor core, depending on the data. This is much faster than the typical disk subsystem. Moreover, we use all available CPU cores, and decompression scales linearly across all physical cores.