• 11 Kubernetes implementation mistakes – and how to avoid them

      I manage a team that designs and introduces in-house Kubernetes aaS at Mail.ru Cloud Solutions. And we often see a lack of understanding as to this technology, so I’d like to talk about common strategic mistakes at Kubernetes implementation in major projects.

      Most of the problems arise because the technology is quite sophisticated. There are unobvious implementation and operation challenges, as well as poorly used advantages, all of those resulting in money loss. Another issue is the global lack of knowledge and experience with Kubernetes. Learning its use by the book can be tricky, and hiring qualified staff can be challenging. All the hype complicates Kubernetes-related decision making. Curiously enough, Kubernetes is often implemented rather formally – just for it to be there and make their lives better in some way.

      Hopefully, this post will help you to make a decision you will feel proud of later (and won’t regret or feel like building a time machine to undo it).
      Read more →
    • Building projects (CI/CD), instruments

        In some projects, the build script is playing the role of Cinderella. The team focuses its main effort on code development. And the build process itself could be handled by people who are far from development (for example, those responsible for operation or deployment). If the build script works somehow, then everyone prefers not to touch it, and no one ever is thinking about optimization. However, in large heterogeneous projects, the build process could be quite complex, and it is possible to approach it as an independent project.If you treat the build script as a secondary unimportant project, then the result will be an indigestible imperative script, the support of which will be rather difficult.

        In this note we will take look at the criteria by which we chose the toolkit, and in the next one — how we use this toolkit. (There is also a Russian version.)

        CI/CD (opensource.com)

        Read more →
      • How to Get Nice Error Reports Using SARIF in GitHub

          Let's say you use GitHub, write code, and do other fun stuff. You also use a static analyzer to enhance your work quality and optimize the timing. Once you come up with an idea - why not view the errors that the analyzer gave right in GitHub? Yeah, and also it would be great if it looked nice. So, what should you do? The answer is very simple. SARIF is right for you. This article will cover what SARIF is and how to set it up. Enjoy the reading!

          Читать далее
        • Prometheus in Action: from default counters to SLO-related queries

          • Tutorial

          All Prometheus metrics are based on time series - streams of timestamped values belonging to the same metric. Each time series is uniquely identified by its metric name and optional key-value pairs called labels. The metric name specifies some characteristics of the measured system, such as http_requests_total - the total number of received HTTP requests. In practice, you often will be interested in some subset of the values of a metric, for example, in the number of requests received by a particular endpoint; and here is where the labels come in handy. We can partition a metric by adding endpoint label and see the statics for a particular endpoint: http_requests_total{endpoint="api/status"}. Every metric has two automatically created labels: job_name and instance. We see their roles in the next section.

          Prometheus provides a functional query language called PromQL. The result of the query might be evaluated to one of four types:

          Scalar (aka float)

          String (currently unused)

          Instant Vector - a set of time series that have exactly one value per timestamp.

          Range Vector - a set of time series that have a range of values between two timestamps.

          At first glance, Instant Vector might look like an array, and Range Vector as a matrix.

          If that would be the case, then a Range Vector for a single time series "downgrades" to an Instant Vector. However, that's not the case:

          Read more
        • Distributed Tracing for Microservice Architecture

          • Tutorial

          What is distributed tracing? Distributed tracing is a method used to profile and monitor applications, especially those built using a microservices architecture. Distributed tracing helps pinpoint where failures occur and what causes poor performance.

          Let’s have a look at a simple prototype. A user fetches information about a shipment from `logistic` service. logistic service does some computation and fetches the data from a database. logistic service doesn’t know the actual status of the shipment, so it has to fetch the updated status from another service `tracking`. `tracking` service also needs to fetch the data from a database and to do some computation.

          In the screenshot below, we see a whole life cycle of the request issued to `logistics` service:

          Read more
        • Ads
          AdBlock has stolen the banner, but banners are not teeth — they will be back

        • The Rules for Data Processing Pipeline Builders

            "Come, let us make bricks, and burn them thoroughly."
            – legendary builders

            You may have noticed by 2020 that data is eating the world. And whenever any reasonable amount of data needs processing, a complicated multi-stage data processing pipeline will be involved.

            At Bumble — the parent company operating Badoo and Bumble apps — we apply hundreds of data transforming steps while processing our data sources: a high volume of user-generated events, production databases and external systems. This all adds up to quite a complex system! And just as with any other engineering system, unless carefully maintained, pipelines tend to turn into a house of cards — failing daily, requiring manual data fixes and constant monitoring.

            For this reason, I want to share certain good engineering practises with you, ones that make it possible to build scalable data processing pipelines from composable steps. While some engineers understand such rules intuitively, I had to learn them by doing, making mistakes, fixing, sweating and fixing things again…

            So behold! I bring you my favourite Rules for Data Processing Pipeline Builders.

            Read more →
          • Patroni cluster (with Zookeeper) in a docker swarm on a local machine

            • Tutorial

            There probably is no way one who stores some crucial data (and well, in particular, using SQL databases) can possibly dodge from thoughts of building some kind of safe cluster, distant guardian to protect consistency and availability at all times. Even if the main server with your precious database gets knocked out deadly - the show must go on, right? This basically means the database must still be available and data be up-to-date with the one on the failed server.

            As you might have noticed, there are dozens of ways to go and Patroni is just one of them. There is plenty of articles providing a more or less detailed comparison of the options available, so I assume I'm free to skip the part of luring you into Patroni's side. Let's start off from the point where among others you are already leaning towards Patroni and are willing to try that out in a more or less real-case setup.

            I am not a DevOps engineer originally so when the need for the high-availability cluster arose and I went on I would catch every single bump on the road. Hope this tutorial will help you out to get the job done with ease! If you don't want any more explanations, jump right in. Otherwise, you might want to read some more notes on the setup I went on with.

            Read more
          • OPPO, Huawei, Xiaomi. Chinese app stores join forces to take on Google

              Major players in the Chinese app market are joining forces to take on the almighty Google Play store. Xiaomi, Oppo and Vivo are reported to launch the Global Developer Service Alliance (GDSA), a platform allowing Android developers to publish their apps in the partnering stores from one upload.

              The GDSA is expected to launch in nine countries—including India, Indonesia, Malaysia, Russia, Spain, Thailand, the Philippines, and Vietnam—although paid app support may vary across the regions. Canalys’ Nicole Peng explains the wide reach of this alliance:

              By forming this alliance each company will be looking to leverage the others’ advantages in different regions, with Xiaomi’s strong user base in India, Vivo and Oppo in Southeast Asia, and Huawei in Europe. 

              Читать далее
            • Agreements as Code: how to refactor IaC and save your sanity?

                Before we start, I'd like to get on the same page with you. So, could you please answer? How much time will it take to:

                • Create a new environment for testing?
                • Update java & OS in the docker image?
                • Grant access to servers?

                There is the spoiler from the TechLeadConf. Unfortunately, it's in Russian

                It will take longer than you expect. I will explain why.

                Read more →
              • Mysql 8.x Group Replication (Master-Slave) with Docker Compose

                  This post is handling the following situation - how to setup up simple Mysql services with group replication being dockerized. In our case, we’ll take the latest Mysql (version 8.x.x)

                  FYI: all mentioned code (worked and tested manually) located here.

                  I will skip not interested steps like ‘what is Mysql, Docker and why we choose them, etc’. We want to set up possibly trouble proof DB. That’s our plan.

                  Read more →
                • InterSystems IRIS – the All-Purpose Universal Platform for Real-Time AI/ML

                    Author: Sergey Lukyanchikov, Sales Engineer at InterSystems

                    Challenges of real-time AI/ML computations

                    We will start from the examples that we faced as Data Science practice at InterSystems:

                    • A “high-load” customer portal is integrated with an online recommendation system. The plan is to reconfigure promo campaigns at the level of the entire retail network (we will assume that instead of a “flat” promo campaign master there will be used a “segment-tactic” matrix). What will happen to the recommender mechanisms? What will happen to data feeds and updates into the recommender mechanisms (the volume of input data having increased 25000 times)? What will happen to recommendation rule generation setup (the need to reduce 1000 times the recommendation rule filtering threshold due to a thousandfold increase of the volume and “assortment” of the rules generated)?
                    • An equipment health monitoring system uses “manual” data sample feeds. Now it is connected to a SCADA system that transmits thousands of process parameter readings each second. What will happen to the monitoring system (will it be able to handle equipment health monitoring on a second-by-second basis)? What will happen once the input data receives a new bloc of several hundreds of columns with data sensor readings recently implemented in the SCADA system (will it be necessary, and for how long, to shut down the monitoring system to integrate the new sensor data in the analysis)?
                    • A complex of AI/ML mechanisms (recommendation, monitoring, forecasting) depend on each other’s results. How many man-hours will it take every month to adapt those AI/ML mechanisms’ functioning to changes in the input data? What is the overall “delay” in supporting business decision making by the AI/ML mechanisms (the refresh frequency of supporting information against the feed frequency of new input data)?

                    Read more →
                  • Data Science vs AI: All You Need To Know

                      What do these terms mean? And what is the difference?


                      Data Science and Artificial Intelligence are creating a lot of buzzes these days. But what do these terms mean? And what is the difference between them?

                      While the terms Data Science and Artificial Intelligence (AI) comes under the same domain and are inter-connected to each other, they have their specific applications and meaning.

                      There’s no slowing down the spread of AI and data science. Many big tech giants are extensively investing in these technologies. As per the recent survey, it is estimated that artificial intelligence could add $15.7 trillion to the global economy by 2030.

                      Through this piece of writing, I will be explaining about the AI and data science concepts and their differences in detail. So, without wasting any more time, let’s get started!
                      Read more →
                    • Lossless ElasticSearch data migration

                      • Translation

                      Academic data warehouse design recommends keeping everything in a normalized form, with links between. Then the roll forward of changes in relational math will provide a reliable repository with transaction support. Atomicity, Consistency, Isolation, Durability — that's all. In other words, the storage is explicitly built to safely update the data. But it is not optimal for searching, especially with a broad gesture on the tables and fields. We need indices, a lot of indices! Volumes expand, recording slows down. SQL LIKE can not be indexed, and JOIN GROUP BY sends us to meditate in the query planner.

                      Read more →
                    • PVS-Studio: analyzing pull requests in Azure DevOps using self-hosted agents

                        Static code analysis is most effective when changing a project, as errors are always more difficult to fix in the future than at an early stage. We continue expanding the options for using PVS-Studio in continuous development systems. This time, we'll show you how to configure pull request analysis using self-hosted agents in Microsoft Azure DevOps, using the example of the Minetest game.
                        Read more →
                      • PVS-Studio and Continuous Integration: TeamCity. Analysis of the Open RollerCoaster Tycoon 2 project

                          One of the most relevant scenarios for using the PVS-Studio analyzer is its integration into CI systems. Even though a project analysis by PVS-Studio can already be embedded with just a few commands into almost any continuous integration system, we continue to make this process even more convenient. PVS-Studio now supports converting the analyzer output to the TeamCity format-TeamCity Inspections Type. Let's see how it works.
                          Read more →