• Critical Transcendence: .NET SDK and Apache Spark

      When Alex Garland’s series Devs (on FX and Hulu) came out this year, it gave developers their own sexy Hollywood workup. Who knew that coders could get snarled into murder plots and love triangles just for designing machine learning programs? Or that their software would cause a philosophical crisis? Sure, the average day of a developer is more code writing than murder but what a thrill to author powerful new program.


      Read more →
    • Making a Tarantool-Based Investment Business Core for Alfa-Bank


        A still from «Our Secret Universe: The Hidden Life of the Cell»

        Investment business is one of the most complex domains in the banking world. It's about not just credits, loans, and deposits — there are also securities, currencies, commodities, derivatives, and all kinds of complex stuff like structured products.

        Recently, people have become increasingly aware of their finances. More and more get involved in securities trading. Individual investment accounts have emerged not so long ago. They allow you to trade in securities and get tax credits or avoid taxes at the same time. All clients coming to us want to manage their portfolios and see their reporting on-line. Most frequently, these are multi-product portfolios, which means that people are clients of different business areas.

        Moreover, the demands of regulators, both Russian and international, also grow.

        To meet the current needs and lay a foundation for future upgrades, we've developed our Tarantool-based investment business core.
        Read more →
      • How to introduce a static code analyzer in a legacy project and not to discourage the team


          It is easy to try a static code analyzer. But it requires skills to introduce it in the development of an old large project. If the approach is incorrect, the analyzer can add work, slow down development, and demotivate the team. Let's briefly discuss how to properly integrate static analysis into the development process and start using it as part of CI/CD.
          Read more →
        • Monitoring and Logging external services in Kubernetes

            image

            Greetings to all!

            As a matter of fact I haven't found so far any generalized guide on logging and monitoring of metrics from external systems to Kubernetes on the web. Here I want to share with you my own version. First of all it is supposed that you have already got working Prometheus and other services.

            As an example I use third-party data for stateful service RDBMS PostgreSQL in a Docker container. In our company we also use the Helm package manager, and you'll find how it works below. You can see same examples in this article below. You can see same examples in this article below. For a complete solution we create a nested chart.
            Read more →
          • HTTP headers checker

            • Translation
            For any site, it is important to properly configure the HTTP headers. A lot of articles have been written on the subject of headings. Here we have summarized the lessons learned, the RFC documentation. Some of the headings are mandatory, some are obsolete, some can be confusing and contradictory. We did a parsing to automatically check the HTTP headers of the web server.

            Correct HTTP headers increase security and trust in the site, including from search engines, can affect the site’s position in Yandex and Google, save server resources, reduce server load, thereby increasing the server response speed, which again affects the ranking of the site in the search, save money on payment powerful hosting, which may not be required for the site when configured correctly.
            Read more →
          • Y messenger Manifesto

            Y messenger - decentralized end-2-end encrypted messenger


            We are a team of independent developers. We have created a new messenger, the purpose of which is to solve the critical problems of the modern Internet and the modes of communication it provides. We see users become hostages to the services they have grown accustomed to and we see corporations exploiting their users and controlling them. And we don’t like it. We believe the Internet should be different.
            In this Manifesto, we disclose our vision of the Internet and describe what we have done to make it better. If you share our ideas — join us. Together we can achieve more than each of us can alone.

            Read more →
          • Ads
            AdBlock has stolen the banner, but banners are not teeth — they will be back

            More
          • Looking back at 3 months of the global traffic shapeshifting

              image
              There would be no TL;DR in this article, sorry.

              Those have been three months that genuinely changed the world. An entire lifeline passed from February, 1, when the coronavirus pandemics just started to spread outside of China and European countries were about to react, to April, 30, when nations were locked down in quarantine measures almost all over the entire world. We want to take a look at the repercussions, cyclic nature of the reaction and, of course, provide DDoS attacks and BGP incidents overview on a timeframe of three months.

              In general, there seems to be an objective pattern in almost every country’s shift into the quarantine lockdown.
              Read more →
            • The QC House of Cards

                There’s Gold in Them Thar Hills

                Gold rushes can make people crazy. 1848 was enough of an indicator of that. When Sam Brannan announced to the world: ‘Gold! Gold! Gold from the American River!’, half the world’s population (or so it seemed to the tiny California population which lived there at the time) descended on the soon to be the newest state of the union.

                San Francisco, before a small hamlet with a few hundred pioneers living there, became a centre of vice, murder and debauchery overnight.

                image

                Two hundred years before tulip mania hit Europe, and like in California with its argonauts or 49ers, it impoverished more than it made rich. In the early 2000s, too, the Dot.Com bubble created a speculative tendency in people when irrationality took over all reason.
                Read more →
              • External Interrupts in the x86 system. Part 2. Linux kernel boot options

                • Tutorial

                In the last part we discussed evolution of the interrupt delivery process from the devices in the x86 system (PIC → APIC → MSI), general theory, and all the necessary terminology.


                In this practical part we will look at how to roll back to the use of obsolete methods of interrupt delivery in Linux, and in particular we will look at Linux kernel boot options:


                • pci=nomsi
                • noapic
                • nolapic

                Also we will look at the order in which the OS looks for interrupt routing tables (ACPI/MPtable/$PIR) and what the impact is from the following boot options:


                • pci=noacpi
                • acpi=noirq
                • acpi=off

                You've probably used some combination of these options when one of the devices in your system hasn't worked correctly because of an interrupt problem. We'll go through these options and find out what they do and how they change the kernel '/proc/interrupts' interface output.

                Read more →
              • How to Deploy Hyperledger Fabric 1.4 to AWS

                  Hyperledger Fabric is an open-source blockchain platform from the Linux Foundation. With its help, you can create blockchain applications and control access and permissions for data in the blockchain. The most popular reason to use Hyperledger Fabric is to create a private blockchain network or set restrictions on what transactions each side can see.

                  The documentation of Hyperledger Fabric describes the creation of a blockchain network on one machine using Docker, where each member of the network is a separate container. But the process of setting up a network on several physical or virtual machines is not well described. Therefore, we decided to discuss this in the article.

                  Our blockchain application is a service for storing the medical data of school and preschool children. The data must remain unchanged. It will be presented in the form of information on vaccination and on the patient's agreement with a particular doctor. The members of the network include Parents (an organization that represents the interests of parents), Hospital (a hospital which represents pediatricians with whom parents can enter into an agreement to care for their children) and Kindergarten (a kindergarten that can request health and vaccinations reports of a child from the hospital).
                  Read more →
                • Bcache against Flashcache for Ceph Object Storage


                    Fast SSDs are getting cheaper every year, but they are still smaller and more expensive than traditional HDD drives. But HDDs have much higher latency and are easily saturated. However, we want to achieve low latency for the storage system, and a high capacity too. There’s a well-known practice of optimizing performance for big and slow devices — caching. As most of the data on a disk is not accessed most of the time but some percentage of it is accessed frequently, we can achieve a higher quality of service by using a small cache.

                    Server hardware and operating systems have a lot of caches working on different levels. Linux has a page cache for block devices, a dirent cache and an inode cache on the filesystem layer. Disks have their own cache inside. CPUs have caches. So, why not add one more persistent cache layer for a slow disk?
                    Read more →
                  • Powershell way to exclude folder from Defender list

                    • Tutorial
                    To exclude some folder from the defender list can be used cmdlet Set-MpPreference
                    Manly, now you can stop reading, go to the PowerShell and update excludes:)
                    But if you are want to see how to perform this in more detail — welcome under the habrcut.

                    UPD: Added a script to add selected folder to excluded Defender locations (at the end of the post)
                    Read more →
                  • Managing hundreds of servers for load testing: autoscaling, custom monitoring, DevOps culture

                      In the previous article, I talked about our load testing infrastructure. On average, we use about 100 servers to create a load, about 150 servers to run our service. All these servers need to be created, configured, started, deleted. To do this, we use the same tools as in the production environment to reduce the amount of manual work:

                      • Terraform scripts for creating and deleting a test environment;
                      • Ansible scripts for configuring, updating, starting servers;
                      • In-house Python scripts for dynamic scaling, depending on the load.

                      Thanks to the Terraform and Ansible scripts, all operations ranging from creating instances to starting servers are performed with only six commands:

                      #launch the required instances in the AWS console
                      ansible-playbook deploy-config.yml #update servers versions
                      ansible-playbook start-application.yml #start our app on these servers
                      ansible-playbook update-test-scenario.yml --ask-vault-pass #update the JMeter test scenario if it was changed
                      infrastructure-aws-cluster/jmeter_clients:~# terraform apply #create JMeter servers for creating the load
                      playbook start-jmeter-server-cluster.yml #start the JMeter cluster
                      ansible-playbook start-stress-test.yml #start the test
                      

                      Read more →
                    • Ansible: CoreOS to CentOS, 18 months long journey


                        There was a custom configuration management solution.


                        I would like to share the story about a project. The project used to use a custom configuration management solution. Migration lasted 18 months. You can ask me 'Why?'. There are some answers below about changing processes, agreements and workflows.

                        Read more →
                      • How to test Ansible and don't go nuts


                          It is the translation of my speech at DevOps-40 2020-03-18:


                          After the second commit, each code becomes legacy. It happens because the original ideas do not meet actual requirements for the system. It is not bad or good thing. It is the nature of infrastructure & agreements between people. Refactoring should align requirements & actual state. Let me call it Infrastructure as Code refactoring.

                          Read more →
                        • Porting packages to buildroot using the Zabbix example

                          • Tutorial


                          The basics of porting


                          Originally, Buildroot offers a limited number of packages. It makes sense — there is everything you need, but any other packages can be added.


                          To add a package, create 2 description files, an optional checksum file, and add a link to the package in the general package list. There are hooks at different stages of the build. At the same time, Buildroot can recognize the needed type of packages:

                          Read more →
                        • When the COVID-19 pandemic will end

                            Dear all,


                            I am the head of Data Science at British Transport Police, and one of our department tasks is to efficiently allocate staff, depending on the crime rates, which correlate to passenger flow. As you understand, the passenger flow will undertake significant change as soon as the Government decides to cancel quarantine or stop some limitations. The question naturally arises: when will the pandemic end and how to prepare for a return to normal life.

                            Read more →
                          • Scientists Turn a Quantum Computer into a Time Machine — At least, for a Second…

                              Scientists said they were able to return the state of a quantum computer a fraction of a second into the past, according to a university press release. The researchers, who are from the Moscow Institute of Physics and Technology, along with colleagues from the U.S. and Switzerland, also calculated the probability that an electron in empty interstellar space will spontaneously travel back into its recent past. The study came out recently in Scientific Reports.
                              “This is one in a series of papers on the possibility of violating the second law of thermodynamics. That law is closely related to the notion of the arrow of time that posits the one-way direction of time: from the past to the future,” commented the study’s lead author Gordey Lesovik, who heads the Laboratory of the Physics of Quantum Information Technology at MIPT.

                              While the researchers don’t expect you to take a trip back to the high school prom just yet, they added that the time reversal algorithm could prove useful for making quantum computers more precise.

                              “Our algorithm could be updated and used to test programs written for quantum computers and eliminate noise and errors,” Lebedev explained.

                              The researchers said that the work builds on some earlier work that recently garnered headlines.

                              “We began by describing a so-called local perpetual motion machine of the second kind. Then, in December, we published a paper that discusses the violation of the second law via a device called a Maxwell’s demon,” Lesovik said. “The most recent paper approaches the same problem from a third angle: We have artificially created a state that evolves in a direction opposite to that of the thermodynamic arrow of time.”
                              Read more →