• Managing hundreds of servers for load testing: autoscaling, custom monitoring, DevOps culture

      In the previous article, I talked about our load testing infrastructure. On average, we use about 100 servers to create a load, about 150 servers to run our service. All these servers need to be created, configured, started, deleted. To do this, we use the same tools as in the production environment to reduce the amount of manual work:

      • Terraform scripts for creating and deleting a test environment;
      • Ansible scripts for configuring, updating, starting servers;
      • In-house Python scripts for dynamic scaling, depending on the load.

      Thanks to the Terraform and Ansible scripts, all operations ranging from creating instances to starting servers are performed with only six commands:

      #launch the required instances in the AWS console
      ansible-playbook deploy-config.yml #update servers versions
      ansible-playbook start-application.yml #start our app on these servers
      ansible-playbook update-test-scenario.yml --ask-vault-pass #update the JMeter test scenario if it was changed
      infrastructure-aws-cluster/jmeter_clients:~# terraform apply #create JMeter servers for creating the load
      playbook start-jmeter-server-cluster.yml #start the JMeter cluster
      ansible-playbook start-stress-test.yml #start the test
      

      Read more →
    • How to test Ansible and don't go nuts


        It is the translation of my speech at DevOps-40 2020-03-18:


        After the second commit, each code becomes legacy. It happens because the original ideas do not meet actual requirements for the system. It is not bad or good thing. It is the nature of infrastructure & agreements between people. Refactoring should align requirements & actual state. Let me call it Infrastructure as Code refactoring.

        Read more →
      • Reliable load testing with regards to unexpected nuances

          We thought about building the infrastructure for large load tests a year ago when we reached the mark of 12,000 simultaneously active online users. In three months, we made the first version of the test, which showed us the limits of the service.

          The irony is that simultaneously with the launch of the test, we reached the limits on the production server, resulting in two-hour service downtime. This further encouraged us to move from making occasional tests to establishing an effective load testing infrastructure. By infrastructure, I mean all tools for working with load testing: tools for launching the test (manual and automatic), the cluster that creates the load, a production-like cluster, metrics and reporting services, scaling services, and the code to manage it all.

          image

          Simplified, this is what our structure looks like: a collection of different servers that somehow interact with each other, each server performing specific tasks. It seemed that to build the load testing infrastructure, it was enough for us to make this diagram, take account of all interactions, and start creating test cases for each block one by one.

          This approach is right, but it would have taken many months, which was not suitable for us because of our rapid growth — over the past twelve months, we have grown from 12,000 to 100,000 simultaneously active online users. Also, we didn’t know how our service infrastructure would respond to the increased load: which blocks would become the bottleneck, and which would scale linearly?
          Read more →
        • About integration tests

            [Previously] I was talking about combinatorial complexity of integration tests in multicomponent systems, so let me remind. Let's build a simple system with only 3 components inside. It can be three independent modules and we want to provide some communication between them (message passing good enough for the purpose of the example) So we have 3! = 6 possible configuration to test. Before going deeper lets see how Quicksort fights a combinatorial complexity.

            [Quicksort] reduces complexity on each pass of a given input. So far so good. Let's imagine example with 10 elements. How many possible permutations do we have? you are right: 10! ~= 3,6 * 10^6. So on the first pass the complexity will be reduced: 5!*5!, on the next pass it will be further reduced till 2!*3!*2!*3!… after logn times we will have 1!*1!*1!...1! = 1 possible position, and our given input is sorted. Gotcha. (attentive reader can mention about worst case scenario of quicksort, but lets discuss it someday) Coming back to our example…
            Read more →
          • About Testing

              Let's talk about Testing, about unit and integration tests which everyone here, i hope, likes to write.

              Despite of lots of benefits, tests have two fundamental problems: there is no architecture at all (quality), nobody knows what is the meaning of the quantity.

              Tests have a cost. Lets imagine dialog between manager and developer:
              — John, how many hours does it take to implement?
              — Approximately one week: 1,5 day for task and 3,5 days to write tests

              So why people write tests? Let's try to build a mental model: we delivered a component, our beta testers found a bug, we wrote tests scenario. So, next time we try to deliver the same component, we already know about bad scenario and have automation to check it. So far so good. Next, we are trying to predict future bugs by writing lot's of tests and cover bad scenarios. Next, we deliver better components, users are happy. Profit. (Despite of all pros, the solution is not scalable — with every new component we will face a reality).
              Read more →
            • What does «clean code» mean in 2020?


                «Clean Code» and a clean cat

                There is nothing developers enjoy better than arguing about clean code: Dan Abramov, for example, has recently fueled the hype with his blog post, «Goodbye, Clean Code».

                However, “clean code” per se doesn’t even have a clear definition. The main book on the subject is Clean Code, where Robert «Uncle Bob» Martin states that there are perhaps as many definitions as there are programmers. But he doesn’t walk away from the fact with a conclusion that there’s no reason to discuss clean code, rather — compare several definitions and highlight general ideas. Therefore he cites the views of several outstanding programmers on what clean code is.

                So we have also become interested in what people in 2020 think of clean code. Have the views changed since the publication of the book? Do opinions vary in different IT fields (maybe backend developers perceive the idea of clean code differently from testers)?

                This spring, Uncle Bob comes to St. Petersburg to give talks at our three conferences: they are about .NET development, testing and JavaScript. Therefore, we’ve asked speakers from each of those conferences to share their opinion on clean code so we could compare the opinions of the industry experts in 2020.

                We've already published the results in Russian, and here's the English version. Since the topic is known to provoke discussions, feel free to give your own definition or argue about those already given!

                UPD: When we posted this article, Uncle Bob had our conferences in his schedule. Unfortunately, the situation has changed. We updated this post on March 12, to avoid any misunderstanding.

                Read more →
              • AdBlock has stolen the banner, but banners are not teeth — they will be back

                More
                Ads
              • Are my open-source libraries vulnerable? (2 min reading to make your life more secure)

                  The explosion of open source and issues related to it


                  The amount of open source or other third party code used in a software project is estimated as 60-90% of a codebase. Components, such as libraries, frameworks, and other software modules, almost always run with full privileges. If a vulnerable component is exploited, such an attack can facilitate serious data loss or server takeover. Applications using components with known vulnerabilities may undermine application defences and enable a range of possible attacks and impacts.



                  Conclusion: even if you perform constant security code reviews, you still might be vulnerable because of third-party components.

                  Some have tried to do this manually, but the sheer amount of work and data is growing and is time consuming, difficult, and error prone to manage. It would require several full time employees and skilled security analysts to constantly monitor all sources to stay on top.
                  Read more →
                • V&V not for vendetta



                    Over the past six years, I have worked on developing and acceptance testing of the applications for conducting and supporting clinical trials. Applications of various sizes and complexity, big data, a huge number of visualizations and views, data warehousing, ETL, etc. The products are used by doctors, clinical trials management and people who are involved in the control and monitoring of research.

                    For the applications that have or can have a direct impact on the life and health of patients, a formal acceptance testing process is required. Acceptance test results along with the rest of the documentation package are submitted for audit to the FDA (Food and Drug Administration, USA). The FDA authorizes the use of the application as a tool for monitoring and conducting clinical trials. In total, my team has developed, tested and sent to the production more than thirty applications. In this article, I will briefly talk about acceptance testing and improvement of tools used for it.

                    Note: I do not pretend to be the ultimate truth and completely understand that most of what I write about is a Captain Obvious monologue. But I hope that the described can be useful to both the entry level and the teams that encounter this in everyday work, or at least it may make happy those who have simpler processes.
                    Read more →
                  • Lessons learned from testing Over 200,000 lines of Infrastructure Code


                      IaC (Infrastructure as Code) is a modern approach and I believe that infrastructure is code. It means that we should use the same philosophy for infrastructure as for software development. If we are talking that infrastructure is code, then we should reuse practices from development for infrastructure, i.e. unit testing, pair programming, code review. Please, keep in mind this idea while reading the article.


                      Russian Version

                      Read more →
                    • Handling Objections: Static Analysis Will Take up Part of Working Time

                        bugTalking to people at conferences and in comments to articles, we face the following objection: static analysis reduces the time to detect errors, but takes up programmers' time, which negates the benefits of using it and even slows down the development process. Let's get this objection straightened out and try to show that it's groundless.
                        Read more →
                      • Configuration of the Warnings Next Generation plugin for integration with PVS-Studio

                          Picture 4


                          The PVS-Studio 7.04 release coincided with the release of the Warnings Next Generation 6.0.0 plugin for Jenkins. Right in this release Warnings NG Plugin added support of the PVS-Studio static code analyzer. This plugin visualizes data related to compiler warnings or other analysis tools in Jenkins. This article will cover in detail how to install and configure this plugin to use it with PVS-Studio, and will describe most of its features.
                          Read more →
                        • Testing SQL Server code with tSQLt

                            FYI: this article is an expanded version of my talk at SQA Days #25.

                            Based on my experience with colleagues, I can state: DB code testing is not a widely spread practice. This can be potentially dangerous. DB logic is written by human beings just like all other «usual» code. So, there can be failures which can cause negative consequences for a product, business or users. Whether these are stored procedures helping backend or it is ETL modifying data in a warehouse — there is always a risk and testing helps to decrease it. I want to tell you what tSQLt is and how it helps us to test DB code.

                            Read more →
                          • Escaping the Thicket of Tests: Building a Shortcut from a Fixture to an Assertion


                              In this article, I would like to propose an alternative to the traditional test design style using functional programming concepts in Scala. This approach was inspired by many months of pain from maintaining dozens of failing tests and a burning desire to make them more straightforward and more comprehensible.


                              Even though the code is in Scala, the proposed ideas are appropriate for developers and QA engineers who use languages supporting functional programming. You can find a Github link with the full solution and an example at the end of the article.

                              Read more →
                            • How to Make Emails and Not Mess Up: Practical Tips

                              • Tutorial


                              A developer, who first encountered generating emails, has almost no chance to write an application, that will do it correctly. Around 40% of emails, generated by corporate applications, are violating some form of standard, and due to this, there are problems with delivery and display. There are reasons for this: emails are technically more difficult than the web, and operating emails is regulated by a few hundred standards, as well as an uncountable number of generally accepted (and not as much) practices, whereas the email clients are more varied and unpredictable than browsers. Testing may significantly improve the situation, but materials that are dedicated to testing the email system, are practically non-existent.

                              Mail.ru regularly interacts with its users by email. In our projects, all the components responsible for generating emails and even individual mailings, are subject to mandatory testing. In this article, we will share our experience (learning from our mistakes).
                              Read more →
                            • Open source tool for intent-based search quality validation

                                No matter how cool your search solution is, without a reliable way of testing it you will not be able to do any improvement without breaking something valuable.

                                Even a tiny increase in overall search result quality might have a noticeable effect on conversion rate. Naturally, you can use exit-rate to judge if your search needs improvements and do required adjustments, but will you be able to make sure those changes are safe? What if improved relevancy for a specific category breaks search for several other categories?

                                image

                                Moreover, high exit-rate means no matter what you are going to do, the problem has taken place, so dozens of customers already didn’t manage to find what they were up to. Basically, the only way to proceed is to have a reliable search quality validation approach.
                                Read more →
                              • Quality as Team's responsibility. Our QA experience

                                Disclaimer: This is a translation of an article. All rights belongs to author of original article and Miro company.


                                I'm a QA Engineer in Miro. Let me tell about our experiment of transferring partially testing tasks to developers and of transforming Test Engineer role into QA (Quality assurance).


                                First briefly about our development process. We have daily releases for client side and 3 to 5 weekly releases of server side. Team have 60+ people spitted onto 10 Functional Scrum Teams.


                                I'm working in Integration team. Our tasks are:


                                • Integration of our service into external products
                                • Integration of external products into our service
                                  For example we have integrated Jira. Jira Cards — visual representation of tasks so it's useful to work with tasks not opening Jira at all.

                                  image

                                How the experiment starts


                                All starts with trivial issue. When someone of Test Engineers had sick leave then team performance was degraded significantly. Team was continued working on tasks. However when code was reached testing phase task was hold on. As a result new functionality didn't reach production in time.


                                Going onto vacation by Test Engineer is a more complex story. He/she needs to find another Test Engineer who ready to take extra tasks and conduct knowledge sharing. Going onto vacation by two Test Engineers at the sane time is not an applicable luxury.

                                Read more →
                              • Introduce Static Analysis in the Process, Don't Just Search for Bugs with It

                                  This article is an authorized translation of the original post. The translation was made with the kind help of the guys from PVS-Studio. Thank you, guys!

                                  What encouraged me to write this article is considerable quantity of materials on static analysis, which recently has been increasingly coming up. Firstly, this is a blog of PVS-Studio, which actively promotes itself on Habr posting reviews of errors, found by their tool in open source projects. PVS-Studio has recently implemented Java support, and, of course, developers from IntelliJ IDEA, whose built-in analyzer is probably the most advanced for Java today, could not stay away.

                                  When reading these reviews, I get a feeling that we are talking about a magic elixir: click the button, and here it is — the list of defects right in front of your eyes. It seems that as analyzers get more advanced, more and more bugs will be found, and products, scanned by these robots, will become better and better without any effort on our part.

                                  Well, but there are no magic elixirs. I would like to talk about what is usually not spoken in posts like «here are things that our robot can find»: what analyzers are not able to do, what's their real part and place in the process of software delivery, and how to implement the analysis properly.


                                  Ratchet (source: Wikipedia).
                                  Read more →
                                • Zen of Unit Testing

                                  • Tutorial


                                  Ability to write good unit tests is an important feature of any developer. But how to understand that your unit tests are correct? Good unit test is like a good chess game. In our case chessmen are the approaches which we are going to discuss in this post. There is no best chessman in a chess game because everything depends on the positions (and a player). Likewise, in unit testing you don't have to distinguish only one approach. In other words, you should use all approaches together to get the best result. So, if you want to win this game, then welcome under the cut.

                                  Read more →