• Waking up of the sleeping institution: how we taking off training-wheels from Python in REAPER

    • Translation

    Image for taking attention


    About a week I wonder around the idea of this article, mainly, because of missing the content here and tidiness of the coronavirus-themes. But, when robotic-harvester, hacking zip with bitcoins and other cool articles released — I decided not to publish crude material.


    However, unexpectedly today the maintainer of this review hero showed up from the lockdown and, several hours ago reapy v0.6.0 has been published on PyPi. Under the cut — the last change log, which contains (to my pleasure) no line where I have not been involved one way, or another.


    Finally: why reapy is needed and how Python works inside REAPER.

    Read more →
  • Checking the GCC 10 Compiler with PVS-Studio

      PVS-Studo vs GCC 10

      The GCC compiler is written with copious use of macros. Another check of the GCC code using PVS-Studio once again confirms the opinion of our team that macros are evil in the flesh. Not only does the static analyzer struggle with reviewing such code, but also a developer. GCC developers are certainly used to the project and are well versed in it. Nonetheless, it is very difficult to understand something on the third hand. Actually, due to macros, it was not possible to fully perform code checking. However, the PVS-Studio analyzer, as always, showed that it can find errors even in compilers.
      Read more →
    • How does strange code hide errors? TensorFlow.NET Analysis

        PVS-Studio and TensorFlow.NET

        Static analysis is an extremely useful tool for any developer, as it helps to find in time not only errors, but also suspicious and strange code fragments that may cause bewilderment of programmers who will have to work with it in the future. This idea will be demonstrated by the analysis of the TensorFlow.NET open C# project, developed for working with the popular TensorFlow machine learning library.
        Read more →
      • Checking the Code of Zephyr Operating System

          PVS-Studio and Zephyr

          Some time ago we announced PVS-Studio's new feature that enabled it to integrate into PlatformIO. Naturally, our team kept in touch with the PlatformIO team while working on that feature, and they suggested that we check the real-time operating system Zephyr to see if we could find any interesting bugs in its code. We thought it was a good idea, and so here's this article about the check results.
          Read more →
        • OpenToonz: inside and out

            Picture 1

            Almost four years have passed since the PVS-Studio team checked the OpenToonz source code. This project is a very powerful tool for creating two-dimensional animation. Since the last check, with its help, such animated works as Mary and the Witch Flower, Batman-Ninja, Promare and others were created. As large studios continue using Toonz, why not check the quality of the source code again?
            Read more →
          • GitHub repositories tracker and its «for convenience» tricks

            Work in several GitHub repositories (repos) brings inconveniences. Repos are separated from each other, open and closed issues live in different lists, linked pull requests (PRs) can't be seen until opening an issue — our team works in more than 70 repos, so I learned that hard. And started to write repos tracker on Python. It's close to the final, I'd like to share it (it's completely free), and few tricks I used in progress.

            What is that


            Scraper tracks several GitHub repos in a single Google Sheet (GS) table. You can see all of the opened and done issues, related PRs, priorities, your teammates comments, use coloring, filtering, sorting and other GS functions. That's a good integration, here is how it looks:

            image

            How does it work, shortly


            There is Spreadsheet() class which contain several Sheet() objects, each of which have it's own configurations, described in config.py. When Scraper updates a sheet, it loads configurations, sees list of repos to track, requests info from GitHub, builds a table and sends it to GS service. Sounds easy, but there were several tough to deal with things, which I've solved mostly with support of my work experience in Google projects, and which I consider as good patterns. Take a pen.
            Read more →
          • AdBlock has stolen the banner, but banners are not teeth — they will be back

            More
            Ads
          • buildroot — my own experience with multi-platform distro creation

              Introduction


              In my previous article (Monitor linux) I wrote, what is this distro and how it works. Now i will write how to do it. It's may be interesting for everyone, who want to study buildroot.


              Target goals


              The result we get from article is the following:


              • Firmware (non-volatile image with restorable config)
              • Easy management via web-interface
              • Cross-platform (qemu x86_64, arm-based SBC like rasberry 4, beagle bone black and asus tinker board)
              • Support without extra effort
              Read more →
            • PVS-Studio Is Now in Chocolatey: Checking Chocolatey under Azure DevOps

                Рисунок 1

                We continue making the use of PVS-Studio more convenient. Our analyzer is now available in Chocolatey, the package manager for Windows. We believe this will make it easier to deploy PVS-Studio, particularly in cloud services. So right off the bat, we also checked the source code of the same Chocolatey. Azure DevOps took on the role of the CI system.
                Read more →
              • Monitor linux — cross platform firmware with zabbix server

                  About


                  This is small cross-platform linux-distro with zabbix server. It's a simple way to deploy powerful monitoring system on ARM platfornms and x86_64.


                  Worked as firmware (non-changeable systemd image with config files), have web-interface for system management like network settings, password and other.


                  Who is interested


                  • System admins/engineers who need to fast deploy of zabbix server.
                  • Everyone, who want to deploy zabbix on ARM.
                  • Enthusiasts
                  Read more →
                • A Brief Comparison of the SDS Architectures for Virtualization

                  • Translation

                  The search for a suitable storage platform: GlusterFS vs. Ceph vs. Virtuozzo Storage


                  This article outlines the key features and differences of such software-defined storage (SDS) solutions as GlusterFS, Ceph, and Virtuozzo Storage. Its goal is to help you find a suitable storage platform.

                  Gluster



                  Let’s start with GlusterFS that is often used as storage for virtual environments in open-source-based hyper-converged products with SDS. It is also offered by Red Hat alongside Ceph.
                  GlusterFS employs a stack of translators, services that handle file distribution and other tasks. It also uses services like Brick that handle disks and Volume that handle pools of bricks. Next, the DHT (distributed hash table) service distributes files into groups based on hashes.
                  Note: We’ll skip the sharding service due to issues related to it, which are described in linked articles.

                  image

                  When a file is written onto GlusterFS storage, it is placed on a brick in one piece and copied to another brick on another server. The next file will be placed on two or more other bricks. This works well if the files are of about the same size and the volume consists of a single group of bricks. Otherwise the following issues may arise:
                  Read more →
                • Developing a symbolic-expression library with C#. Differentiation, simplification, equation solving and many more

                    Hello!

                    Why does programming a calculator seem to be a task, which every beginner undertakes? History might have the answer — computers were created for this exact purpose. Unlike the beginners, we will develop a smart calculator, which, although won't reach the complexity of SymPy, will be able to perform such algebraic operations as differentiation, simplification, and equations solving, will have built-in latex support, and have implemented features such as compilation to speed up the computations.

                    What are the articles about?
                    It will superficially tell about assembling an expression, parsing from a string, variable substitution, analytic derivative, equation numerical solving, and definite integration, rendering to LaTeX format, complex numbers, compiling functions, simplifying, expanding brackets, and blah blah blah.
                    For those who urgently need to clone something, repository link.

                    Let's do it!
                    Read more →
                  • Introducing One Ring — an open-source pipeline for all your Spark applications

                      If you utilize Apache Spark, you probably have a few applications that consume some data from external sources and produce some intermediate result, that is about to be consumed by some applications further down the processing chain, and so on until you get a final result.


                      We suspect that because we have a similar pipeline with lots of processes like this one:


                      A process flowchart with more than 50 applications and about 70 datasets
                      Click here for a bit larger version


                      Each rectangle is a Spark application with a set of their own execution parameters, and each arrow is an equally parametrized dataset (externally stored highlighted with a color; note the number of intermediate ones). This example is not the most complex of our processes, it’s fairly a simple one. And we don’t assemble such workflows manually, we generate them from Process Templates (outlined as groups on this flowchart).


                      So here comes the One Ring, a Spark pipelining framework with very robust configuration abilities, which makes it easier to compose and execute a most complex Process as a single large Spark job.


                      And we just made it open source. Perhaps, you’re interested in the details.

                      We got you covered!
                    • Five Methods For Database Obfuscation

                        ClickHouse users already know that its biggest advantage is its high-speed processing of analytical queries. But claims like this need to be confirmed with reliable performance testing. That's what we want to talk about today.



                        We started running tests in 2013, long before the product was available as open source. Back then, just like now, our main concern was data processing speed in Yandex.Metrica. We had been storing that data in ClickHouse since January of 2009. Part of the data had been written to a database starting in 2012, and part was converted from OLAPServer and Metrage (data structures previously used by Yandex.Metrica). For testing, we took the first subset at random from data for 1 billion pageviews. Yandex.Metrica didn't have any queries at that point, so we came up with queries that interested us, using all the possible ways to filter, aggregate, and sort the data.

                        ClickHouse performance was compared with similar systems like Vertica and MonetDB. To avoid bias, testing was performed by an employee who hadn't participated in ClickHouse development, and special cases in the code were not optimized until all the results were obtained. We used the same approach to get a data set for functional testing.

                        After ClickHouse was released as open source in 2016, people began questioning these tests.

                        Read more →
                      • Our Small Contribution to Avalonia UI's Fight for Fewer Platforms

                          Рисунок 2

                          This article is a review of the bugs found in the Avalonia UI project with the static analyzer PVS-Studio. Avalonia UI is an open-source cross-platform XAML-based UI framework. This is one of the most technologically significant projects in the history of .NET as it enables developers to create cross-platform interfaces based on the WPF system. We hope the project's authors will find this article helpful in fixing some of the bugs, and convincing enough to make static analysis part of their development process.
                          Read more →
                        • Apache Hadoop Code Quality: Production VS Test

                            Рисунок 1

                            In order to get high quality production code, it's not enough just to ensure maximum coverage with tests. No doubts, great results require the main project code and tests to work efficiently together. Therefore, tests have to be paid as much attention as the main code. A decent test is a key success factor, as it will catch regression in production. Let's take a look at PVS-Studio static analyzer warnings to see the importance of the fact that errors in tests are no worse than the ones in production. Today's focus: Apache Hadoop.
                            Read more →
                          • Faster ENUM

                              tl;dr


                              github.com/QratorLabs/fastenum
                              pip install fast-enum

                              What are enums


                              (If you think you know that — scroll down to the “Enums in Standard Library” section).

                              Imagine that you need to describe a set of all possible states for the entities in your database model. You'll probably use a bunch of constants defined as module-level attributes:
                              # /path/to/package/static.py:
                              INITIAL = 0
                              PROCESSING = 1
                              PROCESSED = 2
                              DECLINED = 3
                              RETURNED = 4
                              ...

                              ...or as class-level attributes defined in their own class:
                              class MyModelStates:
                                INITIAL = 0
                                PROCESSING = 1
                                PROCESSED = 2
                                DECLINED = 3
                                RETURNED = 4

                              That helps you refer to those states by their mnemonic names, while they persist in your storage as simple integers. By this, you get rid of magic numbers scattered through your code and make it more readable and self-descriptive.

                              But, both the module-level constant and the class with the static attributes suffer from the inherent nature of python objects: they are all mutable. You may accidentally assign a value to your constant at runtime, and that is a mess to debug and rollback your broken entities. So, you might want to make your set of constants immutable, which means both the number of constants declared and the values they are mapped to must not be modified at runtime.
                              Read more →
                            • Checking the Ark Compiler Recently Made Open-Source by Huawei

                                Picture 1

                                During the summer of 2019, Huawei gave a series of presentations announcing the Ark Compiler technology. The company claims that this open-source project will help developers make the Android system and third-party software much more fluent and responsive. By tradition, every new promising open-source project goes through PVS-Studio for us to evaluate the quality of its code.

                                Introduction


                                The Ark Compiler was first announced by Huawei at the launch of the new smartphone models P30 and P30 Pro. It is claimed that the Ark Compiler will improve the fluency of the Android system by 24% and response speed by 44%. Third-party Android applications will also gain a 60% speed-up after recompilation with the Ark Compiler. The open-source version of the project is called OpenArkCompiler; its source code is available on Gitee, a Chinese fork of GitHub.
                                Read more →
                              • How did the test task become a production library

                                Hello everyone!

                                Today I'm going to tell about how is test task to job interview became the library Image Comparison. It's an open-source library, which is hosting on GitHub.

                                logo

                                Before I start, let me introduce myself. My name is Roman. I'm a husband and father. I'm a software engineer in Epam Systems with 4 years of experience in IT.

                                The main idea of this topic is to tell, that creating an open-source product it's not wasting time, no! It's an amazing experience, which is going from all the open-source community. It's a time when you're a developer, project manager, product manager in one head.

                                While this library is growing I have been working with people from more than 10(!!) countries, such as the USA, Germany, Chine, India, Russia, Ukraine, etc.

                                Let's move on from the start of this story…
                                Read more →