Open source

We continue to investigate external device interrupt routing setup in the x86 system.

In Part 1 (Interrupt controller evolution) we looked at the theory behind interrupt controllers and all the necessary terminology. In Part 2 (Linux kernel boot options) we looked at how in practice the OS chooses between different interrupt controllers. In this part we will investigate how the BIOS sets IRQ to the interrupt controllers routing in a chipset.

None of the modern BIOS developer companies (AwardBIOS/AMIBIOS/Insyde) open their source code. But luсkily there is coreboot — a project aimed at replacing proprietary BIOS with free firmware code. In its source code we'll see what is needed to setup the interrupt routing in a chipset.

The basics of porting

Originally, Buildroot offers a limited number of packages. It makes sense — there is everything you need, but any other packages can be added.

To add a package, create 2 description files, an optional checksum file, and add a link to the package in the general package list. There are hooks at different stages of the build. At the same time, Buildroot can recognize the needed type of packages:

Intro

Like I said earlier in previos articles, Buildroot is a great system for embedded Linux development. But sometimes strange things can happen.

Once upon a workday, I got the following task: add printing system in firmware (Kraftway terminal Linux next generation). Ok, so I had to add cups + cups filter and to build firmware. I set a postscript-printer and got an error "Filter failed". Trivial tasks turned into serious work.

In this article, I wrote my own way of solving this problem. It may be useful for other developers and IT-specialist and, also, for a deeper understanding of the Buildroot.

If you are a Buildroot beginner, I recommend reading my previous articles.

Update 1 may 2020

Revisioned versions of this patches applied to master.

Some time ago, somewhere on the Internet, I stumbled upon a physics engine called Newton Game Dynamics. Knowing that engine projects are usually big and complex, I decided to check its code with PVS-Studio for any interesting defects. I was especially enthusiastic about this one because my co-worker Andrey Karpov already checked it in 2014 and a second check would be a good opportunity to demonstrate our analyzer's evolution over the past six years. As of this writing, the latest version of Newton Game Dynamics is dated February 27, 2020, which means it has been actively developing for the past six years too. So, hopefully, this article will be interesting not only to us but to the engine's developers as well – and for them it's a chance to fix some bugs and improve their code.

About a week I wonder around the idea of this article, mainly, because of missing the content here and tidiness of the coronavirus-themes. But, when robotic-harvester, hacking zip with bitcoins and other cool articles released — I decided not to publish crude material.

However, unexpectedly today the maintainer of this review hero showed up from the lockdown and, several hours ago reapy v0.6.0 has been published on PyPi. Under the cut — the last change log, which contains (to my pleasure) no line where I have not been involved one way, or another.

Finally: why reapy is needed and how Python works inside REAPER.

The GCC compiler is written with copious use of macros. Another check of the GCC code using PVS-Studio once again confirms the opinion of our team that macros are evil in the flesh. Not only does the static analyzer struggle with reviewing such code, but also a developer. GCC developers are certainly used to the project and are well versed in it. Nonetheless, it is very difficult to understand something on the third hand. Actually, due to macros, it was not possible to fully perform code checking. However, the PVS-Studio analyzer, as always, showed that it can find errors even in compilers.

Static analysis is an extremely useful tool for any developer, as it helps to find in time not only errors, but also suspicious and strange code fragments that may cause bewilderment of programmers who will have to work with it in the future. This idea will be demonstrated by the analysis of the TensorFlow.NET open C# project, developed for working with the popular TensorFlow machine learning library.

Some time ago we announced PVS-Studio's new feature that enabled it to integrate into PlatformIO. Naturally, our team kept in touch with the PlatformIO team while working on that feature, and they suggested that we check the real-time operating system Zephyr to see if we could find any interesting bugs in its code. We thought it was a good idea, and so here's this article about the check results.

Almost four years have passed since the PVS-Studio team checked the OpenToonz source code. This project is a very powerful tool for creating two-dimensional animation. Since the last check, with its help, such animated works as Mary and the Witch Flower, Batman-Ninja, Promare and others were created. As large studios continue using Toonz, why not check the quality of the source code again?

The new Windows Terminal version has been recently released. Everything would be fine, but performance of its scrollbar leaves a great deal to be desired. Time has come to poke it and have some mumbo jumbo dances with it.

Work in several GitHub repositories (repos) brings inconveniences. Repos are separated from each other, open and closed issues live in different lists, linked pull requests (PRs) can't be seen until opening an issue — our team works in more than 70 repos, so I learned that hard. And started to write repos tracker on Python. It's close to the final, I'd like to share it (it's completely free), and few tricks I used in progress.

What is that

Scraper tracks several GitHub repos in a single Google Sheet (GS) table. You can see all of the opened and done issues, related PRs, priorities, your teammates comments, use coloring, filtering, sorting and other GS functions. That's a good integration, here is how it looks:

How does it work, shortly

There is Spreadsheet() class which contain several Sheet() objects, each of which have it's own configurations, described in config.py. When Scraper updates a sheet, it loads configurations, sees list of repos to track, requests info from GitHub, builds a table and sends it to GS service. Sounds easy, but there were several tough to deal with things, which I've solved mostly with support of my work experience in Google projects, and which I consider as good patterns. Take a pen.

Introduction

In my previous article (Monitor linux) I wrote, what is this distro and how it works. Now i will write how to do it. It's may be interesting for everyone, who want to study buildroot.

Target goals

The result we get from article is the following:

• Firmware (non-volatile image with restorable config)
• Easy management via web-interface
• Cross-platform (qemu x86_64, arm-based SBC like rasberry 4, beagle bone black and asus tinker board)
• Support without extra effort

We continue making the use of PVS-Studio more convenient. Our analyzer is now available in Chocolatey, the package manager for Windows. We believe this will make it easier to deploy PVS-Studio, particularly in cloud services. So right off the bat, we also checked the source code of the same Chocolatey. Azure DevOps took on the role of the CI system.

About

This is small cross-platform linux-distro with zabbix server. It's a simple way to deploy powerful monitoring system on ARM platfornms and x86_64.

Worked as firmware (non-changeable systemd image with config files), have web-interface for system management like network settings, password and other.

Who is interested

• System admins/engineers who need to fast deploy of zabbix server.
• Everyone, who want to deploy zabbix on ARM.
• Enthusiasts

The search for a suitable storage platform: GlusterFS vs. Ceph vs. Virtuozzo Storage

This article outlines the key features and differences of such software-defined storage (SDS) solutions as GlusterFS, Ceph, and Virtuozzo Storage. Its goal is to help you find a suitable storage platform.

Gluster

Let’s start with GlusterFS that is often used as storage for virtual environments in open-source-based hyper-converged products with SDS. It is also offered by Red Hat alongside Ceph.
GlusterFS employs a stack of translators, services that handle file distribution and other tasks. It also uses services like Brick that handle disks and Volume that handle pools of bricks. Next, the DHT (distributed hash table) service distributes files into groups based on hashes.
Note: We’ll skip the sharding service due to issues related to it, which are described in linked articles.



When a file is written onto GlusterFS storage, it is placed on a brick in one piece and copied to another brick on another server. The next file will be placed on two or more other bricks. This works well if the files are of about the same size and the volume consists of a single group of bricks. Otherwise the following issues may arise:

Hello!

Why does programming a calculator seem to be a task, which every beginner undertakes? History might have the answer — computers were created for this exact purpose. Unlike the beginners, we will develop a smart calculator, which, although won't reach the complexity of SymPy, will be able to perform such algebraic operations as differentiation, simplification, and equations solving, will have built-in latex support, and have implemented features such as compilation to speed up the computations.


Let's do it!

If you utilize Apache Spark, you probably have a few applications that consume some data from external sources and produce some intermediate result, that is about to be consumed by some applications further down the processing chain, and so on until you get a final result.

We suspect that because we have a similar pipeline with lots of processes like this one:

Click here for a bit larger version

Each rectangle is a Spark application with a set of their own execution parameters, and each arrow is an equally parametrized dataset (externally stored highlighted with a color; note the number of intermediate ones). This example is not the most complex of our processes, it’s fairly a simple one. And we don’t assemble such workflows manually, we generate them from Process Templates (outlined as groups on this flowchart).

So here comes the One Ring, a Spark pipelining framework with very robust configuration abilities, which makes it easier to compose and execute a most complex Process as a single large Spark job.

And we just made it open source. Perhaps, you’re interested in the details.

ClickHouse users already know that its biggest advantage is its high-speed processing of analytical queries. But claims like this need to be confirmed with reliable performance testing. That's what we want to talk about today.



We started running tests in 2013, long before the product was available as open source. Back then, just like now, our main concern was data processing speed in Yandex.Metrica. We had been storing that data in ClickHouse since January of 2009. Part of the data had been written to a database starting in 2012, and part was converted from OLAPServer and Metrage (data structures previously used by Yandex.Metrica). For testing, we took the first subset at random from data for 1 billion pageviews. Yandex.Metrica didn't have any queries at that point, so we came up with queries that interested us, using all the possible ways to filter, aggregate, and sort the data.

ClickHouse performance was compared with similar systems like Vertica and MonetDB. To avoid bias, testing was performed by an employee who hadn't participated in ClickHouse development, and special cases in the code were not optimized until all the results were obtained. We used the same approach to get a data set for functional testing.

After ClickHouse was released as open source in 2016, people began questioning these tests.

Emby is quite a popular media server along with Plex and Kodi. In this article, we'll discuss the bugs found in its source code with the static analyzer PVS-Studio. The remark «Built with ReSharper» on the project's official website makes the analysis even more interesting.

This article is a review of the bugs found in the Avalonia UI project with the static analyzer PVS-Studio. Avalonia UI is an open-source cross-platform XAML-based UI framework. This is one of the most technologically significant projects in the history of .NET as it enables developers to create cross-platform interfaces based on the WPF system. We hope the project's authors will find this article helpful in fixing some of the bugs, and convincing enough to make static analysis part of their development process.