Open source *

Привет, Хабр! Меня зовут Александр Тарасов, я архитектор департамента поддержки продаж Arenadata. Уход крупных зарубежных вендоров, таких как Oracle и Microsoft, заставил нас переосмыслить подходы к управлению данными и освоить альтернативные решения. Если с транзакционной нагрузкой всё относительно ясно благодаря PostgreSQL, то в аналитике дела обстоят сложнее, и здесь особенно важно адаптироваться к новым инструментам. Сейчас на первый план выходят системы с открытым исходным кодом, которые не только поддерживаются сообществом, но и открывают новые возможности для роста и экспериментов, оставаясь актуальными в условиях меняющегося рынка.

В этой статье я расскажу, как разработчикам и архитекторам адаптироваться к новым условиям, какие технологии стали достойной заменой западным решениям и какие шаги помогут специалистам эффективно использовать открытые платформы для укрепления своего карьерного трека.

The first meetups of the Portable SystemVerilog Examples group at Hacker Dojo in Mountain View, California were a kind of brainstorming sessions. We discussed the electronic industry, the essence of modern chip design, and the challenges of educating new design engineers. Then we moved to a new mode of action. We started weekly meetings of the core R&D team with the goal to prepare educational materials for the events for a larger audience. The meetings are generally held on Sundays from 11 pm to 2 pm. If you cannot come to Mountain View, you can join online.

We are focusing on interview-level microarchitectural and CPU design examples, providing FPGA vendor-neutral infrastructure and compatibility with open-source ASIC design tools.

Ссылка на русскую версию / link to Russian version

Understanding valid/ready protocol is extremely important for every microarchitect.

Valid/ready is one of the main protocols used to organise flow-control inside a logic block as well as on inter-block (SoC) level.

In the last lesson, we explored FIFO buffer using hdlgadgets - human-in-the-loop HDL training tool.

This time we will take two FIFO buffers (which form a pipeline with valid/ready handshakes) and will experiment with it by changing flow-control logic of the pipeline.

We will show that valid/ready is not only a mechanism for transferring data from one FIFO queue to another, but also a method for organizing various kinds of logical functionality between queues.

If you have not worked with valid/ready protocol before, you will be surprised how easy it is to achieve desired functionality of the design by simply writing couple of lines of Verilog code in the handshaking logic block between two FIFOs.

We released the new version of Chipmunk, software for viewing/analyzing log files. V.3 is wholly reworked with an accent to performance and rethought considering usability. Below short list of the most important changes and a general description.

Ссылка на русскую версию / link to Russian version

FIFO is a key concept in hardware design. Understanding of FIFO is necessary for understanding the valid/ready protocol, which in turn is necessary for organisation of flow-control within a design.

Unfortunately, there are very few books on this topic, and to be fair, microarchitectural concepts are quite difficult to master from books, since understanding of these concepts are coming with practice. In other words it is more about developing hardware intuition.

The idea of the HDL training tool is that it can help develop a hardware intuition, providing the opportunity to explore ready-made scenarios in a step-by-step interactive way. The tool also provides detailed visualization of a simulated scenario.

Since the tool is a front-end for the HDL simulator, the real, synthesized SystemVerilog is executed on the simulator itself, which can be viewed and even modified.

So, the video of exploring FIFO on the training tool is here:

Currently I am investigating firmware development for STM32 microcontrollers and I would like to share with you my experience for doing it in Qt Creator IDE.

There are a lot of IDEs, which are used for firmware development of STM32. Some of them, being quite comfortable, have restrictions for trial license. For example, the one of the most known IDE, IAR Embedded, suggests either a limited amount of product usage time (30 days) or the limited firmware size of 32 MB, which is not too much.

Within this scope of the publication, we investigate the method of setting up an environment that allows one to develop the full value of the STM32 firmware in Qt Creator.

lsFusion platform is designed for rapid development of business applications. It is distributed under the terms of a Lesser General Public License (LGPLv3). The source code of the platform is available on Github.

lsFusion is best suited for creating complex systems with large numbers of entities and forms, where users need to input and process large amounts of data. However, the platform can also be used to quickly create simple applications instead of spreadsheets when Excel’s functionality is not enough.

At the same time the use of the platform will not give a great advantage when developing applications aimed at interaction with a large number of “external” users or without the need for any complex calculations. You should also take into account that the web interface is a single page application using JavaScript. Therefore, the lsFusion platform is not well-suited for creating websites, for example.

OpenTeleletry Collector is an open source data collection pipeline that allows you to monitor CPU, RAM, disk, network metrics, and many more.

Collector itself does not include built-in storage or analysis capabilities, but you can export the data to Uptrace and ClickHouse, using them as a replacement for Grafana and Prometheus.

When compared to Prometheus, ClickHouse can offer small on-disk data size and better query performance when analyzing millions of timeseries.

I will continue the story "How to put the whole world into a regular laptop: PostgreSQL and OpenStreetMap" with secrets about OpenStreetMap geodata, on which many companies have built their business, but not everyone shares the details... Well, today we will open crucial details.

The OSM database in PosgreSQL after loading from the dump takes up more than 587 GB. This is already a large database by the standards of a DBMS, and one huge table for each type of object will not work. For manageability, such data must be partitioned, it's good that PostgreSQL supports declarative data partitioning. It remains only to figure out how to split geographical data. After searching and comparing, the H3 hierarchical hexagonal geospatial indexing system came to rescue. All this was implemented in my openstreetmap_h3 project for fast processing and loading of the world dump into the PostGIS database.

I considered following options from geopartitioning systems...

When a person used to say that he controls the whole world, he was usually placed in the next room with Napoleon Bonaparte. I hope that these times are in the past and everyone can analyze the geodata of the entire Earth and get answers to their global questions in minutes and seconds. I published Openstreetmap_h3 - my project, which allows you to perform geoanalytics on data from OpenStreetMap in PostGIS or in any query engine that can work with Apache Arrow / Parquet.

First of all, I say hello to the haters and skeptics. What I developed is really unique and solves the problem of transforming and analyzing geodata using the usual and familiar tools available to every analyst and data science specialist without bigdata, GPGPU, FPGA. What looks easy to use and code now is my personal project where I have invested my vacations, weekends, sleepless nights and a lot of personal time over the past 3 years. Maybe I will share the background of the project and the rake that I went through, but first I will still describe the end result.

Today I will discover America to you based on OpenStreetMap data in PostgreSQL15/PostGIS and my project openstreetmap_h3. Let's run the query and compare its execution time on the Citus column store in PostgreSQL and on the standard 100GB database partitioned by H3 geoindex.

We will find the top15 buildable locations in North America and the total length of roads, as well as their type and surface. I will not overload the publication with program logs, let's focus on the data! You can easily repeat all requests yourself on your laptop/computer.

Do you need to produce satellite interferometry results for your work or study? Or should you find the way to process terabytes of radar data on your common laptop? Maybe you aren't confident about the installation and usage of the required software. Fortunately, there is the next generation of satellite interferometry products available for you. Beginners can build the results easily and advanced users might work on huge datasets. Open Source software PyGMTSAR is available on GitHub for developers and on DockerHub for advanced users and on Google Colab for everyone. This is the cloud-ready product, and it works the same as do you run it locally on your old laptop as on powerful cloud servers.

Nfun is an embedded language and expression executor that supports primitive types, arrays, structures and lambda expressions.

Most likely, you have already met tasks that require such a tool, and in this article I want to show examples of its application, its capabilities and why it may be useful to you.

Overview

Once every true-linux engineer gets a trouble: there is no any software in his distro or it's built without needed options. I am keen on the phrase: "Only source control gives you freedom".

Of course, you can build this software on your computer without any src-packages, directly (with simplification: configure, make, make install). But it's a non-reproducible solution, also hard for distribution.

The better way is to make distro-aligned package that can be built if needed and that produces lightly distributed binary-packages. It's about debian-source packages(debian,ubuntu,etc), pkgbuild (for arch), ebuild for gentoo, src-rpm for red hat-based, and many others.

I will use cri-o like a specimen.

Before reading the text below I strongly recommend to get familiarized with the official Debian policy manual placed here and debhelper manpage.

Also you will be required to setup some variables like DEBMAIL and DEBFULLNAME for proper data in changelog and other places.

Global corporations became a part of our everyday life for a long time ago, their products often don’t leave an alternative option for users. Either is it exist? This article touches an issue of dominating big companies in certain areas, but also contains a row of useful tips. Spoiler of one of them: if you’re a user of Android then the tips will help you increase time between charging your phone and improve your privacy.

Overview

Sometimes all of us need to make a graphical installer for one's own linux distro. It goes without saying that you are able to use a distro-specific installer like Anaconda for RedHat-based or DebianInstaller for debian-based. On the other hand Calamares is a graphical installer which is not aligned with only one package manager.

I want to share my experience how to make a universal install solution with GUI. I did not find any complete article about it, hence, I reinvented the wheel.

We are often asked whether we send bug reports to developers of open-source projects. The answer is yes. More than that— we sometimes track the progress. This article is about one of the cases when this habit prevented a sloppy bug fix.

Ссылка на русскую версию / link to Russian version

FPGA InsideOut is an attempt to make a set of educational FPGA videos presented in the “human-in-the-loop” style. In these videos we will not only show how we are interfacing with an actual FPGA board but will also provide synchronous real-time visualisation of FPGA's internal logic.

For our first video we have picked a CRC circuit (cycle redundancy check) which is based on a linear feedback shift register. This circuit goes through several transformations during the course of the video. Intrigued? - let’s watch the video.

In 2021 we published several articles and showed you errors found in open-source projects. The year 2021 ends, which means it's time to present you the traditional top 10 of the most interesting bugs. Enjoy!