![](https://habrastorage.org/r/w1560/getpro/habr/upload_files/4c2/5c4/a55/4c25c4a55f8002100b4ec3e5cf072f6b.png)
![](http://habrastorage.org/getpro/habr/hub/545/277/1b2/5452771b2aa9ca1a52d1d693cdc24c3e.png)
Data Engineering *
discuss data collection and preparation
Utilizing Wearable Digital Health Technologies for Cardiovascular Monitoring
![](https://habrastorage.org/r/w1560/getpro/habr/upload_files/37c/3bf/8a7/37c3bf8a76d00c764db7be99a0b68fc8.png)
Wearable Digital Health Technologies for Monitoring in Cardiovascular Medicine
This review article presents a three-part true-life clinical vignette that illustrates how digital health technology can aid providers caring for patients with cardiovascular disease. Specific information that would identify real patients has been removed or altered. Each vignette is followed by a discussion of how these methods were used in the care of the patient.
User-defined aggregation functions in Spark
![](https://habrastorage.org/r/w1560/getpro/habr/upload_files/fd3/6c7/e5d/fd36c7e5ddad88b03d593266ed3dea4d.png)
Below, we will discuss user-defined aggregation functions (UDAF) using org.apache.spark.sql.expressions.Aggregator
, which can be used for aggregating groups of elements in a DataSet into a single value in any user-defined way.
Let’s start by examining an example from the official documentation that implements a simple aggregation
Validation of custom data with Minimal API filters, sensible and without fuss
![](https://habrastorage.org/r/w1560/getpro/habr/upload_files/916/618/9ed/9166189edab852559f81806b3de6a628.png)
This is a step-by-step research of a clear and intuitive approach to validate custom data in .NET applications with help of the Minimal API filters and the FlatValidator.
Evaluating Performance: CosmosDB vs. Azure SQL
![](https://habrastorage.org/r/w780/getpro/habr/upload_files/5bd/8b0/ca6/5bd8b0ca6ea36180b1c5b0046af257ff.jpg)
In the evolving landscape of database technology, choosing the right database management system is crucial for the efficiency and scalability of applications. This article presents a detailed comparison of the performance between Microsoft's CosmosDB and MS SQL Server. We'll examine how each database performs under various load conditions and share some interesting findings.
UX Designer Job Interview: 10 questions to answer, 5 questions to ask
![](https://habrastorage.org/r/w780/getpro/habr/upload_files/e0f/8ba/918/e0f8ba9183e317103a2a5f6f2180afaa.jpeg)
No matter how many degrees you have or how high your experience level is, your recruiters need to evaluate your knowledge of UX design as a whole. But keep in mind that a job interview is not an exam, so here you are expected not to recite the textbook definitions learned by heart, but rather share your personal understanding of UX and your role as a designer in general. Consider talking about how you define UX, what creates value in the design, what are the necessary parts of a UX design process, what are the current trends in UX. You might also be asked to explain the difference between UI and UX to see how you understand the role of each in the development process.
A (more) accurate camera sensor dynamic range measurement
![](https://habrastorage.org/getpro/habr/upload_files/d68/e95/db6/d68e95db636189c1784e28ba68014a2a.gif)
Hello, everyone! In this post, let's talk about how to (more) accurately measure the dynamic range of a camera sensor and what can be done with these measurements.
Of course, I am not an expert in computer vision, a programmer or a statistician, so please feel free to correct me in the comments if I make mistakes in this post. Here my interest was primarily focused on everyday and practical tasks, such as photography, but I believe the results may also be useful to computer vision professionals.
How to access real-time smart contract data from Python code (using Lido contract as an example)
![](https://habrastorage.org/r/w1560/getpro/habr/upload_files/b70/ce4/364/b70ce43646407c7a042444e4df3b87fe.png)
Let’s imagine you need access to the real-time data of some smart contracts on Ethereum (or Polygon, BSC, etc.) like Uniswap or even PEPE coin to analyze its data using the standard data scientist/analyst tools: Python, Pandas, Matplotlib, etc. In this tutorial, I’ll show you more sophisticated data access tools that are more like a surgical scalpel (The Graph subgraphs) than a well-known Swiss knife (RPC node access) or hammer (ready-to-use APIs). I hope my metaphors don’t scare you ?.
Mathematical meaning of principal component analysis (PCA)
![](https://habrastorage.org/r/w1560/getpro/habr/upload_files/b7c/01a/da4/b7c01ada4c2d50ceaecfb95b3e38e18f.png)
This article aims at explaining the mathematical sense of the Principal Component Analysis (PCA) in practice.
Pixel image rotation
![](https://habrastorage.org/r/w780/getpro/habr/upload_files/94e/1df/4ef/94e1df4ef0dc9d075cbc54abe71727a7.jpg)
Brief problem formulation
The program accepts as input the absolute path to the image in the bmp extension and the path where you save the result of the work. Then, it rotates the image by 90 degrees counterclockwise. Afterwards, the program saves the new image.
The program is executed on C.
Blinking into Morse code
Explaining main algorithm.
For a while I’ve been thinking of writing a scientific article. I wanted it to have certain utility.
Morse code is binary: it takes only two values – either dot (short) or hyphen (long). I figured out that short (s) can stand for two-eye blinking whilst long (l) can indicate left-eye blinking. Another question emerged: how to understand when does one-symbol recording stop?
Empty space between two symbols can be presented by right-eye blinking – r. If I input singly symbol of short (dot) and long (hyphen), I will blink my right eye once to indicate the space between two symbols.
To separate independent words, one has to blink her right eye twice and get rr.
Hence, I have collected an ordered set of symbols – r, l, s, - that can be converted into a full-fledged text. Once I accomplish the transformation, I get an answer.
Feature Engineering: Techniques and Best Practices for Data Scientists
![](https://habrastorage.org/r/w1560/getpro/habr/upload_files/8d6/26b/c79/8d626bc79e878204134865aa51c511eb.png)
The most important stage in the data science process is feature engineering, which entails turning raw data into useful features that might enhance the performance of machine learning models. It calls for creativity, data-driven thinking, and domain expertise. Data scientists can improve the prediction capability of their models and find hidden patterns in the data by choosing, combining, and inventing relevant features. Handling missing data, scaling features, encoding categorical variables, constructing interaction terms, and other procedures are examples of feature engineering techniques. The best practises involve investigating the data, testing and improving features iteratively, and applying domain knowledge to draw out important information. The accuracy and effectiveness of machine learning models are significantly influenced by effective feature engineering.
How we designed the user interface for an enterprise analytical system
![](https://habrastorage.org/r/w1560/getpro/habr/upload_files/cc8/49a/4ef/cc849a4ef1a61e5c45af1876d3518dfd.png)
In 2021, we were contacted by an industrial plant that was faced with the need to create a system for analyzing processes in its production. The enterprise team studied ready-made solutions, but none of the analytics system designs fully covered the required functionality. So they turned to us with a request to develop their own analytical system that would collect data from all machines and allow it to be analyzed to see bottlenecks in production. For this project, we created a data-driven UI/UX design and also developed a web-based interface for the equipment monitoring system.
How Yandex Made Their Biggest Improvement in the Search Engine with the Help of Toloka
![](https://habrastorage.org/r/w780/getpro/habr/upload_files/f51/7c0/6f8/f517c06f86214f69a15e899456bbd781.jpg)
Toloka is a crowdsourcing platform and microtasking project launched by Yandex to quickly markup large amounts of data. But how can such a simple concept play a crucial role in improving the work of neural networks?
What are neural networks and what do we need them for?
![](https://habrastorage.org/r/w1560/getpro/habr/upload_files/fb9/011/b08/fb9011b08dbd66280180fd7a92ffc27a.png)
Explaining through simple examples
For a long time, people have been thinking on how to create a computer that could think like a person. The advent of artificial neural networks is a significant step in this direction. Our brain consists of neurons that receive information from sensory organs and process it: we recognize people we know by their faces, and we feel hungry when we see delicious food. All of this is the result of brain neurons working and interacting with each other. This is also the principle that artificial neural networks are based on, simulating the processes occurring in the human brain.
What are neural networks
Artificial neural networks are a software code that imitates the work of a brain and is capable of self-learning. Like a biological network, an artificial network also consists of neurons, but they have a simpler structure.
If you connect neurons into a sufficiently large network with controlled interaction, they will be able to perform quite complex tasks. For example, determining what is shown in a picture, or independently creating a photorealistic image based on a text description.
Math introduction to Deep Theory
![](https://habrastorage.org/r/w1560/getpro/habr/upload_files/1ca/d4c/c02/1cad4cc02df9881be59b66da8d7c567f.png)
In this article, we would like to compare the core mathematical bases of the two most popular theories and associative theory.
Millions of orders per second matching engine testing
![](https://habrastorage.org/r/w780/getpro/habr/upload_files/751/c03/16d/751c0316d8ce9b5ddc15748e3d418eeb.jpg)
I had some experience in the matching engine development for cryptocurrency exchange some time ago. That was an interesting and challenging experience. I developed it in clear C++ from scratch. The testing of it is also quite a challenging task. You need to get data for testing, perform testing, collect some statistics, and at last, analyze collected data to find weak points and bottlenecks. I want to focus on testing the C++ matching engine and show how testing can give insights for optimizations even without the need to change the code. The matching engine I developed can do more than 1’000’000 TPS (transactions per second) and is 10x times faster than the matching engine of the Binance cryptocurrency exchange (see one post on Binance Blog).
Big Data Tools with IntelliJ IDEA Ultimate, PyCharm Professional, DataGrip 2021.3 EAP, and DataSpell Support
![](https://habrastorage.org/r/w1560/getpro/habr/upload_files/918/e06/902/918e069020ce32f06f266386b4179dde.png)
Recently we released a new build of the Big Data Tools plugin that is compatible with the 2021.3 versions of IntelliJ IDEA and PyCharm. DataGrip 2021.3 support will be available immediately after the release in October. The plugin also supports our new data science IDE – JetBrains DataSpell. If you still use previous versions, now is the perfect time to upgrade both your IDE and the plugin.
This year, we introduced a number of new features as well as some features that have been there for a while, for example, running Spark Submit with a run configuration.
Here’s a list of the key improvements:
One of the ways to dynamically deserialize a part of a JSON document with an unknown structure
![](https://habrastorage.org/r/w780/getpro/habr/upload_files/09d/a14/6d5/09da146d5bad3f74bd470fb183861985.jpg)
In this topic, I will tell you how to dynamically parse and deserialize only part of the whole JSON document. We will create an implementation for .NET Core with C# as a language.
For example, we have the next JSON as a data source for the report. Notice that we will get this JSON in the runtime and at the compile step we don't know the structure of this document. And what if you need to select only several fields for processing?
Benefits of Hybrid Data Lake: How to combine Data Warehouse with Data Lake
![](https://habrastorage.org/r/w1560/getpro/habr/upload_files/800/4e1/906/8004e19068cec3c20458fe058542afa9.png)
Hey, hey! I am Ilya Kalchenko, a Data Engineer at NIX, a fan of big and small data processing, and Python. In this article, I want to discuss the benefits of hybrid data lakes for efficient and secure data organization.
To begin with, I invite you to figure out the concepts of Data Warehouses and Data Lake. Let’s delve into the use cases and delimit areas of responsibility.
Authors' contribution
Aleron75 265.3s_valuev 234.0paramonov_ruvds 206.0kzzzr 176.3Doctor_IT 166.0skillfactory_school 148.8editor_ruvds 127.0Nester 115.0selesnow 107.6feanoref 101.0