Pull to refresh
66.93
Rating

Data storage *

What we have, we store

Show first
Rating limit

The way we made an external PCIe RAM disk based on the DDR memory

High performance *Cloud computing *Data storage *Computer hardware Data storaging

RAM disk, this is a disk based on RAM memory chips. This kind of disk is not able to retain data after the power is turned off (unless a supporting battery is used), but has an exceptionally high read/write speed (especially for random access) and an unlimited lifespan. It is important in tasks that need a lot of cycles to write over information, even professional SSD drives don’t live long. To the operating system the RAM disk is indistinguishable from an SSD or HDD disk and no special drivers or setup is required. Unlike a disk that is virtually located in the computer’s RAM memory, where the maximum memory capacity is limited to 128-256 GB in the best consumer motherboards, a RAM disk for a PCIe slot, in general, has no volume limits and can work in any MB with a PCIe slot.

Read more
Total votes 3: ↑3 and ↓0 +3
Views 5.6K
Comments 0

HDB++ TANGO Archiving System

Open source *Python *IT Infrastructure *Data storage *Data storages *
Translation
Tutorial
main

What is HDB++?


This is a TANGO archiving system, allows you to save data received from devices in the TANGO system.


Working with Linux will be described here (TangoBox 9.3 on base Ubuntu 18.04), this is a ready-made system where everything is configured.


What is the article about?


  • System architecture.
  • How to set up archiving.

It took me ~ 2 weeks to understand the architecture and write my own scripts for python for this case.


What is it for?


Allows you to store the history of the readings of your equipment.


  • You don't need to think about how to store data in the database.
  • You just need to specify which attributes to archive from which equipment.
Read more →
Rating 0
Views 619
Comments 0

IIoT platform databases – How Mail.ru Cloud Solutions deals with petabytes of data coming from a multitude of devices

VK corporate blog Database Administration *Data storage *IOT Tarantool *


Hello, my name is Andrey Sergeyev and I work as a Head of IoT Solution Development at Mail.ru Cloud Solutions. We all know there is no such thing as a universal database. Especially when the task is to build an IoT platform that would be capable of processing millions of events from various sensors in near real-time.

Our product Mail.ru IoT Platform started as a Tarantool-based prototype. I’m going to tell you about our journey, the problems we faced and the solutions we found. I will also show you a current architecture for the modern Industrial Internet of Things platform. In this article we will look into:

  • our requirements for the database, universal solutions, and the CAP theorem
  • whether the database + application server in one approach is a silver bullet
  • the evolution of the platform and the databases used in it
  • the number of Tarantools we use and how we came to this
Read more →
Total votes 19: ↑19 and ↓0 +19
Views 1.3K
Comments 0

Bcache against Flashcache for Ceph Object Storage

Selectel corporate blog IT Infrastructure *Server Administration *Data storage *Data storages *

Fast SSDs are getting cheaper every year, but they are still smaller and more expensive than traditional HDD drives. But HDDs have much higher latency and are easily saturated. However, we want to achieve low latency for the storage system, and a high capacity too. There’s a well-known practice of optimizing performance for big and slow devices — caching. As most of the data on a disk is not accessed most of the time but some percentage of it is accessed frequently, we can achieve a higher quality of service by using a small cache.

Server hardware and operating systems have a lot of caches working on different levels. Linux has a page cache for block devices, a dirent cache and an inode cache on the filesystem layer. Disks have their own cache inside. CPUs have caches. So, why not add one more persistent cache layer for a slow disk?
Read more →
Total votes 16: ↑16 and ↓0 +16
Views 1.6K
Comments 0

Why Enterprise Chat Apps isn’t built on Server-side Database like Hangouts, Slack, & Hip chat?

IT Infrastructure *Server optimization *Data storage *Data storages *
Sandbox
One of the most significant tools for any organization to smoothen their collaborative world is only through a real-time chat application whether the conversation takes place on mobile or desktop. Hangouts, Slack and Hipchat have been in action for businesses to establish a decent conversation between their internal employees and clients right from small-scale to enterprises.

Those big players come into play where there requires team collaboration. The big players are built on a server-side database where the messages shared from one device to another is stored in their server database. Ultimately, this results in storing a huge amount of data within the server-side database (Cloud-database).

The consumption of cloud storage will be pretty high. The client-side database is more efficient where the messages relayed is stored in the client device. The messages will be queued to minimize the consumption of data in the server.
image
Read more →
Total votes 3: ↑3 and ↓0 +3
Views 2.4K
Comments 0

A Brief Comparison of the SDS Architectures for Virtualization

Open source *IT Infrastructure *Data storage *Development for Linux *Data storages *
Translation

The search for a suitable storage platform: GlusterFS vs. Ceph vs. Virtuozzo Storage


This article outlines the key features and differences of such software-defined storage (SDS) solutions as GlusterFS, Ceph, and Virtuozzo Storage. Its goal is to help you find a suitable storage platform.

Gluster



Let’s start with GlusterFS that is often used as storage for virtual environments in open-source-based hyper-converged products with SDS. It is also offered by Red Hat alongside Ceph.
GlusterFS employs a stack of translators, services that handle file distribution and other tasks. It also uses services like Brick that handle disks and Volume that handle pools of bricks. Next, the DHT (distributed hash table) service distributes files into groups based on hashes.
Note: We’ll skip the sharding service due to issues related to it, which are described in linked articles.

image

When a file is written onto GlusterFS storage, it is placed on a brick in one piece and copied to another brick on another server. The next file will be placed on two or more other bricks. This works well if the files are of about the same size and the volume consists of a single group of bricks. Otherwise the following issues may arise:
Read more →
Total votes 2: ↑1 and ↓1 0
Views 1.6K
Comments 0

Tarantool Data Grid: Architecture and Features

VK corporate blog High performance *Lua *Data storage *Tarantool *


In 2017, we won the competition for the development of the transaction core for Alfa-Bank's investment business and started working at once. (Vladimir Drynkin, Development Team Lead for Alfa-Bank's Investment Business Transaction Core, spoke about the investment business core at HighLoad++ 2018.) This system was supposed to aggregate transaction data in different formats from various sources, unify the data, save it, and provide access to it.

In the process of development, the system evolved and extended its functions. At some point, we realized that we created something much more than just application software designed for a well-defined scope of tasks: we created a system for building distributed applications with persistent storage. Our experience served as a basis for the new product, Tarantool Data Grid (TDG).

I want to talk about TDG architecture and the solutions that we worked out during the development. I will introduce the basic functions and show how our product could become the basis for building turnkey solutions.
Read more →
Total votes 39: ↑38 and ↓1 +37
Views 1.7K
Comments 0

Quintet instead of Byte — data storage and retrieval approach

Programming *System Analysis and Design *SQL *IT Standards *Data storage *
Quintet is a way to present atomic pieces of data indicating their role in the business area. Quintets can describe any item, while each of them contains complete information about itself and its relations to other quintets. Such description does not depend on the platform used. Its objective is to simplify the storage of data and to improve the visibility of their presentation.



We will discuss an approach to storing and processing information and share some thoughts on creating a development platform in this new paradigm. What for? To develop faster and in shorter iterations: sketch your project, make sure it is what you thought of, refine it, and then keep refining the result.

The quintet has properties: type, value, parent, and order among the peers. Thus, there are 5 components including the identifier. This is the simplest universal form to record information, a new standard that could potentially fit any programming demands. Quintets are stored in the file system of the unified structure, in a continuous homogeneous indexed bulk of data. The quintet data model — a data model that describes any data structure as a single interconnected list of basic types and terms based on them (metadata), as well as instances of objects stored according to this metadata (data).
Read more →
Total votes 8: ↑8 and ↓0 +8
Views 1.3K
Comments 2

Bypassing LinkedIn Search Limit by Playing With API

JavaScript *API *Reverse engineering *Data storage *Social networks and communities
Translation
[Because my extension got a lot of attention from the foreign audience, I translated my original article into English].

Limit


Being a top-rated professional network, LinkedIn, unfortunately, for free accounts, has such a limitation as Commercial Use Limit (CUL). Most likely, you, same as me until recently, have never encountered and never heard about this thing.

image

The point of the CUL is that when you search people outside your connections/network too often, your search results will be limited with only 3 profiles showing instead of 1000 (100 pages with 10 profiles per page by default). How ‘often’ is measured nobody knows, there are no precise metrics; the algorithm decides it based on your actions – how frequently you’ve been searching and how many connections you’ve been adding. The free CUL resets at midnight PST on the 1st of each calendar month, and you get your 1000 search results again, for who knows how long. Of course, Premium accounts have no such limit in place.

However, not so long ago, I’ve started messing around with LinkedIn search for some pet-project, and suddenly got stuck with this CUL. Obviously, I didn’t like it that much; after all, I haven’t been using the search for any commercial purposes. So, my first thought was to explore this limit and try to bypass it.

[Important clarification — all source materials in this article are presented solely for informational and educational purposes. The author doesn't encourage their use for commercial purposes.]
Read more →
Total votes 10: ↑9 and ↓1 +8
Views 15K
Comments 3

Authors' contribution