All streams

Unlock the power of Transformer Neural Networks and learn how to build your own GPT-like model from scratch. In this in-depth guide, we will delve into the theory and provide a step-by-step code implementation to help you create your own miniGPT model. The final code is only 400 lines and works on both CPUs as well as on the GPUs. If you want to jump straight to the implementation here is the GitHub repo.

Transformers are revolutionizing the world of artificial intelligence. This simple, but very powerful neural network architecture, introduced in 2017, has quickly become the go-to choice for natural language processing, generative AI, and more. With the help of transformers, we've seen the creation of cutting-edge AI products like BERT, GPT-x, DALL-E, and AlphaFold, which are changing the way we interact with language and solve complex problems like protein folding. And the exciting possibilities don't stop there - transformers are also making waves in the field of computer vision with the advent of Vision Transformers.

Introduction

For the last five years, there have virtually been almost no global-scale application-layer attacks.

During this period, the industry has learned how to cope with the high bandwidth network layer attacks, including amplification-based ones. It does not mean that botnets are now harmless.

End of June 2021, Qrator Labs started to see signs of a new assaulting force on the Internet – a botnet of a new kind. That is a joint research we conducted together with Yandex to elaborate on the specifics of the DDoS attacks enabler emerging in almost real-time.

At the beginning of the year 2021, Qrator Labs is celebrating its 10 year anniversary. On January 19 our company marks the official passing of a formal 10 years longevity mark, entering its second decade of existence. 

Everything started a little bit earlier - when at the age of 10 Alex saw the Robotron K 1820 - in 2008, when Alexander Lyamin - the founder and CEO of Qrator Labs, approached the Moscow State University superiors, where he worked as a NOC engineer at the time, with an idea of a DDoS-attack mitigation research project. The MSU's network was one of the largest in the country and, as we know now, it was the best place to hatch a future technology.

That time MSU administration agreed, and Mr Lyamin took his own hardware to the university, simultaneously gathering a team. In two years, by summer 2010, the project turned out to be that successful. It courted the DDoS attack of a bandwidth exceeding the MSU's upstream bandwidth capability. And on June 22 MSU superiors gave Mr Lyamin a choice - to shut down or find money to incorporate.

Alexander Lyamin chose to incorporate with his own means, which effectively meant that the needed infrastructure must be built from scratch. The initial design should be distributed instead of concentrated within one network, which resources were not enough for this specific task. And by September 1, 2010, those first server sites were ready and running.

tl;dr

pip install fast-enum

What are enums

# /path/to/package/static.py:
INITIAL = 0
PROCESSING = 1
PROCESSED = 2
DECLINED = 3
RETURNED = 4
...

class MyModelStates:
  INITIAL = 0
  PROCESSING = 1
  PROCESSED = 2
  DECLINED = 3
  RETURNED = 4

TL;DR

The problem we were trying to solve

• Inter-bank transfer systems (see a research paper by Narula et al.)
• Privacy-focused management of digital assets (see a proof of concept by J.P. Morgan and zCash)
• KYC (see a proof of concept by ING)
• Self-sovereign identity (see an attribute-based credentials EU project)
• Voting (see a proxy voting prototype by Russian National Security Depository)

Introduction