Pull to refresh
467.52

Python *

Interpreted high-level programming language for general-purpose programming

Show first
Rating limit
Level of difficulty

Building a GPT-like Model from Scratch with Detailed Theory and Code Implementation

Reading time14 min
Views39K

Unlock the power of Transformer Neural Networks and learn how to build your own GPT-like model from scratch. In this in-depth guide, we will delve into the theory and provide a step-by-step code implementation to help you create your own miniGPT model. The final code is only 400 lines and works on both CPUs as well as on the GPUs. If you want to jump straight to the implementation here is the GitHub repo.

Transformers are revolutionizing the world of artificial intelligence. This simple, but very powerful neural network architecture, introduced in 2017, has quickly become the go-to choice for natural language processing, generative AI, and more. With the help of transformers, we've seen the creation of cutting-edge AI products like BERT, GPT-x, DALL-E, and AlphaFold, which are changing the way we interact with language and solve complex problems like protein folding. And the exciting possibilities don't stop there - transformers are also making waves in the field of computer vision with the advent of Vision Transformers.

Read more

Tips and tricks from my Telegram-channel @pythonetc, December 2019

Reading time2 min
Views1.6K


It is a new selection of tips and tricks about Python and programming from my Telegram-channel @pythonetc.

Previous publications.


Different asyncio tasks obviously have different stacks. You can view at all of them at any moment using asyncio.all_tasks() to get all currently running tasks and task.get_stack() to get a stack for each task.
Read more →

Tips and tricks from my Telegram-channel @pythonetc, November 2019

Reading time3 min
Views2.7K

Tips and tricks from my Telegram-channel @pythonetc, November 2019

It is a new selection of tips and tricks about Python and programming from my Telegram-channel @pythonetc.

Previous publications.



PATH is an environment variable that stores paths where executables are looked for. When you ask your shell to run ls, the shell looks for the ls executable file across all paths that are presented in PATH.
Read more →

Faster ENUM

Reading time9 min
Views2.5K

tl;dr


github.com/QratorLabs/fastenum
pip install fast-enum

What are enums


(If you think you know that — scroll down to the “Enums in Standard Library” section).

Imagine that you need to describe a set of all possible states for the entities in your database model. You'll probably use a bunch of constants defined as module-level attributes:
# /path/to/package/static.py:
INITIAL = 0
PROCESSING = 1
PROCESSED = 2
DECLINED = 3
RETURNED = 4
...

...or as class-level attributes defined in their own class:
class MyModelStates:
  INITIAL = 0
  PROCESSING = 1
  PROCESSED = 2
  DECLINED = 3
  RETURNED = 4

That helps you refer to those states by their mnemonic names, while they persist in your storage as simple integers. By this, you get rid of magic numbers scattered through your code and make it more readable and self-descriptive.

But, both the module-level constant and the class with the static attributes suffer from the inherent nature of python objects: they are all mutable. You may accidentally assign a value to your constant at runtime, and that is a mess to debug and rollback your broken entities. So, you might want to make your set of constants immutable, which means both the number of constants declared and the values they are mapped to must not be modified at runtime.
Read more →

Tips and tricks from my Telegram-channel @pythonetc, August 2019

Reading time4 min
Views1.6K


It is a new selection of tips and tricks about Python and programming from my Telegram-channel @pythonetc.

Previous publications


If an instance of a class doesn’t have an attribute with the given name, it tries to access the class attribute with the same name.

>>> class A:
...     x = 2
...
>>> A.x
2
>>> A().x
2
Read more →

Contextual Emotion Detection in Textual Conversations Using Neural Networks

Reading time10 min
Views3.9K

Nowadays, talking to conversational agents is becoming a daily routine, and it is crucial for dialogue systems to generate responses as human-like as possible. As one of the main aspects, primary attention should be given to providing emotionally aware responses to users. In this article, we are going to describe the recurrent neural network architecture for emotion detection in textual conversations, that participated in SemEval-2019 Task 3 “EmoContext”, that is, an annual workshop on semantic evaluation. The task objective is to classify emotion (i.e. happy, sad, angry, and others) in a 3-turn conversational data set.
Read more →

Google News and Leo Tolstoy: visualizing Word2Vec word embeddings using t-SNE

Reading time7 min
Views14K

Everyone uniquely perceives texts, regardless of whether this person reads news on the Internet or world-known classic novels. This also applies to a variety of algorithms and machine learning techniques, which understand texts in a more mathematical way, namely, using high-dimensional vector space.

This article is devoted to visualizing high-dimensional Word2Vec word embeddings using t-SNE. The visualization can be useful to understand how Word2Vec works and how to interpret relations between vectors captured from your texts before using them in neural networks or other machine learning algorithms. As training data, we will use articles from Google News and classical literary works by Leo Tolstoy, the Russian writer who is regarded as one of the greatest authors of all time.

We go through the brief overview of t-SNE algorithm, then move to word embeddings calculation using Word2Vec, and finally, proceed to word vectors visualization with t-SNE in 2D and 3D space. We will write our scripts in Python using Jupyter Notebook.

Read more →

Making a DIY thermal camera based on a Raspberry Pi

Reading time6 min
Views61K
image

Hi everyone!

Winter has arrived, and so I had to check the thermal insulation of my out of town residence dacha. And it just turned out a famous Chinese marketplace started to sell cheap thermal camera modules. So I decided to DIY it up and build a rather exotic and useful thing — a heat visor for the home. Why not? Especially since I had a Raspberry Pi lying around anyway… The result is down below.
Read more →

Authors' contribution