Machine learning *

← Предыдущая глава

О чем говорит гипотеза о лотерейном билете, и чем она полезна на практике, если оказывается верной?

Гипотеза о лотерейном билете — это идея, которая появилась в 2018 году в контексте обучения нейронных сетей. Она утверждает, что в случайно инициализированной нейронной сети существует подсеть (или «выигрышный билет»), которая, если ее обучить независимо, сможет достичь такой же точности на тестовом датасете, как и полная сеть после такого же количества шагов обучения. Авторы гипотезы — Джонатан Франкл и Майкл Карбин.

В этой главе мы подробно рассмотрим гипотезу лотерейного билета. Сначала мы шаг за шагом разберемся, как она работает, а затем обсудим прунинг весов — один из ключевых методов, который позволяет создавать более компактные сети. Этот процесс является частью методологии, основанной на гипотезе лотерейного билета. В конце главы мы проанализируем практические применения и ограничения данной гипотезы.

These days, it seems like everyone is talking about AI. AI here, AI there—AI will replace us all, and so on. I started to wonder: how exactly is AI going to replace us? I decided to dig into this question and examine the technical foundations, mainly to understand it for myself—how exactly is AI supposed to replace us all? Spoiler: it isn’t planning to just yet, but what’s already available today is impressive.

CAPTCHA has become a familiar part of the internet: distorted texts, “find all the traffic lights” images, audio riddles, and other challenges that distinguish humans from machines. Every bot-system developer or QA engineer automating web scenarios has at least once run into a script suddenly stumbling over a CAPTCHA. A natural question arises: can a program be taught to solve CAPTCHAs the way a human does—quickly and reliably? In this article I will try to figure out how AI CAPTCHA solvers are built, from classical OCR methods to modern neural networks.

We at Verilog Meetup constructed an exam/interview problem that has an interesting property: if a student tries to figure out a solution by thinking by himself, he usually succeeds; however if he dumps the problem on ChatGPT, the solution fails (does not pass the automated test), and the student goes into a death spiral of futility, kicking ChatGPT to get the solution right.

There is nothing weird about the problem, we do this in the industry all the time:

Admit it, how many times have you wanted to quickly create an image for a post or presentation, but instead got stuck in an editor or endless searches for a suitable image on Google? Wouldn't it be great if the picture in your head could just appear instantly? Time is money, inspiration is on pause, and that's where AI comes to the rescue. Neural networks can generate anything you want, including the craziest ideas. No need to spend hours searching when, with a few clicks, you can see what was in your thoughts just a second ago.

By the way, notice the cover with the dinosaur? Let's call him Rex. Rex is himself a product of neural network creation. Today he'll be the main star of our experiments. But what will we do? Remember I mentioned crazy ideas? Well, to understand all the possibilities of generation, let's give AI a difficult task. We'll send Rex somewhere in space, for example to the Moon, let him put on a spacesuit and and have him grill some barbecue with Earth in the background. Interested? Then buckle up, we're heading into the world of image generation.

The explicit reparameterization trick is often used to train various latent variable models due to the ease of calculating gradients of continuous random variables. However, due to its peculiarities, explicit reparameterization trick is not applicable to several important continuous standard distributions, such as mixture, Gamma, Beta and Dirichlet.

An alternative method for calculating reparameterization gradients relies on implicit differentiation of cumulative distribution functions. The implicit reparameterization trick is much more expressive and applicable to a wider class of distributions

This article provides an overview of various reparameterization tricks and announces a new Python library, irt.distributions, for sampling from various distributions using the implicit reparameterization trick.

Over the past two years, artificial intelligence has become one of the main topics in the media and many famous people have expressed their thoughts on this topic. But if you start searching on the Internet for collections of quotes about AI, you will mostly find quotes from CEOs of multi-billion dollar companies, futurists and scientists conducting research in this field. Moreover, these collections are so similar to each other, which sometimes gives the impression that they were compiled by AI. In this article, I have collected quotes from world famous people who are usually not included in such collections of quotes:

My name is Paul Karol and I work as a director in a Russian IT company that mostly sells their products in the international market.

I have been deeply involved in a project and I haven’t had time to write here lately.  

But out of this work came an understanding of two very large Mistakes that are being made that prevent your pre-sales from succeeding. 

I’m going to go into deeper detail on each one of these issues in the next articles, but I will introduce them here now.

Let me make you a promise….. if you correct these two issues, your software and development projects will sell.  

**I have a proven track record. If these areas are fixed the company begins to sell 150 % more than they’ve ever sold in the past. 

150% more profits! 

For one of my projects I was exploring Reddit to understand how players create characters in video games, what is important to them in this process, and what their preferences are. It turns out that communities sharing their creations or seeking help with specific character designs remain active even for games released years ago. This realization sparked the idea for a hobby project that could assist these players in creating the characters they envision.

ChatGPT-4, the latest model from OpenAI, boasts impressive capabilities like text generation, question answering, problem-solving, coding, and even image analysis. However, accessing it requires a $20 monthly subscription on OpenAI's website. For residents of certain countries, accessing the service poses additional challenges due to restrictions, necessitating the use of foreign payment methods and VPNs.

We've created a list of the top-4 services that offer completely free access to ChatGPT-4. This article will delve into the advantages and limitations of each option, comparing them side-by-side.

Recently, there has been a lot of talk about the success of artificial intelligence (AI), although this usually means another achievement in the field of generative neural networks.

And few people, speaking about AI, try to explain what they themselves understand by the term “artificial intelligence.” After all, it’s one thing to write about “AI problems,” and quite another to endow an ordinary computer algorithm with at least the rudiments of intelligence.

After all, the etymology of the established phrase “artificial intelligence” is not unambiguous and can take on different meanings depending on what meaning the author is trying to put into it.

Machine Learning (ML) technologies are becoming increasingly popular and have various applications, ranging from smartphones and computers to large-scale enterprise infrastructure that serves billions of requests per day. Building ML tools, however, remains difficult today because there are no industry-wide standardised approaches to development. Many engineering students studying ML and Data Science must re-learn once they begin their careers. In this article, I've compiled a list of the top five problems that every ML specialist faces only on the job, highlighting the gap between university curriculum and real-world practice. 

Problem

Trendwatching is a powerful tool for driving strategic innovations. It helps to discover new teсhnologies, business models and products, that may be used for idea generation and technology transfer. It is a powerful tool for product managers, business stream managers, top managers and "strategists" and is mostly used on a regular basis.

I know it can be hard to learn a new programming language. In this article, I want to share my plan with you. It's a way to learn Python in eight weeks using videos, articles, and practice exercises. Exercises are very important because I think the best way to learn is by doing them.

I've created this learning plan for people who don't have much free time. You only need about 30-50 minutes a day and consistency. In my plan, I use the 80/20 principle, which will help you learn the most important things first and improve the rest through practice.

For those who read this article to the end, I have prepared a learning tracking sheet to help you track your progress.

In this third part, we will discuss Machine Learning, specifically the prediction task in the context of information theory.

The concept of Mutual Information (MI) is related to the prediction task. In fact, the prediction task can be viewed as the problem of extracting information about the signal from the factors. Some part of the information about the signal is contained in the factors. If you write a function that calculates a value close to the signal based on the factors, then this will demonstrate that you have been able to extract MI between the signal and the factors.

Many of us have heard stories where one was able to complete days worth of work in minutes using AI, even being outside of one's area of expertise. Indeed, often LLM's do (almost) miracles, but today I had a different experience:

In Part 1, we became familiar with the concept of entropy.

In this part, we will delve into the concept of Mutual Information, which opens doors to error-resistant coding, compression algorithms, and offers a fresh perspective on regression and Machine Learning tasks.

It is an essential component that will pave the way, in the next section, for tackling Machine Learning problems as tasks of extracting mutual information between features and the predicted variable.

Here, there will be three interesting and crucial visualizations.

The first one will visualize entropy for two random variables and their mutual information.
The second one will shed light on the very concept of dependency between two random variables, emphasizing that zero correlation does not imply independence.
The third one will demonstrate that the bandwidth of an information channel has a straightforward geometric interpretation through the convexity measure of the entropy function.

In the meantime, we will prove a simplified version of the Shannon-Hartley theorem regarding the maximum bandwidth of a noisy channel. Let's dive in!

I've long wanted to create educational materials on the topic of Information Theory + Machine Learning. I found some old drafts and decided to polish them up here, on Habr.

Information Theory and Machine Learning seem to me like an interesting pair of fields, the deep connection between which is often unknown to ML engineers, and whose synergy has not yet been fully revealed.

Let's start with basic concepts like Entropy, Information in a message, Mutual Information, and channel capacity. Next, there will be materials on the similarity between tasks of maximizing Mutual Information and minimizing Loss in regression problems. Then there will be a section on Information Geometry: Fisher metric, geodesics, gradient methods, and their connection to Gaussian processes (moving along the gradient using SGD is moving along the geodesic with noise).

It's also necessary to touch upon AIC, Information Bottleneck, and discuss how information flows in neural networks – Mutual Information between layers (Information Theory of Deep Learning, Naftali Tishby), and much more. It's not certain that I'll be able to cover everything listed, but I'll try to get started.

This is a translated and adopted article I wrote for the Aha'22 (30 May 2022) conference. It describes an approach to a marketplace prices optimisation. Here I've outlined some important definitions and tried to define the scopes and roles of ML, algorithms, and humans in optimal pricing. Although the article covers rather basic things, still, you can find out some new formulas and ideas, because these basics are somewhat "well-known only in a very closed clubs", and besides, the real gem found here is the detailed recipe for ML engineers how to build optimal pricing systems.

In this post I'll share my experience in adjustment of WiFi physical channel. The channel was implemented on a software defined radio (SDR) platform. WiFi looks like a very complicated thing standardized over hundreds of pages. Could a non-expert with a PC and a couple of 100$ devices (HackRFs) somehow improve it? Here I try to develop a WiFi optimization approach basically agnostic of protocol implementation details. There's some math and Python programming in it.