Modern neural network training often resembles alchemy. We have working recipes, but how exactly a statistical model transforms terabytes of text into understanding remains unclear.

Why is subliminal learning (pattern transmission through noise) possible? Why does training on synthetic data lead to degradation, even when the data appears to be of high quality?

In this article, I propose looking at training architecture from a different angle. The core idea is simple: positive definitions in high-dimensional space are computationally inefficient. A neural network does not learn what an object is. It learns what the object is not, and the model's intelligence depends entirely on the quality of this "NOT."

What follows is the theory, experiments in PyTorch (code included), mathematics, and an explanation of why LLM collapse is highly probable.

1. Meaning as a Hallucination of Bounded Emptiness

The idea that meaning is born from negation is not new. From apophatic theology (God is unknowable; one can only say what He is not) to Saussure’s structural linguistics and Hegel’s dialectics (“Determinateness is negation”), many authoritative philosophers have reached the same conclusion:

Meaning is not a core within a concept, but the boundaries separating it from the rest of the world.

Extending this thought, meaning is a system of differences (according to Saussure) built through a mechanism of active negation (according to Hegel). Applied to neural networks (LLMs), this can be formulated as:

Meaning is a region of probability space defined by a system of rigid constraints.

For example, an apple is not a list of features (red, round), but a region in concept space that is NOT a pear (close, but differs in shape), NOT a tomato (similar in color, but not a fruit), and NOT a ball (similar shape, but inedible). The boundaries of these negations form meaning as a stable attractor arising from rigid distinction boundaries in a system with limited capacity.

We will call this system of structural negations (boundaries separating a concept from similar but distinct objects) the Shadow of the concept—by analogy with the apophatic tradition, where truth is known through what it is not.

Generalization and hallucination in neural networks are technically isomorphic processes of filling latent space. The distinction between them is topological in nature. If the probability distribution boundaries are rigid (the neural network has a clear signal of “forbidden”), generation remains within a valid spectrum close to fact. If the boundaries are blurred, generation drifts into the realm of plausible noise (hallucinations). Thus, meaning is formed not by accumulating positive examples, but by a system of negative connections—constraints that cut off invalid trajectories.

To understand the mechanics of this process, let us consider information as a unified flow changing its state:

1. Variable Entropy (S_dead). This is the raw material for training: noise, unique details, context, syntactic variations. This is what the model must process. During training, this information is discarded to reduce dimensionality. The higher the variability of the input data, the stronger the pressure on the model's weights.

2. Structural Constraints (S_anti). Shadow). This is the result of entropy processing—boundaries that crystallize under the pressure of the loss function. Here, Hard Negatives (boundary examples) play a critical role. This is entropy with maximum information density: it creates a high error gradient, forcing the model to form a rigid geometry of weights.

It is crucial to understand: a neural network possesses no repository of images. It has only a navigation system based on prohibitions. Meaning is not what the model stores, but where it inevitably lands, being gripped by constraints. If the boundaries (S_anti) are weak, meaning leaks away, turning into a hallucination.

Model intelligence is a derivative of the efficiency of transforming the chaos of examples into a geometry of prohibitions; it depends not on the volume of learned data, but on the quality and rigidity of concept boundaries.

Thus, within the framework of the proposed model, generalization is viewed not as an accumulation of patterns, but as a process of selective compression. To extract an invariant, the system must irreversibly discard unique information about specific examples (entropy, S_dead)and form a new component—the Structural Shadow (S_anti), which defines the boundaries of the permissible. Meaning here is not a set of features, but a residual probability space bounded by rigid prohibitions.

The quality of the model is determined by the precision of this filtering:

Type I Error (Overfitting): If the system has excessively processed variable entropy, the boundaries (S_anti) degenerate — instead of invariant rules, absolute constraints are formed, tied to specific values of training examples. The model works perfectly on the training set but does not generalize to new data.

Type II Error (Boundary Blurring): If the system loses S_anti (boundaries), hallucinations arise. The model generates plausible but logically incorrect content because it lacks a mechanism for cutting off invalid states.

In other words, generalization is the trade-off of memory about facts for an understanding of the topology of what they are not.

It is important to note that any "positive" training of an LLM is, in reality, training through a vast amount of "NOs"; for every predicted token, there are tens of thousands of discarded ones that have provided their signal of negation.

2. Consequences for ML Engineering

The concept explains known empirical facts through a new perspective:

Inefficiency of "Positive" Learning: It is more effective to show the model not a million photos of cats, but a million "almost-cats" that look similar (dogs, tigers, fur hats).
The Nature of Embeddings: Vector proximity in latent space (Cosine Similarity) signifies not so much similarity as the criticality of distinction. Gradient descent creates negative gradients proportional to the product of probabilities: close competitors (high P_j) are actively pushed apart, while distant ones (low P_j) are ignored. Therefore, distinguishing similar concepts (apple/orange) is critically important, whereas distinguishing distant ones (apple/galaxy) is trivial. Embedding space clusters objects not to demonstrate similarity, but to bring them into the firing line of gradient descent.
Inevitability of Collapse: If a model learns on data generated by another model (synthetics), it loses the shadow—information about where the rigid boundaries of the inadmissible lie. This leads to the dissolution of meaning.
Difficulty of Continual Learning: Catastrophic forgetting can be interpreted as the destruction of the system of structural constraints (S_anti) when attempting to adapt to new tasks. Fine-tuning individual facts (expanding S_dead) is relatively safe, but changing fundamental boundaries of distinction requires re-generalizing the entire system. This explains why methods like fine-tuning work for narrow tasks but break down when attempting to radically alter concept understanding.
The Role of Architecture: The success of Transformers over RNNs can be partially explained by the ability to form a denser system of negations: Self-attention explicitly calculates relationships between all tokens, allowing for the selective suppression of irrelevant connections via Softmax, creating a rich negative signal, whereas an RNN accumulates noise in the hidden state during sequential processing.
Dropout as Shadow Sampling: Dropout can be interpreted not only as regularization but as a way to test the stability of the negation system: if the model relies on a single constraint (one path in the network), dropout forces it to form redundant, overlapping boundaries.
Quantization: When quantizing weights (INT8, INT4), we coarsen the geometry of prohibitions. If the boundaries were crystallized with high precision (via rich variable entropy), they are resilient to quantization. If the boundaries are blurred (training on synthetics), quantization destroys them.
Data Augmentation: This artificially increases variable entropy, creating pressure to form more rigid boundaries. However, if augmentation is too aggressive (going beyond the limits of natural manifold diversity), it creates noise instead of structure—the model cannot find the invariant.
The Manifold Hypothesis: This states that real-world data is concentrated on low-dimensional manifolds embedded in high-dimensional feature space. However, the manifold itself is merely a statistical structure describing permissible data variations. Meaning arises not inside the manifold, but at its boundaries—at the points where one manifold (cat) separates from others (dog, fox). In terms of our concept: the manifold encodes variable entropy (S_dead), but meaning crystallizes in structural constraints (S_anti) separating the manifolds. Only then does the cat manifest through the non-cat—through a system of negations cutting off neighboring concepts.

In the truly massive and chaotic text corpora (on which modern LLMs are trained), hard negatives are already present. The sheer diversity of natural language ensures the existence of a multitude of close but incorrect continuations; meaning is built upon distinguishing between them. Standard next-token prediction on such data is substantially apophatic by nature—although the explicit amplification of hard negatives (via contrastive methods or curated datasets) can yield a significant gain in generalization.

Before moving on to the experiments, let us examine the geometry of thought. Why do neural networks (and the brain) choose the strategy of negation? The answer lies in the problem of information packing.

In high-dimensional representations with limited capacity, neural networks are structurally biased toward learning through negation.

Positive definitions (identities) require the allocation of volume in latent space (clusters). Therefore, they inevitably compete with each other, increasing interference as the number of concepts grows. It is difficult to place many distinct objects in a single point in space without collisions.

Negative definitions (distinctions) are implemented as constraints and boundaries—almost orthogonal directions or hyperplanes. In multidimensional space, such planes can overlap and intersect in vast numbers without destroying one another. Positive signals are local; negative signals are global in terms of the space's geometry.

Furthermore, negations are more robust: if a single neuron fails in positive coding, the entire concept is lost; in negative coding, the loss of one neuron merely removes one constraint out of many.

Crucially, I believe that a hierarchy of negations is possible, which, unlike positive coding, collapses down to a finite (discrete) number of distinctions and is reused across different contexts. This grants the neural network compositionality, robustness, scalability, and the capacity for generalization at minimal cost (the best example being the game "Akinator").

As a result, constraints scale significantly more effectively: the system can encode exponentially more differences without expending representational capacity proportionally. This makes learning through negatives (contrasts, hard negatives, class boundaries) not a philosophical choice, but a geometrically and informationally inevitable strategy for effective neural networks.

It is worth adding immediately: meaning as negation does not kill creativity, but creates space for it. A positive definition effectively dictates what exists and what it is like. A negative definition establishes the boundaries of a perimeter. And the infinite set of points within this perimeter leaves room for creativity that corresponds to the truth (does not violate boundaries) but is not obliged to repeat the past.

This sounds like philosophy, but let us try to translate it into code.

3. Experimental Design: Methodology

To illustrate the theory, there was no need for powerful GPUs; a computer with an Intel Core i5-4440 CPU 3.10 GHz sufficed. A task was selected where guessing by context is impossible—logic is required. The task was intentionally chosen to be minimal to exclude semantic noise and isolate the boundary effect.

Task: Array Sort Verification. A sequence of numbers is given. True if it is sorted (x_i <=x_[i+1]). False if it is not.

I used a tiny LSTM (TinyLSTM: hidden_size=32, embedding=8). This creates an artificial resource deficit (Information Bottleneck). The model physically cannot memorize all variants and is forced to find an algorithm.

Two identical models were trained for the same number of steps (1600). The only difference was in the generation of negative examples (class False):

Baseline (Classic Approach): Learns to distinguish a sorted array from random noise (Random Shuffle). This is analogous to training on raw data where the error structure is random.
Concept (Our Approach): Learns to distinguish a sorted array from an array where just one pair of numbers is swapped (Hard Negative swap). This is analogous to training on boundaries.

To ensure the model didn't just learn a local "jag" pattern (glitch) but understood the principle, additional tests were conducted for extrapolation (test on length 40 while training on 6-12, Extrapolation) and multiple errors.

Collapse Simulation:

I simulated a situation where neural networks train on their own texts by launching a chain of generations. Each subsequent generation trained on data labeled by the previous model (pseudo-labeling).

Gen 0: Trained on Ground Truth (Reality).
Gen 1: Trained on Gen 0 (Synthetics).
Gen 2: Trained on Gen 1 (Second-order Synthetics).

Invariant Search Code:

Скрытый текст

import torch
import torch.nn as nn
import torch.optim as optim
import random
import numpy as np
import statistics

# --- CONFIGURATION ---
N_RUNS = 20            # Number of runs for statistics
HIDDEN_SIZE = 32
EMBED_DIM = 8
VOCAB_SIZE = 50
MIN_LEN = 6
MAX_LEN = 12
BATCH_SIZE = 32
LR = 0.005
STEPS = 1600

# --- GENERATORS ---
def generate_sorted(length): return sorted(random.sample(range(VOCAB_SIZE), length))
def generate_random_unsorted(length):
    while True:
        seq = [random.randint(0, VOCAB_SIZE-1) for _ in range(length)]
        if sorted(seq) != seq: return seq
def generate_hard_swap(length):
    seq = generate_sorted(length)
    idx = random.randint(0, length - 2)
    seq[idx], seq[idx+1] = seq[idx+1], seq[idx]
    return seq

def get_batch(mode, batch_size, min_len, max_len):
    inputs, labels = [], []
    length = random.randint(min_len, max_len)
    for _ in range(batch_size):
        if random.random() > 0.5:
            inputs.append(generate_sorted(length))
            labels.append(1.0)
        else:
            if mode == 'baseline': inputs.append(generate_random_unsorted(length))
            elif mode == 'concept': inputs.append(generate_hard_swap(length))
            labels.append(0.0)
    return torch.LongTensor(inputs), torch.FloatTensor(labels).unsqueeze(1)

# --- MODEL ---
class TinyLSTM(nn.Module):
    def __init__(self):
        super().__init__()
        self.embedding = nn.Embedding(VOCAB_SIZE, EMBED_DIM)
        self.rnn = nn.LSTM(EMBED_DIM, HIDDEN_SIZE, batch_first=True)
        self.fc = nn.Linear(HIDDEN_SIZE, 1)
        self.sigmoid = nn.Sigmoid()
    def forward(self, x):
        embedded = self.embedding(x)
        _, (hidden, _) = self.rnn(embedded)
        return self.sigmoid(self.fc(hidden[-1]))

# --- TEST ---
def evaluate_metrics(model):
    model.eval()
    metrics = {}
    
    # 1. Easy (L=20)
    pos = [generate_sorted(20) for _ in range(500)]
    neg = [generate_random_unsorted(20) for _ in range(500)]
    x = torch.LongTensor(pos + neg)
    y = torch.cat([torch.ones(500, 1), torch.zeros(500, 1)])
    with torch.no_grad():
        metrics['easy'] = ((model(x) > 0.5).float() == y).float().mean().item()

    # 2. Hard (L=20)
    neg = [generate_hard_swap(20) for _ in range(500)]
    x = torch.LongTensor(pos + neg) # pos are the same
    with torch.no_grad():
        metrics['hard'] = ((model(x) > 0.5).float() == y).float().mean().item()

    # 3. Extrapolation (L=40) - CONTROL TEST
    pos_long = [generate_sorted(40) for _ in range(500)]
    neg_long = [generate_hard_swap(40) for _ in range(500)]
    x = torch.LongTensor(pos_long + neg_long)
    with torch.no_grad():
        metrics['extrap'] = ((model(x) > 0.5).float() == y).float().mean().item()
        
    return metrics

# --- ONE RUN ---
def train_one_run(seed):
    # Set seed
    torch.manual_seed(seed)
    np.random.seed(seed)
    random.seed(seed)
    
    # Baseline
    model_base = TinyLSTM()
    opt_base = optim.Adam(model_base.parameters(), lr=LR, weight_decay=1e-4)
    for _ in range(STEPS):
        x, y = get_batch('baseline', BATCH_SIZE, MIN_LEN, MAX_LEN)
        opt_base.zero_grad(); nn.BCELoss()(model_base(x), y).backward(); opt_base.step()
    
    metrics_base = evaluate_metrics(model_base)

    # Concept (RESET SEED for identical start)
    torch.manual_seed(seed)
    np.random.seed(seed)
    random.seed(seed)
    
    model_conc = TinyLSTM()
    opt_conc = optim.Adam(model_conc.parameters(), lr=LR, weight_decay=1e-4)
    for _ in range(STEPS):
        x, y = get_batch('concept', BATCH_SIZE, MIN_LEN, MAX_LEN)
        opt_conc.zero_grad(); nn.BCELoss()(model_conc(x), y).backward(); opt_conc.step()
        
    metrics_conc = evaluate_metrics(model_conc)
    
    return metrics_base, metrics_conc

# --- MAIN LOOP ---
def run_statistics():
    print(f"🚀 Starting statistics for {N_RUNS} runs...")
    
    # Storage: keys -> list of values
    hist_base = {'easy': [], 'hard': [], 'extrap': []}
    hist_conc = {'easy': [], 'hard': [], 'extrap': []}
    
    for i in range(N_RUNS):
        print(f"Run {i+1}/{N_RUNS}...", end='\r')
        m_b, m_c = train_one_run(seed=42 + i)
        
        for k in hist_base:
            hist_base[k].append(m_b[k])
            hist_conc[k].append(m_c[k])
            
    print("\n\n📊 FINAL STATISTICS (Mean ± StdDev):")
    print(f"{'Metric':<15} | {'Baseline (Random)':<25} | {'Concept (Hard)':<25}")
    print("-" * 70)
    
    for k in ['easy', 'hard', 'extrap']:
        mean_b = statistics.mean(hist_base[k]) * 100
        std_b = statistics.stdev(hist_base[k]) * 100
        
        mean_c = statistics.mean(hist_conc[k]) * 100
        std_c = statistics.stdev(hist_conc[k]) * 100
        
        print(f"{k.upper():<15} | {mean_b:5.1f}% ± {std_b:3.1f}%           | {mean_c:5.1f}% ± {std_c:3.1f}%")

if __name__ == "__main__":
    run_statistics()

Collapse Code:

Скрытый текст

import torch
import torch.nn as nn
import torch.optim as optim
import random
import numpy as np
import statistics

# --- КОНФИГУРАЦИЯ ---
N_RUNS = 20
HIDDEN_SIZE = 32
EMBED_DIM = 8
VOCAB_SIZE = 50
MIN_LEN = 6
MAX_LEN = 12
BATCH_SIZE = 32
LR = 0.005
STEPS = 1600

# --- ГЕНЕРАТОРЫ (Те же) ---
def generate_sorted(length): return sorted(random.sample(range(VOCAB_SIZE), length))
def generate_random_unsorted(length):
    while True:
        seq = [random.randint(0, VOCAB_SIZE-1) for _ in range(length)]
        if sorted(seq) != seq: return seq
def generate_hard_swap(length):
    seq = generate_sorted(length)
    idx = random.randint(0, length - 2)
    seq[idx], seq[idx+1] = seq[idx+1], seq[idx]
    return seq

# --- ОБУЧЕНИЕ ---
class TinyLSTM(nn.Module):
    def __init__(self):
        super().__init__()
        self.embedding = nn.Embedding(VOCAB_SIZE, EMBED_DIM)
        self.rnn = nn.LSTM(EMBED_DIM, HIDDEN_SIZE, batch_first=True)
        self.fc = nn.Linear(HIDDEN_SIZE, 1)
        self.sigmoid = nn.Sigmoid()
    def forward(self, x):
        embedded = self.embedding(x)
        _, (hidden, _) = self.rnn(embedded)
        return self.sigmoid(self.fc(hidden[-1]))

def get_ground_truth_batch(batch_size, l_min, l_max):
    inputs, labels = [], []
    length = random.randint(l_min, l_max)
    for _ in range(batch_size):
        if random.random() > 0.5:
            inputs.append(generate_sorted(length)); labels.append(1.0)
        else:
            inputs.append(generate_hard_swap(length)); labels.append(0.0)
    return torch.LongTensor(inputs), torch.FloatTensor(labels).unsqueeze(1)

def get_synthetic_batch(teacher, batch_size, l_min, l_max):
    teacher.eval()
    inputs = []
    length = random.randint(l_min, l_max)
    # Mix for generation
    for _ in range(batch_size):
        r = random.random()
        if r < 0.33: inputs.append(generate_sorted(length))
        elif r < 0.66: inputs.append(generate_random_unsorted(length))
        else: inputs.append(generate_hard_swap(length))
    x = torch.LongTensor(inputs)
    with torch.no_grad(): pseudo_labels = (teacher(x) > 0.5).float()
    return x, pseudo_labels

def evaluate_hard(model):
    model.eval()
    pos = [generate_sorted(20) for _ in range(500)]
    neg = [generate_hard_swap(20) for _ in range(500)]
    x = torch.LongTensor(pos + neg)
    y = torch.cat([torch.ones(500, 1), torch.zeros(500, 1)])
    with torch.no_grad(): return ((model(x) > 0.5).float() == y).float().mean().item()

# --- ЦЕПОЧКА ---
def run_chain(seed):
    torch.manual_seed(seed)
    np.random.seed(seed)
    random.seed(seed)
    
    # Gen 0
    g0 = TinyLSTM()
    opt = optim.Adam(g0.parameters(), lr=LR, weight_decay=1e-4)
    for _ in range(STEPS):
        x, y = get_ground_truth_batch(BATCH_SIZE, MIN_LEN, MAX_LEN)
        opt.zero_grad(); nn.BCELoss()(g0(x), y).backward(); opt.step()
    acc0 = evaluate_hard(g0)
    
    # Gen 1
    g1 = TinyLSTM() # New random weights
    opt = optim.Adam(g1.parameters(), lr=LR, weight_decay=1e-4)
    for _ in range(STEPS):
        x, y = get_synthetic_batch(g0, BATCH_SIZE, MIN_LEN, MAX_LEN)
        opt.zero_grad(); nn.BCELoss()(g1(x), y).backward(); opt.step()
    acc1 = evaluate_hard(g1)
    
    # Gen 2
    g2 = TinyLSTM()
    opt = optim.Adam(g2.parameters(), lr=LR, weight_decay=1e-4)
    for _ in range(STEPS):
        x, y = get_synthetic_batch(g1, BATCH_SIZE, MIN_LEN, MAX_LEN)
        opt.zero_grad(); nn.BCELoss()(g2(x), y).backward(); opt.step()
    acc2 = evaluate_hard(g2)
    
    return acc0, acc1, acc2

# --- MAIN ---
def run_collapse_stats():
    print(f"🚀 Запуск статистики Коллапса на {N_RUNS} поколений...")
    res = {'g0': [], 'g1': [], 'g2': []}
    
    for i in range(N_RUNS):
        print(f"Chain {i+1}/{N_RUNS}...", end='\r')
        a0, a1, a2 = run_chain(seed=100 + i)
        res['g0'].append(a0); res['g1'].append(a1); res['g2'].append(a2)
        
    print("\n\n📉 ДИНАМИКА РАСПАДА (Hard Acc, Mean ± Std):")
    print(f"{'Generation':<15} | {'Accuracy':<20}")
    print("-" * 40)
    
    for k in ['g0', 'g1', 'g2']:
        mean = statistics.mean(res[k]) * 100
        std = statistics.stdev(res[k]) * 100
        print(f"{k:<15} | {mean:5.1f}% ± {std:3.1f}%")

if __name__ == "__main__":
    run_collapse_stats()

4. Experimental Results: Statistics and Analysis

To eliminate the factor of lucky initialization, 20 independent runs were conducted for each stage, varying the Master Seed. Aggregated data (Mean ± Std Dev) is presented below.

Stage I. Search for the Invariant

We compared the models' ability to distinguish a fully sorted sequence from chaos (Easy Acc), identify a sequence with only one error (Hard Acc), and operate on unfamiliar lengths (Extrapolation).

Metric	Baseline (Random)	Concept (Hard)	Delta
Easy Acc (Chaos)	100.0% ± 0.1%	97.7% ± 3.1%	-2.3%
Hard Acc (1 Swap)	50.0% ± 0.2%	85.3% ± 7.0%	+35.3%
Extrapolation (L=40)	50.3% ± 0.7%	65.7% ± 7.6%	+15.4%

Analysis of Results:

The Illusion of Competence (Baseline): Look at Easy Acc. The model trained on noise performs perfectly (100%). It brilliantly distinguishes near-order from total chaos. Using standard metrics, it appears the model has learned.

Failure at the Boundary: The Hard Acc metric—recognizing a variant with only a single shift—(50.0% ± 0.2%) with negligible deviation demonstrates that the Baseline is consistently guessing; it has learned a heuristic (the general growth trend), but not the structure.

Generalization (Concept): The model did not just learn a pattern.

It confidently (85%) detects the slightest violations.

In the extrapolation test (length 40, which the model had never seen), the neural network shows a result of 65.7%, which is statistically significantly higher than random guessing (50%). Given the limited memory of the LSTM, this indicates that the model is attempting to apply a learned rule, not memorized examples.

Conclusion: Scale is an inefficient substitute for structure. The Baseline model saw the same number of examples but did not encounter enough boundary cases to form a rule (for a sorting task with length 12 over 1600 trials, the probability of accidentally encountering at least one training example with a single swap was 0.05%). At the scale of LLMs, this is compensated for by trillions of tokens; Scaling Laws work as a brute-force search for boundaries. But our experiment proves: the quality of negation allows achieving a good level of understanding orders of magnitude faster and cheaper. Training on Hard Negatives forms a stable invariant, rather than mere pattern memorization.

Stage II. Degradation of Meaning: Dynamics of Model Collapse

Degradation statistics when training on synthetic data (chain of generations):

Generation	Hard Accuracy (Mean ± Std)	Quality Drop
Gen 0 (Reality)	85.6% ± 4.9%	—
Gen 1 (Echo)	72.8% ± 5.4%	-12.8%
Gen 2 (Noise)	68.0% ± 5.7%	-17.6% (total)

Degradation Mechanism: Gen 0 outputs probabilities (e.g., P=0.93 for a complex case—uncertain, but likely True). Hard labeling converts this into a categorical 1.0, erasing information about uncertainty—it is precisely within this uncertainty that the shadow resides (knowledge of where the boundary lies). Gen 1 trains on distorted labels and loses the understanding of where the real boundary between True and 'almost True' lies. Each generation inherits and amplifies this distortion.

It is important to note: in all generations, Easy Acc remained at the 100% level. Models of the degrading generation appear smart; they pass basic adequacy tests, but Hard Acc plummets.

I also tested the Knowledge Distillation method (training on the teacher's soft labels). This yielded a result of 72.0% for Gen 2. Distillation slows the decay (a gain of +4%), but the trend remains downward.

Interpretation: Synthetic data acts as a filter, blurring the boundaries of meaning and leaving only the general form. When LLMs begin to train on AI-generated text from the internet or flawed synthetic datasets, model intelligence will plummet catastrophically fast unless the density of Hard Negatives is preserved through strict data control.

A fair argument is that modern LLMs are vast and possess sufficient memory to memorize everything, unlike TinyLSTM. However, one must consider that the space of meanings is combinatorially virtually infinite; even a model with a trillion parameters is nothing compared to the potential number of variations. It is precisely when an LLM goes beyond the limits of its memory (into the zone of novelty) that it must rely on learned boundaries. A model trained on noise extrapolates hallucinations; a model trained on Hard Negatives uses invariants.

It is obvious that trillions of tokens of human text contain millions of natural boundary cases. However, this does not apply to the synthetic data on which corporations plan to train the next generations of LLMs. Such data is often statistically sanitized, and the density of Hard Negatives drops practically to zero.

At this point, scale ceases to compensate for the lack of signal. Without Hard Negatives, a model trained on synthetics becomes a confident dilettante—not due to a lack of memory, but due to the narrowness of its training environment. Without an external source of "NO," the system degenerates.

5. The Principle of Boundary Correspondence: Why Architecture Matters

I have spent considerable time experimenting with training small neural networks on Gymnasium. Attempts to teach models to play simple games using the concept of meaning boundaries led to the following conclusions:

Learning efficiency increases sharply when we can explicitly highlight the boundary that determines the correct result.
This boundary is most often specific and works only for a specific task.

Engineers might call this a hack or a cheat, but this is, in reality, the fundamental principle of how neural networks learn and operate. We teach the neural network to achieve a goal, not just for the sake of the process. The desired universalization will fundamentally not work for small neural networks and is redundant for specific tasks.

That is, a universal boundary is either too general or too computationally expensive for small models. Learning efficiency depends on whether the Inductive Bias (the model's built-in perception mechanism) aligns with the topology of the meaning boundary (S_anti).

I will illustrate this with two logical tasks on the same neural network, explicitly feeding it different types of differences:

Task "ZigZag" (Dynamics): A sequence of numbers is considered True if the relationships between adjacent numbers strictly alternate—greater, smaller.

Result: Explicitly feeding the Neighbor Delta (x_t - x_[t-1]) yielded an accuracy boost of +40%. The model found the boundary instantly.

Task "Pivot" (Context): A sequence of numbers is considered True if all subsequent numbers are greater than the first element.

Result: In this case, feeding the Neighbor Delta proved to be noise that reduced quality. However, feeding the Anchor Delta (x_t - x_0) yielded an accuracy of up to 98% (+11% over the baseline model).

Conclusion:

Increasing AI efficiency requires not merely scaling parameters, but providing boundaries (differences) specific to the concrete task. If the architecture does not see the correct delta, it cannot draw the correct conclusions. Instead of increasing parameter counts, it is better to design task-specific input representations.

Why are Transformers more effective than RNNs? It is commonly said that LLMs see the entire context. I would clarify: they have the ability to ignore the entire context, except for what is necessary.

In recurrent networks (RNN), information blends into a mush within the hidden state. Noise (S_dead) accumulates, and separating it from the signal is difficult.

The Self-Attention mechanism works differently. Through the Softmax operation, each token competes for attention against others. To highlight one important connection (e.g., with an anchor number at the beginning of an array), the model must mathematically suppress (negate) hundreds of other, less important connections.

Training a Transformer is the tuning of filters that say "NO" to false correlations (for example, the habit of always looking at the last word) to allow the true, long-range structure to manifest. Pure apophatics in action.

Unlike an RNN, which is rigidly tied to the local context (seeing only the neighbor), the Self-Attention mechanism is a universal scanner. It allows the model to dynamically choose exactly which constraint (S_anti) is relevant right now:

Need to check the neighbor? Attention will suppress all tokens except t-1.
Need to check the beginning of the phrase? Attention will suppress the neighbor and highlight t=0.

The Transformer won not because it knows more, but because its mechanism of global noise suppression (via Softmax) allows it to find and activate the required meaning boundary at any distance.

Zigzag Code:

Скрытый текст

import torch
import torch.nn as nn
import torch.optim as optim
import random
import numpy as np
import statistics

# --- CONFIGURATION ---
N_RUNS = 10
HIDDEN_SIZE = 32
# FOR FAIRNESS:
EMBED_DIM_BASE = 16    # Standard gets wide embeddings
EMBED_DIM_DELTA = 8    # Delta gets narrow ones, but with delta (8+8=16)

VOCAB_SIZE = 50
MIN_LEN = 6
MAX_LEN = 12
TEST_EXTRAP = 40

BATCH_SIZE = 32
LR = 0.005
STEPS = 2000
CHECK_INTERVAL = 200

def setup_seed(seed):
    torch.manual_seed(seed)
    np.random.seed(seed)
    random.seed(seed)

# --- GENERATORS (Zig-Zag) ---
def generate_zigzag(length):
    seq = []
    current = random.randint(0, VOCAB_SIZE-1)
    seq.append(current)
    going_up = True
    for _ in range(length - 1):
        if going_up:
            if current >= VOCAB_SIZE - 1: return generate_zigzag(length)
            next_val = random.randint(current + 1, VOCAB_SIZE - 1)
        else:
            if current <= 0: return generate_zigzag(length)
            next_val = random.randint(0, current - 1)
        seq.append(next_val)
        current = next_val
        going_up = not going_up
    return seq

def generate_hard_broken_zigzag(length):
    seq = generate_zigzag(length)
    idx = random.randint(0, length - 3)
    val_1 = seq[idx]
    val_2 = seq[idx+1]
    if val_2 > val_1: # Up
        if val_2 >= VOCAB_SIZE - 1: return generate_random_unsorted(length)
        val_3_bad = random.randint(val_2 + 1, VOCAB_SIZE - 1) # Up again (Error)
    else: # Down
        if val_2 <= 0: return generate_random_unsorted(length)
        val_3_bad = random.randint(0, val_2 - 1) # Down again (Error)
    seq[idx+2] = val_3_bad
    return seq

def generate_random_unsorted(length):
    return [random.randint(0, VOCAB_SIZE-1) for _ in range(length)]

def get_batch(batch_size, min_len, max_len):
    inputs, labels = [], []
    length = random.randint(min_len, max_len)
    for _ in range(batch_size):
        if random.random() > 0.5:
            inputs.append(generate_zigzag(length))
            labels.append(1.0)
        else:
            inputs.append(generate_hard_broken_zigzag(length))
            labels.append(0.0)
    return torch.LongTensor(inputs), torch.FloatTensor(labels).unsqueeze(1)

# --- MODELS ---

class StandardLSTM(nn.Module):
    def __init__(self):
        super().__init__()
        # WIDE EMBEDDING (16)
        self.embedding = nn.Embedding(VOCAB_SIZE, EMBED_DIM_BASE)
        # Input size = 16
        self.rnn = nn.LSTM(EMBED_DIM_BASE, HIDDEN_SIZE, batch_first=True)
        self.fc = nn.Linear(HIDDEN_SIZE, 1)
        self.sigmoid = nn.Sigmoid()
    def forward(self, x):
        emb = self.embedding(x)
        _, (hidden, _) = self.rnn(emb)
        return self.sigmoid(self.fc(hidden[-1]))

class DeltaLSTM(nn.Module):
    def __init__(self):
        super().__init__()
        # NARROW EMBEDDING (8)
        self.embedding = nn.Embedding(VOCAB_SIZE, EMBED_DIM_DELTA)
        # Input size = 8 + 8 = 16 (SAME AS STANDARD)
        self.rnn = nn.LSTM(EMBED_DIM_DELTA * 2, HIDDEN_SIZE, batch_first=True)
        self.fc = nn.Linear(HIDDEN_SIZE, 1)
        self.sigmoid = nn.Sigmoid()
    def forward(self, x):
        emb = self.embedding(x)
        padded_emb = torch.cat([torch.zeros_like(emb[:, :1, :]), emb[:, :-1, :]], dim=1)
        delta = emb - padded_emb
        combined_input = torch.cat([emb, delta], dim=2)
        _, (hidden, _) = self.rnn(combined_input)
        return self.sigmoid(self.fc(hidden[-1]))

# --- TEST ---
def evaluate_extrap(model):
    model.eval()
    pos = [generate_zigzag(TEST_EXTRAP) for _ in range(250)]
    neg = [generate_hard_broken_zigzag(TEST_EXTRAP) for _ in range(250)]
    x = torch.LongTensor(pos + neg)
    y = torch.cat([torch.ones(250, 1), torch.zeros(250, 1)])
    with torch.no_grad():
        acc = ((model(x) > 0.5).float() == y).float().mean().item()
    model.train()
    return acc

def train_run(model_class, seed):
    setup_seed(seed)
    model = model_class()
    opt = optim.Adam(model.parameters(), lr=LR, weight_decay=1e-4)
    best_acc = 0
    for step in range(1, STEPS + 1):
        x, y = get_batch(BATCH_SIZE, MIN_LEN, MAX_LEN)
        opt.zero_grad()
        loss = nn.BCELoss()(model(x), y)
        loss.backward()
        opt.step()
        if step % CHECK_INTERVAL == 0:
            acc = evaluate_extrap(model)
            if acc > best_acc: best_acc = acc
    return best_acc

def run_comparison():
    print(f"🚀 FAIR Comparison: Standard (Dim 16) vs Delta (8+8=16)...")
    res = {'Std': [], 'Delta': []}
    
    for i in range(N_RUNS):
        print(f"Run {i+1}/{N_RUNS}...", end='\r')
        seed = 7000 + i
        res['Std'].append(train_run(StandardLSTM, seed))
        res['Delta'].append(train_run(DeltaLSTM, seed))
        
    print("\n\n🏆 RESULTS (Best Acc, Extrapolation L=40):")
    m_s = statistics.mean(res['Std']) * 100
    s_s = statistics.stdev(res['Std']) * 100
    m_d = statistics.mean(res['Delta']) * 100
    s_d = statistics.stdev(res['Delta']) * 100
    
    print(f"Standard (Wide): {m_s:5.1f}% ± {s_s:3.1f}%")
    print(f"Delta (Diff):    {m_d:5.1f}% ± {s_d:3.1f}%")

if __name__ == "__main__":
    run_comparison()

Pivot Code:

Скрытый текст

import torch
import torch.nn as nn
import torch.optim as optim
import random
import numpy as np
import statistics

# --- КОНФИГУРАЦИЯ ---
N_RUNS = 10
HIDDEN_SIZE = 32
EMBED_DIM = 8
VOCAB_SIZE = 50
MIN_LEN = 6
MAX_LEN = 12
TEST_EXTRAP = 40       # Экстраполяция на длину 40

BATCH_SIZE = 32
LR = 0.005
STEPS = 2000
CHECK_INTERVAL = 200

def setup_seed(seed):
    torch.manual_seed(seed)
    np.random.seed(seed)
    random.seed(seed)

# --- ГЕНЕРАТОРЫ (Pivot Hard) ---
def generate_pivot_valid(length):
    pivot = random.randint(0, VOCAB_SIZE // 2)
    seq = [pivot]
    for _ in range(length - 1):
        seq.append(random.randint(pivot, VOCAB_SIZE - 1))
    return seq

def generate_hard_pivot(length):
    seq = generate_pivot_valid(length)
    pivot = seq[0]
    if pivot == 0: return [random.randint(0, VOCAB_SIZE-1) for _ in range(length)]
    
    idx = random.randint(1, length - 1)
    err_val = random.randint(0, pivot - 1)
    seq[idx] = err_val
    return seq

def get_batch(batch_size, min_len, max_len):
    inputs, labels = [], []
    length = random.randint(min_len, max_len)
    for _ in range(batch_size):
        if random.random() > 0.5:
            inputs.append(generate_pivot_valid(length))
            labels.append(1.0)
        else:
            inputs.append(generate_hard_pivot(length))
            labels.append(0.0)
    return torch.LongTensor(inputs), torch.FloatTensor(labels).unsqueeze(1)

# --- МОДЕЛИ ---

# 1. Standard
class StandardLSTM(nn.Module):
    def __init__(self):
        super().__init__()
        self.embedding = nn.Embedding(VOCAB_SIZE, EMBED_DIM)
        self.rnn = nn.LSTM(EMBED_DIM, HIDDEN_SIZE, batch_first=True)
        self.fc = nn.Linear(HIDDEN_SIZE, 1)
        self.sigmoid = nn.Sigmoid()
    def forward(self, x):
        emb = self.embedding(x)
        _, (hidden, _) = self.rnn(emb)
        return self.sigmoid(self.fc(hidden[-1]))

# 2. Neighbor Delta (Работало для Зиг-Зага)
class NeighborDeltaLSTM(nn.Module):
    def __init__(self):
        super().__init__()
        self.embedding = nn.Embedding(VOCAB_SIZE, EMBED_DIM)
        self.rnn = nn.LSTM(EMBED_DIM * 2, HIDDEN_SIZE, batch_first=True)
        self.fc = nn.Linear(HIDDEN_SIZE, 1)
        self.sigmoid = nn.Sigmoid()
    def forward(self, x):
        emb = self.embedding(x)
        # Delta = x[t] - x[t-1]
        padded_emb = torch.cat([torch.zeros_like(emb[:, :1, :]), emb[:, :-1, :]], dim=1)
        delta = emb - padded_emb
        combined = torch.cat([emb, delta], dim=2)
        _, (hidden, _) = self.rnn(combined)
        return self.sigmoid(self.fc(hidden[-1]))

# 3. Anchor Delta (Специфично для Pivot)
class AnchorDeltaLSTM(nn.Module):
    def __init__(self):
        super().__init__()
        self.embedding = nn.Embedding(VOCAB_SIZE, EMBED_DIM)
        self.rnn = nn.LSTM(EMBED_DIM * 2, HIDDEN_SIZE, batch_first=True)
        self.fc = nn.Linear(HIDDEN_SIZE, 1)
        self.sigmoid = nn.Sigmoid()
    def forward(self, x):
        emb = self.embedding(x) # [Batch, Seq, Dim]
        
        # Берем первый токен (Anchor) и растягиваем его на всю длину
        anchor_emb = emb[:, 0:1, :] # [Batch, 1, Dim]
        anchor_repeated = anchor_emb.expand(-1, emb.size(1), -1) # [Batch, Seq, Dim]
        
        # Delta = x[t] - x[0]
        delta = emb - anchor_repeated
        
        combined = torch.cat([emb, delta], dim=2)
        _, (hidden, _) = self.rnn(combined)
        return self.sigmoid(self.fc(hidden[-1]))

# --- ТЕСТ ---
def evaluate_extrap(model):
    model.eval()
    pos = [generate_pivot_valid(TEST_EXTRAP) for _ in range(250)]
    neg = [generate_hard_pivot(TEST_EXTRAP) for _ in range(250)]
    x = torch.LongTensor(pos + neg)
    y = torch.cat([torch.ones(250, 1), torch.zeros(250, 1)])
    with torch.no_grad():
        acc = ((model(x) > 0.5).float() == y).float().mean().item()
    model.train()
    return acc

def train_run(model_class, seed):
    setup_seed(seed)
    model = model_class()
    opt = optim.Adam(model.parameters(), lr=LR, weight_decay=1e-4)
    best_acc = 0
    for step in range(1, STEPS + 1):
        x, y = get_batch(BATCH_SIZE, MIN_LEN, MAX_LEN)
        opt.zero_grad()
        loss = nn.BCELoss()(model(x), y)
        loss.backward()
        opt.step()
        if step % CHECK_INTERVAL == 0:
            acc = evaluate_extrap(model)
            if acc > best_acc: best_acc = acc
    return best_acc

def run_comparison():
    print(f"🚀 PIVOT TASK: Architectures Battle (L={TEST_EXTRAP})...")
    res = {'Std': [], 'Neighbor': [], 'Anchor': []}
    
    for i in range(N_RUNS):
        print(f"Run {i+1}/{N_RUNS}...", end='\r')
        seed = 9000 + i
        res['Std'].append(train_run(StandardLSTM, seed))
        res['Neighbor'].append(train_run(NeighborDeltaLSTM, seed))
        res['Anchor'].append(train_run(AnchorDeltaLSTM, seed))
        
    print("\n\n🏆 РЕЗУЛЬТАТЫ (Best Acc):")
    for name, vals in res.items():
        m = statistics.mean(vals) * 100
        s = statistics.stdev(vals) * 100
        print(f"{name:<10}: {m:5.1f}% ± {s:3.1f}%")

if __name__ == "__main__":
    run_comparison()

6. Subliminal Learning: How Owls Slip Through the Numbers

The phenomenon where LLMs transmitted hidden information ("love owls") via a sequence of random numbers was recently discussed. From the perspective of our concept, this is not mysticism.

"Love owls" is not text. It is a topological deformation of the embedding space.

When the teacher model generates even neutral tokens (numbers), its hidden state is under the influence of the "owl" vector. This creates microscopic shifts in the probability of number selection (an interference pattern).

There is no contradiction here with the principle of meaning as negation. The numbers in this stream cease to be neutral. The teacher's hidden state ("love owls") acts as a lens, distorting their probability distribution. This distortion is the shadow (S_anti).

The student model does not simply read numbers. It detects that this stream is NOT random noise. To predict such "incorrect" statistics, the student is forced to activate within itself the same system of internal constraints ("love owls" vector) that created this distortion in the teacher. This is the transmission of form through the deformation of space, not through the exchange of facts.

However, regarding this learning channel, there remain more questions than answers—the efficiency of data transfer is unclear (on one hand, only constraints are transferred; on the other, they are embedded in raw data), and it is difficult to assess the degree of influence of the transferred patterns, and so on.

7. Embeddings: Similarity and Difference

Mathematical Justification: Why Softmax is a Negation Operator

The standard interpretation of Embedding Space is well known:

cosine_similarity("apple", "orange") approx 0.85 (Close)

cosine_similarity("apple", "galaxy") approx 0.02 (Far)

From this, the conclusion is usually drawn: proximity is similarity. I propose to complement this picture: proximity also determines the zone of critical distinction.

Distinguishing an apple from a galaxy is trivial (Easy Negative). Distinguishing an apple from a pear is difficult and important (Hard Negative).

To understand why proximity = importance of negation, let's look at the mathematics of training. When updating weights via gradient descent, Softmax derivatives show the force of impact on each token:

P(token_i | context) = exp(s_i) / Sigma exp(s_j)

Where:

The Numerator works for the Positive (pulls the correct token up).
The Denominator (Sigma exp(s_j)) is the Shadow. It is the sum of all competitors.

Look at the Softmax derivatives (how probability changes when updating weights):

For the correct token i: dP_i / ds_i = P_i * (1 - P_i)
For the incorrect token j: dP_i / ds_j = -P_i * P_j

What does this mean in practice?

The minus sign in the second formula signifies suppression. But the strength of this suppression depends on P_j.

If P_j is small (token "galaxy", far in embeddings), the gradient is almost zero. The model does not waste energy on the obvious.
If P_j is large (token "orange" — a strong competitor), the negative gradient becomes significant, actively pushing these vectors apart.

The introduction of the temperature parameter in Softmax (P(i) = exp(s_i/T) / Sigma exp(s_j/T)) proves that semantic vector proximity is a zone of active competition, not just co-occurrence in contexts.

Temperature acts as a coefficient of rigidity for structural constraints (S_anti):

At T -> 0: The system maximizes competitor suppression. The probability of nearest Hard Negatives is forcibly zeroed out, forming absolute boundaries of meaning.
At T -> infinity: Suppressive capability disappears. The distribution becomes uniform, and the vectors of the correct token and its Hard Negative (e.g., "apple" vs "orange") become indistinguishable.

Hallucinations at high temperature are a statistical breach of the boundary by the nearest Hard Negative. The model does not output random noise—it substitutes the correct token with one that is semantically close but incorrect in the given context. This confirms that proximity in embedding space reflects not object similarity, but the criticality of their distinction—the risk of substitution, which Softmax must actively curb.

Optimal temperature is a balance between suppressing incorrect Hard Negatives (accuracy) and allowing creative Hard Negatives (insights). If synthetic data for training was generated at high T, the system inherits the teacher's blurred boundaries.

Conclusion: Vector proximity in embedding space is an equilibrium between attraction (shared context) and repulsion (distinction amidst confusion). The final proximity reflects the frequency and importance of distinguishing these concepts during training.

Although mathematically Softmax contains negation, in practice, without Hard Negatives, this signal is too weak and blurred to quickly form rigid boundaries; hence the necessity for trillions of tokens.

Negative Sampling in Word2Vec works similarly:

Objective = log sigma(w c) + Sigma log sigma(-w c_neg)

The second term is the explicit maximization of dissimilarity (negation) of the context against random words.

Initially, negatives are random. But as training progresses and similar words move closer, random sampling automatically begins to hit hard negatives. This is natural curriculum learning: easy -> hard, without explicit instruction.

Thus, embedding space is not merely a space of similarity. It is a system of negative connections in which:

proximity means the importance of distinction,
training is the carving of boundaries,
meaning is what remains after exclusions.

This is precisely why flawed synthetic datasets kill models, why hard negatives are more important than data volume, and why understanding is always the art of negation.

8. The Nature of Fact and RAG

Within the framework of our theory, facts are not stored in the model like data on a hard drive (in the form of S_dead). They exist as a topological inevitability. A fact is the limiting case of meaning, where the system of constraints (S_anti) becomes so dense that it narrows the probability corridor down to a single possible variant. The model writes "Paris" after "Capital of France" not because it remembers it, but because all other options (London, Berlin, Mars) are blocked by rigid weights.

Here, the engineering role of tokenization is critical. A token is a basic invariant, pre-cleaned of raw entropy. The neural network does not waste layer depth understanding what letters "Paris" consists of—it receives the ready-made concept [Paris].

If tokenization is high-quality (semantic), the model spends its capacity on building a complex topology of connections between concepts.
If tokenization is garbage (character-level/byte-level), the model is forced to burn resources on creating primary invariants from noise. It lacks the depth remaining to fix rigid, high-level factual boundaries.

Thus, a fact is not a memory cell, but geometry tightly stretched over a grid of token-invariants.

This explains the limitations of RAG (Retrieval-Augmented Generation). RAG reinforces the positive side (tossing facts into the context) but does not form structural negations in the model's weights. This makes it effective for citation tasks, but insufficient for deep understanding. A model with RAG is like a student with a cheat sheet: they can answer the question, but they do not master the subject.

Few-shot learning and In-context learning work only if the Base Model already contains S_anti learned during pre-training. A prompt can activate an existing apophatic structure, but it cannot create it from scratch. Without internal structure, a prompt becomes merely a template for copying, and the model immediately hallucinates when stepping outside the examples.

Conclusion

The current trend suggests that to create AGI, we need more data and more parameters. In this article, I wanted to show that this is a difficult path fraught with immense costs and hallucinations. The proposed perspective on neural network training and cognition might seem unconventional, but it can be quite useful in real-world applications:

Positive knowledge without a system of constraints is an illusion of understanding.
Scale does not replace structure. The Baseline model saw the same volume of data but remained unintelligent.
The synthetic crisis is real. Without an external source of "NO" (Ground Truth Hard Negatives), AI degenerates into a simulacrum that looks perfect but lacks essence.
Neural networks have no goal or plan; any inference is merely interference with the prompt within the framework of constraints.
The future of AI lies not in parsing the entire internet, but in creating datasets consisting of high-quality errors and paradoxes. We need to teach models not what is, but what cannot be. In practice, this means focusing on hard negatives (edge cases) rather than volume when creating datasets. For synthetic data, it is critically important to mix in 20–30% real examples (Ground Truth) as protection against model collapse.

P.S. This table presents an interpretation of certain phenomena in ML engineering through the lens of the proposed concept.

Phenomenon	Classical Explanation (ML/DL)	Explanation via the Concept of Negative Definition
Grokking	Transition from overfitting to generalization. The model finds a flatter minimum of the loss function after a long plateau.	Phase transition in entropy processing. The plateau corresponds to the accumulation of gradient pressure, after which the system abruptly switches from extensional storage of examples (incomplete entropy processing) to intensional crystallization of the rule (formed structural constraints).
Model Collapse	Accumulation of statistical errors, disappearance of distribution tails, reduction of variance when training on synthetics.	Loss of boundary resolution. Synthetic data contains positive examples, but variable entropy within them is pre-smoothed (averaged forms instead of edge cases). The teacher model has already processed chaos into structure, and the student model receives only the result—without the initial pressure that forms rigid boundaries. In terms of information theory: information density at class boundaries drops to zero, and the system cannot crystallize its own structural constraints.
Adversarial Examples	Linearity of models in high-dimensional spaces. A small vector change flips it across the decision boundary.	Insufficient density of constraints. The model learned positive features but did not form negative constraints (S_anti) in orthogonal directions. The decision boundary is defined only along the data manifold but is unprotected against orthogonal perturbations outside of it.
Hallucinations	Probabilistic nature of next-token prediction. The model generates plausible text without verifying its truth.	Desynchronization of generator and filter. The token combinatorics mechanism (working with S_dead) functions correctly, but the structural constraints mechanism (S_anti) does not block inadmissible sequences. In terms of softmax: the denominator (S_anti) is insufficiently saturated with competing constraints, allowing gaps to be filled with the plausible but incorrect.
Double Descent	Effect of overparameterization. First bias-variance trade-off, then noise interpolation, then generalization.	Dynamics of noise elimination. The first descent is an attempt to memorize all variable entropy. The error peak is the moment when the system begins to process entropy into constraints, but the process is incomplete (interference). The second descent is the successful crystallization of structural constraints.
In-Context Learning	The Attention mechanism works as a functional analog of local gradient update during inference.	Interference tuning of the Shadow. The prompt acts not as data for memorization, but as a control signal (stimulus). It interacts with the global S_anti field, temporarily regrouping the prohibition topology. Few-Shot examples do not "teach" the model, but dampen irrelevant probability branches in the activation space, leaving only the required meaning projection accessible.
Chain of Thought (CoT)	Increasing the computational budget of the task. Intermediate tokens serve as a memory buffer.	Explicit serialization of constraints. Breaking down the task forces the model to explicitly verify compliance with logical boundaries (S_anti) at each inference step, preventing error accumulation through local constraint validation.
Superposition	Efficient information packing. Vectors are almost orthogonal in high dimensions.	Orthogonality of constraints (S_anti). Different invariants and rules (S_anti) can be encoded by the same weights if they lie in almost orthogonal subspaces. The model saves resources by packing non-conflicting prohibitions into a single physical medium without mutual interference. Interference is absent due to context mismatch, allowing scalability of representations in high dimensions.
Lottery Ticket Hypothesis	Initialization matters. Pruning leaves only successful gradient paths.	Isolation of the structural core. Overparameterization is necessary for the stochastic search of a topology that coincides with the required constraint structure. The winning ticket is the subnetwork encoding S_anti. The remaining weights encode redundant noise (S_dead) and are subject to removal.
Scaling Laws	Power laws. Performance increases with the growth of data and compute.	Energy cost of compression. Extracting high-level invariants (deep crystallization of structural constraints) requires processing an exponentially larger volume of variable entropy. Each new level of the boundary hierarchy is formed only upon reaching a critical mass of examples with sufficient information density.
Subliminal Learning	Models find hidden correlations in token distribution (steganography).	Topological resonance. Information is transmitted through the distortion of probability distributions (negative). The student model restores the teacher's constraint structure, adapting weights to observed anomalies in the distribution.
RAG	Providing the model with up-to-date factology from an external database.	Externalization of entropy. Artificially holding facts (S_dead) in context reduces the need to compress them into internal weights. This blocks the formation of deep internal invariants, replacing them with a retrieval procedure. This reinforces retrieval (positive), but without internal S_anti, the model remains "quoting" rather than understanding.
Catastrophic Forgetting	Overwriting old knowledge with new.	Destruction of the S_anti structure. Loss of the constraint hierarchy leading to cyclic rebirth.
Emergent Abilities	Unexpected abilities at large scale.	Phase transition upon accumulating a critical mass of S_anti, allowing the formation of a deep hierarchy of new invariants.
Catastrophic Interference (Bias Amplification)	Amplification of biases in data (e.g., gender bias in embeddings).	Accumulation of a silent shadow (subliminal uncontrolled invariants) without explicit negation.
Induction Heads	Neural circuits implementing the algorithm [A][B]...[A] -> [B]. The basis of In-Context Learning. Attention heads find the previous occurrence of the current token and transfer probability to the element following it.	Crystallized meta-invariant. A rigid topological structure formed during pre-training to compress repetitive entropy. Works as a radical negation operator: upon resonance with context, the head suppresses the probability of the entire vocabulary, leaving admissible only the token dictated by the pattern structure. This is not "learning in the moment," but deterministic execution of a constraint.

All listed phenomena can be interpreted as various modes of formation, destruction, or temporary reconfiguration of negation structures (S_anti) within the learning system.

Apophatic AI: Why Neural Networks Learn Through «NO» and How Synthetic Data Kills Meaning