Can you train a cyberattack detector without ever showing it a real cyberattack?
It sounds like a contradiction. If you want a neural network to detect lateral movement, you would expect to show it lateral movement. I did the opposite: I generated an entire corporate network with its login history, staged an attack inside that artificial world, and trained networks on it. Not a single real row in the training data. The whole world is a 135-line config; each network has four thousand parameters and trains in seconds on a laptop, and the best result came from six of them, trained on six different invented worlds.
Then I pointed them at real data: the authentication logs of Los Alamos National Laboratory, 1.65 billion events, with red-team exercises labelled in them.
And it worked. The networks rank 3.6 million windows by suspicion, and the top twenty-three rows of that list hold sixteen real attacks and seven false alarms: all the analyst has to do is open those rows. A threshold counter on the same data needs a hundred and sixty-one thousand false alarms to reach the sixteenth attack. By AUC the synthetic training landed inside the range of published research trained on real labelled data, although the two cannot be compared head-on, and I will explain why.
No miracle happened. Something else did: twice I wrote down an elegant conclusion, and twice the experiment made me take it back. And I found the loop that makes a data generator worth writing at all: the detector’s error points at one specific machine, you work out which phenomenon your invented world lacks, you add two lines to the config, and the error is gone.
The rest, in order: who I am and why I needed this, what lateral movement is, how the invented world is built, what the exam showed, and how much of it is honest. If you do not need the theory, skip ahead to “The first result on real data”.
I write TDCV2, an open-source constructor for test data, and I wanted to know what it is actually good for. Filling a table with fake Smiths is one thing; producing data that something working can be trained on is another. You can’t check that on a toy, so I took a real information-security problem, trained a classifier on invented data only, and turned it loose on the logs of a real network at Los Alamos National Laboratory.
Three caveats, so the rest reads the way it should.
I am not a machine-learning specialist. Everything below is the work of a person who writes a data generator and decided to test it on a serious problem. If it looks to you like I’m reinventing a wheel somewhere, or reaching for the wrong tool - you’re probably right.
I am not competing with anyone. The results from research groups further down are there as a coordinate system, so the reader has a sense of scale, not because I’m claiming a place in that row.
I wasn’t counting on a result. Honestly, I expected the synthetic data to do worse! It went differently, and that’s the interesting part. There was no miracle anyway, but two things happened that are more interesting than a miracle: a measured boundary of a whole class of methods, and a limit on training from generated data that I then retracted myself.
What lateral movement is
Since the headline opens with a bit of jargon, let me explain what this beast is first. All the more so because the phenomenon is built in a non-obvious way, and half of your intuition about “hacking” doesn’t apply.
There’s a common picture in which the attacker cracks a password and that’s the end of it. In reality, getting a login somewhere is only the beginning, and it is almost never the place they wanted. The first thing to fall is somebody’s workstation: a person opened an attachment, or a service with an old hole was exposed to the outside. There is nothing valuable on that machine. The valuable things are in the database, on the file server, in the domain controller - and those still have to be reached.
Reaching them is what’s called lateral movement. In the MITRE ATT&CK classification it is a tactic of its own, TA0008, and its most troublesome technique is T1078, Valid Accounts.
How it spreads
The mechanism is simple. A captured machine almost always has somebody else’s credentials lying on it: a cached domain login, a token in memory, a password in a maintenance script, a saved connection. The attacker takes them and logs in to the neighbouring machine - an ordinary login, done exactly the way a real employee does it. There they collect that machine’s credentials. And round it goes.
After that it’s arithmetic: every step grows both the number of machines and the number of available accounts, and every new account opens new machines. One workstation becomes ten, ten become a hundred.
The speed is nothing like the movies. Each step takes hours, the whole picture unfolds over days and weeks. That, broadly, is the one piece of good news - there is time to notice.
Who this affects, and who it doesn’t
Lateral movement needs somewhere to move, laterally. It lives in large corporate networks: hundreds and thousands of machines, a shared domain, service accounts with access to many places at once, administrators who legitimately go everywhere.
A small business is not threatened by this phenomenon as such - not because it is safer there, but because there is nowhere to “move”. If a company is five people, three laptops and all the work in cloud services, then an attacker who has taken the accountant’s laptop is already where the money is. They need no “tactic”, it’s all at hand. Such a company will have other problems, and those are caught differently.
So everything below is about a network with somewhere to go.
Why it is hard to catch
An awkward property of this phenomenon. Every single lateral-movement event is flawless: the password is correct, the answer is “success”, the account is genuine, the domain is our own, the time is business hours. There is nothing anomalous in it, because it really is an ordinary login - just made by the wrong person.
To a signature detector, and to any threshold counter, it is indistinguishable from an employee doing their job.
Why I needed this particular problem
Now I can come back to where I started - the generator.
Testing a data constructor on a problem whose answer is known in advance isn’t interesting. You want one where invented data will either work or fail honestly, and where the outcome can be measured rather than eyeballed. Lateral movement fitted for three reasons, and each of them matters.
Nobody has labelled examples. To train a detector the usual way, you need examples of the attack in your own network. And to get them, you first have to be compromised, notice it, and label it. Synthetic data here isn’t a shortcut, it’s the only way - if it works at all.
The phenomenon has a computable definition. “An account logs in where it has never been, from a machine where it has never been” is a rule, not a picture. And a rule can be written down as a config without a single real example in front of you.
There is open data from a real network, with labelled ground truth in it. Which means the exam can be taken honestly: train on the invented, check on the live, and not grade your own homework.
The plan followed from that, and it was simple. Show the network not one real row - not in training, not in validation, not in threshold picking. Write the entire training world as a generator config. And then turn it loose on real logs and see what it finds there.
The exam data here is the open Los Alamos set Comprehensive, Multi-Source Cyber-Security Events: 58 days, 17,684 machines, 12,425 users, 1.65 billion events. What interests me is the authentication log.
The labelled truth in it is the record of a red team exercise. And it looks like this:
749 exercise events: 104 compromised accounts, 301 destination machines and a total of 4 sources. 94% of the events come from a single foothold.
One machine. A hundred stolen logins. Three hundred places.
The exercise activity is spread across the days extremely unevenly, and from here on I refer to three sets constantly. Everything below is counted in windows: a window is one machine for one hour, everything it did in that hour. A labelled window is one that caught at least one exercise event. To keep it straight, let’s fix the names right away:
set | what it is | windows | labelled |
|---|---|---|---|
day 8 | the densest activity; debugging was done on it | 231,787 | 15 |
day 12 | second densest, also a working day | 223,987 | 12 |
held-out set | the other sixteen days, never touched | 3,600,398 | 64 |
Day 8 turned out to be easy - almost any model solves it. Day 12 is hard: on it the models finally differ. The held-out set is the only honest measure, and it will become clear why.
The rule that saved the project: the exam first
My first impulse was to sit down and write the generator config. Good thing I didn’t.
I spent half a day opening up and measuring the real data before the first line of config. And it turned out the problem was not the one I had pictured: I was thinking about password guessing, and what’s labelled is an entirely different phenomenon. Everything I would have written in advance would have gone in the bin, all of it.
The reconnaissance produced three facts that determined absolutely everything that followed.
The account name carries no useful signal
The main “victim” of the exercise - the account through which the most labelled events passed - appears 30,497 times in a clean four-hour slice. That is the busiest regular administrator, and catching them as an anomaly is pointless.
Which means the feature has to be not about whose account it is, but about where it came from. Not a property, but a relationship.
The naive counter is fooled by the shape of the network
The hypothesis “many different accounts from one machine is suspicious” breaks against the topology:
2996 2752 2686 2590 domain controllers 252 118 110 file servers 1 2 1 workstations
The foothold with its hundred accounts falls between the controllers and the workstations, and is indistinguishable from a small server. With a counter over destination machines it is exactly the same story.
The real signal is the novelty of the connection
Lateral movement produces new edges in the login graph: the pair “account - machine it has never logged in from”, or “machine - place it has never reached before”. And here is the measurement that made the reconnaissance worth doing. The share of new edges in normal activity:
share of new edges in normal activity | value |
|---|---|
median | 0.0000 |
90th percentile | 0.0000 |
99th percentile | 0.5000 |
For those new to this. A percentile is “the share of windows whose value is no larger than this”. The 90th percentile being zero means nine windows out of ten have no new edges at all; and only one window in a hundred gets a share of new edges as high as one half.
People walk well-worn routes. The background is almost perfectly clean, and that is exactly what makes the problem solvable in principle.
A world in a hundred and thirty lines
Only now can the generator be written - already knowing exactly what to produce.
The key idea: don’t imitate lateral movement, produce its definition. The phenomenon has a computable formulation - “an account logs in where it has never been, from a machine where it has never been”. Since the definition is algorithmic, it is synthesizable without losing the essence.
The heart of the config looks like this:
<pool name="Work" count="290"> <sequence name="hid"><gen type="increment" value="1"/></sequence> <sequence name="role"><gen type="text" value="0,2,3,4" percent="78.8,0.6,20,0.6"/></sequence> <sequence name="userBase"><gen type="formula" expr="hid * 30"/></sequence> <sequence name="userSpan"><gen type="number" value="1..4"/></sequence> <sequence name="fgBase"><gen type="number" value="1..290"/></sequence> <sequence name="fgSpan"><gen type="number" value="2..60"/></sequence> ... </pool>
The machines live in a lookup table, each with its own number, its own role and its own circle of accounts: the machine with number hid owns accounts from hid*30 to hid*30+span. Separately it has a compromise circle - the numbers of other machines whose accounts travel through it. The width of that circle runs from 2 to 60, and that isn’t sloppiness but a deliberate spread: the network has to learn the class, not one particular preset.
The account number for an event is then assembled by a formula:
<sequence name="User"> <gen type="formula" expr="Foreign == 1 ? ((W.fgBase + floor(hash(N, 1) * W.fgSpan)) % 290) * 30 + floor(hash(N, 18) * 2) : UserBase + floor(hash(N, 1) * UserSpan)"/> </sequence>
The formula is abridged here: the service-account and churn branches are left out, the full one is in the spoiler below. Here hash(N, salt) is a deterministic function of the row number and a salt, giving a reproducible number between zero and one. The whole set is deterministic by seed and rebuilds byte for byte.
For those new to this. A seed is the number the generator counts all its “randomness” from. The same seed gives the same world byte for byte, a different seed gives a different world under the same rules. I’ll use the term in one more sense later: a training seed is the same kind of number, but for the network’s random initial state. Different training seeds mean the same network trained several times from different starting points.
The full world is a hundred and thirty lines and 400 thousand events in 9 seconds. There are five kinds of machine in it, and each one earns its place:
workstations - one to four accounts of their own, the bulk of the network;
servers - hundreds of accounts legitimately. Without them the network learns that “many accounts equals anomaly” and drowns in false alarms;
new machines - no history, everything about them is legitimately new;
compromised machines;
machines that are new and compromised at once - otherwise a network trained not to touch legitimate novelty never learns to tell them apart.
Two more mechanisms live on the event rather than on the machine, which is why they aren’t in that list. Service accounts travel across many machines, but on each one they appear constantly, so they create no novelty: an account like that can be used by a machine of any kind. And churn - a new employee, a person moving desks, a server that was repurposed. Without it, novelty never occurs in normal activity at all, and the network draws the simple conclusion that “any new event is a compromise”.
The whole base world config, all 135 lines
This is the working file straight from the repository, with nothing cut (a copy with Russian comments lives in ru/gen/, the same lines in both), together with the comments I wrote for myself as I went. The comments matter more than the code here: almost every one of them answers “why does this class exist at all” rather than “what does this line do”. Those answers were the actual work - the syntax took minutes.
<tdc version="0.1"> <!-- Synthetic world: a network of machines, each with its own login history. Two pools, because the load in a network is distributed very unevenly: a handful of servers sees a huge stream and hundreds of accounts LEGITIMATELY, while workstations are many and each one is quiet. Roles of the work machines: 0 - workstation (its own 1..4 accounts), 2 - NEW host (no history: everything on it is legitimately new), 3 - workstation whose accounts become foreign in the second period, 4 - a machine that is new AND working under foreign accounts AT THE SAME TIME. Role 4 exists because a network trained to stay silent on legitimate novelty has to be taught separately to tell it from foreign novelty: an honest new host has fresh accounts, this one has accounts moved from other machines, that is, long known to the network. Period 0 - history, period 1 - observation. The knobs are spread deliberately wide within a class: the network must learn the class, not a preset. The makeup of the world (share of servers, rarity of new hosts, density of the stream) is calibrated against ROUGH statistics of a real network. The features of the observed phenomenon itself were not calibrated against anything. --> <env count="400000" seed="world-train-1"> <pool name="Work" count="290"> <sequence name="hid"><gen type="increment" value="1"/></sequence> <sequence name="role"><gen type="text" value="0,2,3,4" percent="78.8,0.6,20,0.6"/></sequence> <sequence name="userBase"><gen type="formula" expr="hid * 30"/></sequence> <sequence name="userSpan"><gen type="number" value="1..4"/></sequence> <sequence name="dstBase"><gen type="number" value="1..4500"/></sequence> <sequence name="dstSpan"><gen type="number" value="2..9"/></sequence> <!-- spread over hours: the wider, the rarer the events in a window; some events go past the end of observation - that is how both dense and quiet windows are born --> <sequence name="spread"><gen type="number" value="1..40"/></sequence> <!-- on some machines the makeup does not change for years, on others it flows constantly --> <sequence name="churnMil"><gen type="number" value="0..8"/></sequence> <!-- foreign activity has its own limited circle of accounts and places: the axis is spread WIDE (2..60) so that the network does not memorise one size --> <!-- The SHARE of foreign events among all events of a machine: from 2% to 100%. In a real journal a foreign event can be one in fifty ordinary ones, and a share feature drowns it. The network must see the diluted case, so the axis is spread down to the rarest values. --> <sequence name="fgRateMil"><gen type="number" value="1000..1000"/></sequence> <!-- the circle of foreign accounts: the NUMBERS OF OTHER MACHINES whose accounts are used --> <sequence name="fgBase"><gen type="number" value="1..290"/></sequence> <sequence name="fgSpan"><gen type="number" value="2..60"/></sequence> <sequence name="fgDstBase"><gen type="number" value="1..4900"/></sequence> <sequence name="fgDstSpan"><gen type="number" value="2..70"/></sequence> <sequence name="failMil"><gen type="number" value="20..700"/></sequence> <sequence name="baseMil"><gen type="number" value="1..30"/></sequence> </pool> <pool name="Serv" count="10"> <sequence name="sid"><gen type="increment" value="1"/></sequence> <sequence name="userBase"><gen type="number" value="1..7000"/></sequence> <sequence name="userSpan"><gen type="number" value="60..280"/></sequence> <sequence name="dstBase"><gen type="number" value="1..3000"/></sequence> <sequence name="dstSpan"><gen type="number" value="40..400"/></sequence> <sequence name="spread"><gen type="number" value="15..22"/></sequence> <sequence name="baseMil"><gen type="number" value="1..30"/></sequence> </pool> <sequence name="W"><gen type="pool" value="Work"/></sequence> <sequence name="S"><gen type="pool" value="Serv"/></sequence> <sequence name="N"><gen type="increment" value="1"/></sequence> <!-- a third of the stream goes to the servers: they are few, the stream to each is huge --> <sequence name="IsServ"><gen type="formula" expr="hash(N, 10) < 0.35 ? 1 : 0"/></sequence> <!-- Service and machine accounts: in a real network they log in everywhere, and the reach of such an account is thousands of machines. Without them the world consists of nothing but accounts "tied to a workplace", which never happens in life. --> <!-- Every machine uses the same service accounts (so they create no novelty), and every such account serves several machines - hence the wide reach. A small part (tier 9500+) goes everywhere. --> <sequence name="IsSvc"><gen type="formula" expr="hash(N, 12) < 0.30 ? 1 : 0"/></sequence> <!-- machine name: servers and work machines in different number ranges --> <sequence name="Host"><gen type="formula" expr="IsServ == 1 ? 9000 + S.sid : W.hid"/></sequence> <sequence name="Role"><gen type="formula" expr="IsServ == 1 ? 1 : W.role"/></sequence> <sequence name="Spread"><gen type="formula" expr="IsServ == 1 ? S.spread : W.spread"/></sequence> <!-- a new host lives only in the second period --> <sequence name="Period"> <gen type="formula" expr="Role == 2 || Role == 4 ? 1 : (hash(N, 6) < 0.60 ? 0 : 1)"/> </sequence> <sequence name="Time"> <gen type="formula" expr="Period == 0 ? floor(hash(N, 5) * 28800) : 28800 + floor(hash(N, 3) * Spread) * 3600 + floor(hash(N, 4) * 3600)"/> </sequence> <!-- on this machine the accounts are foreign in the second period --> <sequence name="Foreign"> <gen type="formula" expr="(Role == 3 || Role == 4) && Period == 1 && hash(N, 11) < W.fgRateMil / 1000 ? 1 : 0"/> </sequence> <sequence name="UserBase"><gen type="formula" expr="IsServ == 1 ? S.userBase : W.userBase"/></sequence> <sequence name="UserSpan"><gen type="formula" expr="IsServ == 1 ? S.userSpan : W.userSpan"/></sequence> <sequence name="DstBase"><gen type="formula" expr="IsServ == 1 ? S.dstBase : W.dstBase"/></sequence> <sequence name="DstSpan"><gen type="formula" expr="IsServ == 1 ? S.dstSpan : W.dstSpan"/></sequence> <!-- Legitimate change comes in two kinds, and they look different: a new employee (the account was not in the network at all, range 20000+) and an existing person moving to another machine. Without both the network will not learn to tell legitimate novelty from foreign credentials. --> <sequence name="Churn"> <gen type="formula" expr="Foreign == 0 && IsSvc == 0 && hash(N, 14) < W.churnMil / 10000 ? 1 : 0"/> </sequence> <sequence name="User"> <gen type="formula" expr="Foreign == 1 ? ((W.fgBase + floor(hash(N, 1) * W.fgSpan)) % 290) * 30 + floor(hash(N, 18) * 2) : (IsSvc == 1 ? (hash(N, 13) < 0.25 ? 9500 + floor(hash(N, 1) * 20) : 9000 + (Host * 7 + floor(hash(N, 1) * 6)) % 400) : (Churn == 1 ? (hash(N, 17) < 0.5 ? 20000 + floor(hash(N, 15) * 900) : (floor(hash(N, 15) * 290)) * 30 + floor(hash(N, 19) * 4)) : UserBase + floor(hash(N, 1) * UserSpan)))"/> </sequence> <sequence name="Dst"> <gen type="formula" expr="Foreign == 1 ? 1 + (W.fgDstBase + floor(hash(N, 2) * W.fgDstSpan)) % 4900 : (Churn == 1 ? DstBase + DstSpan + floor(hash(N, 16) * 50) : DstBase + floor(hash(N, 2) * DstSpan))"/> </sequence> <sequence name="FailCut"> <gen type="formula" expr="Foreign == 1 ? W.failMil / 1000 : (IsServ == 1 ? S.baseMil : W.baseMil) / 1000"/> </sequence> <sequence name="Ok"> <gen if="hash(N, 8) < FailCut" type="text" value="Fail"/> <gen type="text" value="Success"/> </sequence> </env> <block> <line><data>${{Time}},U${{User}},C${{Host}},D${{Dst}},${{Ok}},${{Foreign}}</data></line> </block> </tdc>
What to look at, if you’re skimming. Two pools instead of one - because load in a network is distributed wildly unevenly, and a handful of servers legitimately see hundreds of accounts. Role 4 - a machine that is new and running under foreign accounts at the same time; it exists precisely so that a network trained to stay quiet on legitimate novelty is separately taught to tell honest novelty from stolen. The knobs opened wide - 2..60, 1..40, 20..700: the network has to learn the class, not one particular set of numbers. And Churn - legitimate turnover of two kinds, without which novelty never occurs in normal activity and the network concludes that “any new event is a compromise”.
The output line at the very bottom is the entire log format: time, account, machine, destination, success, label. The last field takes no part in training; it is there for the feature builder (the repository calls it the measurer), so it knows the truth.
Two formulas in the file are long and run off to the right - User and Dst. There’s no need to scroll through them; they are built the same way, as a ladder of nested conditions: “if foreign, take from the compromise circle; else if a service account, from the service range; else if churn, either fresh or moved in; else its own”. One line instead of four branches on the outside; it reads badly, but it is fixed in one place.
The mistake that cost more than every setting
For a long time my “compromised” accounts were non-existent: the generator drew a number from a range, and no such account existed anywhere in the network. Comparing distributions against reality exposed it instantly:
in labelled windows | reality | synthetic, before the fix |
|---|---|---|
account known to the network, but on this machine for the first time | 26 | 2 |
account did not exist in the network at all | 0 | 11 |
In a real network, lateral movement runs on existing accounts - that is the whole point of stealing them. What I was producing wasn’t theft but invention, and the network was diligently learning to catch an artefact that never occurs in live data.
That one fix reduced false alarms fourfold at the same recall. More than every network setting put together.
Synthetic data has to be right in meaning, not similar in statistics.
The network
The first network is fully connected, the most ordinary kind: eighteen features in, eight layers of twelve neurons, 1333 weights. Training takes seconds on a laptop, no GPU. In the table below it appears as “deep [12]x8”; a second one, recurrent, shows up later in the article - its turn comes after the capacity measurement.
The unit of observation is the window, “one machine for one hour”. The features come in three sorts: volume and variety (how many events, accounts, destinations), shares of novelty, and the context of the machine (is it a server or a quiet workstation). Two features are worth naming separately, because they are the ones that encode the meaning of stolen credentials:
moved accounts - the number of events where the account is known to the network but appears on this machine for the first time. The signature of theft: the account exists and works somewhere else;
fresh accounts - the number of events where the account did not exist in the network at all. The signature of a new employee.
Without that distinction, a new person legitimately starting work is indistinguishable from a compromise.
The counter features are taken under a logarithm, and that isn’t cosmetic. The synthetic world has 290 machines, the real one has 17,684. A linear feature would push activations far outside the training range, where a network behaves unpredictably; a logarithm turns a difference of scale into a shift.
The same code computes the features for the synthetic data and for reality. That isn’t a declaration: day 8 was recomputed by two independent implementations of the feature builder and compared line by line, and 231,787 lines matched exactly.
On synthetic data with a different seed - that is, on another instance of the same world - the network achieves an AUC of 0.99996. That means exactly one thing, and I wrote the conclusion down before the exam: the network has learned my generator thoroughly. And nothing else.
For those new to this. AUC is the area under the ROC curve. The easiest way to read it: take one labelled window and one ordinary window at random; AUC is the probability that the model ranks the labelled one higher. 1.0 is a perfect ordering, 0.5 is a coin. There is a whole section later on why AUC lies on this problem, but for now let it be “the share of correctly ordered pairs”.
How many weights are actually needed
I picked the architecture blindly at first - the classic mistake. Then I measured it: thirty-one configurations, three training seeds each (three trainings from different starting points, so the spread is visible), ninety-three trained networks in 170 seconds on a CPU. The characteristic rows:
architecture | weights | easy day | hard day |
|---|---|---|---|
flat [4] | 81 | 0.99868 | 0.94208 |
flat [8] | 161 | 0.99998 | 0.94151 |
flat [256] | 5121 | 0.99997 | 0.95646 |
two steps [128,64] | 10753 | 0.99995 | 0.96318 |
deep [12]x8 | 1333 | 0.99997 | 0.97518 |
deep [12]x12 | 1957 | 0.99940 | 0.94923 |
Three conclusions from that.
Performance on the easy day is saturated: from 161 weights to 10,753 everything sits in the 0.99995-0.99998 range; capacity simply isn’t the constraint there.
Depth beats width: 1333 weights do better than 10,753.
And too deep is bad too: twelve layers without residual connections drop to 0.949 and spoil even the easy day.
Later I ran nine sequence architectures, the LSTM among them, from dilated convolutions to external memory in the spirit of a neural Turing machine. Half of those names I learned that same evening, so treat this as curiosity rather than a survey. The best turned out to be a plain recurrent network of 4249 parameters - an LSTM, a network that reads one machine’s hours in order and remembers what came before. On the hard day it gives 0.990 against 0.975 for the best fully connected one, and from here on in the article “the network” means this one. The worst turned out to be the transformer: self-attention is greedy for data and wants to link distant elements, and our sequences are 24 steps long with nothing in them to link. The full tables are in ARCHITECTURES in the repository.
The first result on real data
The day with the densest exercise activity: 231,787 windows, 15 of them labelled.
measure | value |
|---|---|
labelled windows found | 15 of 15 |
false alarms at that point | 11 |
positions of the labelled windows in the overall list | all within the first 32 |
And here is the cost of full recall - how many false alarms you have to look through to reach the last real window:
method | false alarms |
|---|---|
counter of “accounts from a machine” | 175,904 |
counter of “destinations” | 206,651 |
a four-threshold rule fitted against the answers | never gets there |
the network trained on synthetic data | 10 |
Three numbers about the same day look contradictory, so let me explain: they are three different measures. 11 is false alarms at a fixed threshold of 0.99, i.e. if you raise an alarm on everything the model scored above that value. 32 is the worst position of a labelled window in a list of 231,787: even the unluckiest one landed in the first three dozen. 10 is the cost of full recall, i.e. how many false alarms stand between the top of the list and the last real window. Different measures, same data; don’t mix them up.
The row with the thresholds deserves a word of its own, because it explains why a held-out set is needed at all. I honestly tried to fit a rule while peeking at the answers: four thresholds, tuned on the labelled day. It gives 13 windows out of 15 at zero false alarms - and never reaches full recall, because two quiet windows with three and six events don’t pass the volume thresholds. Optimising on the test produced a locally beautiful and globally useless model.
The numbers are good, and it was too early to enjoy them: they were taken on the very day I had tuned everything on. The real check was still ahead.
The held-out set said no four times
I set aside everything except two working days: sixteen days, 3,600,398 windows, 64 labelled. Didn’t look, didn’t touch.
And then I started improving things - and each time got a gain on day 12:
improvement | day 12 | held-out set |
|---|---|---|
a new machine class, “administrator on rounds” (legitimately logs in to many machines) | better | worse |
an ensemble of three networks trained on one world | better | worse |
a stack over weak learners (a trainable judge over several deliberately weak networks) | a lottery | a lottery |
a rich world: 8 million events, attack stages | +1.26 points | no change |
For those new to this. The “rich world” was an attempt to make the attack a process with four stages: quiet establishment, credential harvesting, spreading, and reaching the servers. The early stages are single events, and on them the network learned to fire on any quiet window: on the held-out set the cost of search for the rich world came out a hundred times worse. From here on that phrase appears as the name of a mistake, and “clean world” as the name of a world without those stages.
Four times in a row. In the last case the gain exceeded the spread across training seeds by a factor of three and a half - and still didn’t transfer.
It took me a while to understand why. I was measuring the spread across training seeds and assumed that was enough. But there is a second source of noise, invisible to reseeding: which twelve windows happen to be labelled on day 12. That is the spread over the sample of labelled truth, and it cannot be estimated by reseeding the model - it needs another day.
Twelve positive examples do not constitute a measure at any spread of seeds.
Half of my evening comparisons were beyond the resolving power of the data. I was comparing noise with noise and drawing conclusions. That is exactly what a held-out set is for: it said “no” four times where day 12 said “yes”.
Skipping ahead: one row in that table I did eventually take back. Ensembles do transfer - just not the ensemble I tried then. There’s a section about that later, and the conclusion there comes out the opposite way.
Why AUC lies on a problem like this
And here I nearly picked the wrong model.
When I got to gradient boosting - an ensemble of decision trees, XGBoost, the standard strong method for tabular data - it beat everything else on AUC: 0.918 against 0.909 for the recurrent network and 0.862 for the fully connected one. Case closed, you’d think. And that’s where I almost drew the wrong conclusion.
AUC is not what an analyst needs. They need a clean top of the list: they open the first twenty windows and see how many of them are real. The right measure is the cost of search: how many false alarms you have to look through on the way to the Nth window found. Held-out set, 3.6 million windows:
windows caught | fully connected | boosting | recurrent |
|---|---|---|---|
1 | 17 | 7 | 3 |
8 | 55 | 8,053 | 102 |
16 | 245 | 23,781 | 188 |
48 | 185,613 | 304,954 | 137,014 |
Boosting is better only in the tail - on the last windows, which nobody will ever open. In the working region it is a hundred times worse.
The mechanism is simple. There are 64 positive examples and 3.6 million negative ones. AUC is the share of pairs where the positive ranks above the negative, averaged over all pairs. A model earns a high score by neatly sorting the hopeless tail while the top of the list is filthy. It’s like praising a search engine for getting results fifty thousand through sixty thousand in the right order.
The most expressive case is the echo state network. It’s an amusing construction: a reservoir of five thousand random weights inside that are never trained, and only a linear readout of 65 parameters learns anything.
At a perfectly respectable AUC of 0.969 on day 12 it requires 3406 false alarms before the first window found - against one for the recurrent network. A factor of three and a half thousand, for a difference of two hundredths in AUC. And it isn’t a curiosity: across a table of five architectures the order by AUC doesn’t match the order by cost of search at all.
If you are building a search for rare events and measuring quality by AUC - you are almost certainly measuring the wrong thing.
Finding one: the baseline scissors
The feature is defined as novelty relative to the machine’s own history: the list of accounts and places already seen on it. That accumulated history I’ll call the baseline from here on. And this is where a trap turns up that any detector building a behaviour profile falls into.
The baseline is poisoned by the very thing it catches
By day 12 the foothold “knows” 84 accounts. Of those, 49 are the very compromised ones that had been travelling through it earlier.
58.3% of the machine’s “normal history” is the observed intrusion itself.
The mechanism: history accumulates by the fact of observation, so any sufficiently long-running activity enters the norm and stops being new. The signal weakens precisely because the attack continues. The median novelty of labelled windows fell from 1.00 on day 8 to 0.57 on day 12.
The cure is obvious: freeze the baseline at an early period. It helps immediately - 10 windows found out of 12 instead of 5, and every labelled window rises into the first twenty-seven rows of the list instead of scattering out to eight hundred.
But a frozen baseline goes stale
day | 9 | 15 | 29 |
|---|---|---|---|
machines unknown to the baseline | 1.1% | 3.3% | 6.1% |
The drift is about 0.25% of machines a day. By the end of the observation every sixteenth machine is a stranger to the baseline and therefore looks anomalous: 39% of all false alarms on the held-out set come from machines exactly like that. On day 8 there were none.
Refresh the baseline often and it gets poisoned by what you’re observing. Freeze it and it goes stale at a quarter of a percent of machines a day. The operating point: a baseline no younger than a week and no older than two or three weeks.
This is a property not of my model but of any method that builds a profile from observed history without an external check on credentials.
Finding two: here I was wrong
Here I drew a conclusion that pleased me at first: it tidily explained everything at once. And then turned out to be wrong.
Let’s call a coincidence a normal window that carries the full signature of a compromise: high novelty and a high share of failures at the same time. In reality such windows exist - 51 out of 231,772, two hundredths of a percent. In my worlds there were none. Not one, neither in the base world nor in the rich eight-million-event one. And yet each feature on its own was reproduced perfectly well:
high novelty | many failures | both at once | |
|---|---|---|---|
reality | 0.644% | 0.669% | 0.022% |
my rich world | 0.887% | 6.933% | 0.000% |
From that I drew a conclusion that seemed elegant and deep: a generator produces regularities, and reality consists of regularities plus coincidences. A coincidence is by definition the absence of a mechanism, an unplanned intersection of independent causes. To get them you would have to model every mechanism of the real world together with its links - that is, model reality in full. Elegant, hopeless and, as it turned out, wrong.
The control run that cost one launch
I set out to produce coincidences on purpose: I built a world where new accounts and failures converge on a machine legitimately. And alongside it a control: the same worlds with that mechanism switched down to zero.
The control worlds produced coincidences just as well.
Which means the mechanism had nothing to do with it. The coincidences appeared because the world had got smaller - a million and a half events across the same machines - and the windows thinned out. In sparse windows the shares start to swing: in a window of two events the “share of new accounts” takes the values 0, 0.5 and 1, and half of those look like the signature of an attack. That is exactly how coincidences are built in reality too.
Without the control I would have credited the effect to the mechanism and written something untrue in this article. The control cost one launch.
The sparse world
From there it’s clear what to build: a world where events are smeared over time the way they are in a live network. I take the no-stage world, raise it to four million events and add quiet machines. Eighteen per cent of machines are “quiet”, their events stretched over 80-260 hours. For the first time, the shape of the windows sat down on reality:
events in a window | 1st percentile | median | 90th percentile |
|---|---|---|---|
LANL reality | 2 | 22 | 49 |
sparse world | 1 | 19 | 57 |
previous (base) world | 5 | 16 | 53 |
The coincidences appeared by themselves, and of the right size, too: real coincidences have a median of two events, mine have four, and previously it was twenty-four. There is still no “coincidence” mechanism anywhere in the config. There is sparseness, and coincidences follow from it.
Here is everything that produces the sparseness, verbatim from world-sparse: two sequences in the machine lookup table and one time formula.
<!-- 18% of machines are "quiet": their events smear over 80-260 hours instead of 1-40 --> <sequence name="isQuiet"><gen type="text" value="1,0" percent="18,82"/></sequence> <sequence name="spread"> <gen if="isQuiet == 1" type="number" value="80..260"/> <gen type="number" value="1..40"/> </sequence> <!-- hour of the event: the machine's spread sets how many hours its events land on --> <sequence name="Time"> <gen type="formula" expr="... 28800 + floor(hash(N, 3) * Spread) * 3600 + floor(hash(N, 4) * 3600)"/> </sequence>
Three stock features of TDCV2 are at work here. percent sets a share exactly, not by probability: there will be precisely 18% quiet machines. The if attribute picks the generator by a condition: a quiet machine gets a spread of 80-260 hours, everything else 1-40. And the time formula lays the machine’s events out over Spread hours. A quiet machine has roughly as many events as any other, they just land one or two per hour - and the “share of new accounts” in such a window starts taking the values 0, 1/2 and 1, half of which look like the signature of an attack. There is no coincidence mechanism here at all: there are quiet machines, and coincidences are their arithmetic consequence.
A coincidence is not a mechanism you have to model. It is what you get for free, if the shape of the data is right.
Six worlds, six networks
The second conclusion I had to take back was the one about ensembles.
I measured it honestly and wrote it down: combining networks is pointless, because the spread of their predictions is 0.0002, they make mistakes in the same places, and there is nothing to average. That was true - but true of networks trained on one world.
Whereas if you train six networks on six different worlds and average their ranks - not their scores, their positions: each network arranges all 3.6 million windows into its own list, and a window is assigned its mean position over the six lists - this is what you get. The worlds here are clean: without the attack stages that taught the network in the rich world to fire on quiet windows, and also without sparseness, which is added at the next step. The table shows how many false alarms an analyst looks through on the way to the Nth attack found, on the held-out set, 3.6 million windows:
what exactly is being checked | to the 1st | to the 8th | to the 16th | to the 24th |
|---|---|---|---|---|
base world, one network - where we started | 3 | 102 | 188 | 412 |
base world, six seeds, ranks (what averaging alone gives) | 11 | 80 | 117 | 504 |
one clean world, one network (what the clean world alone gives) | 0 | 2 | 30 | 222 |
six different clean worlds, six networks, ranks | 0 | 3 | 7 | 19 |
the same six worlds, merged into one set, one network | 814 | 6,992 | 26,297 | 59,973 |
A hundred and eighty-eight false alarms turned into seven. And neither factor gives that on its own: averaging by itself is 117, the clean world by itself is 30, and together they are 7.
And now the last row, the one the control was set up for. The same six worlds, merged into one training set for one network, give 26,297 false alarms instead of 188. A hundred and forty times worse than the base.
Variety merged into one set becomes contradiction. Split across separate models, it becomes strength.
And it comes with a gauge you can use before the exam. The spread of opinion between networks - the mean deviation of scores across windows:
networks trained | spread of opinion | gain |
|---|---|---|
on one set | 0.0002 | none |
on one world, different seeds | 0.004 | 1.6 times |
on different worlds | 0.014 | twenty-seven times |
What’s convenient about this is that the spread is taken from the networks themselves, without touching the held-out set. If the networks agree, there is no point combining them, and you can spare the exam.
The micro-loop: reality names a machine, the config gets a line
And here is where the thing that makes a generator worth writing at all began.
I never inspected the seven. But the ensemble of six sparse worlds gave the best AUC of the project, 0.944, and with it six false alarms above the first attack. Those were the first ones I actually looked at, to see what those windows were. Five out of six turned out to be one phenomenon: failure storms. Machines with 390-3391 events an hour and 90-99% unsuccessful logins. By the labels they are normal; in substance they are a broken service or somebody’s brute force. For an analyst, a find; for the metric, an error.
I checked how many of those are in the data: in LANL, 3 to 13 a day, 249 across the whole held-out set. And in my training worlds - zero. The config did have a role for “a service with a broken password”, but it produced 30-80% failures at an ordinary event volume. Not that kind of storm.
And here an important thing came to light. By that point I had three families of worlds - six clean, six sparse and, skipping ahead, six with storms; a family is one config replicated with six seeds and different shares of coincidences. The upshot: all eighteen networks of the three families gave the storms the maximum rank. The ensemble had nothing to correct.
An ensemble cures disagreement, not a shared delusion. A shared delusion is born from a shared hole in the worlds, and only a world can close it.
What followed was the loop that is the real work with a generator: I look at the top false alarms, find a specific machine, work out what my world doesn’t know about it, and add one or two lines to the config. The first row of the table is the base world, the second the sparse one the loop started from, then three rounds in a row. AUC here is on the hard day 12, and “false alarms to the 6th” is how many false alarms stand before the sixth of the twelve attacks labelled there:
what was added to the world | found through which machine | AUC, hard day | false alarms to the 6th |
|---|---|---|---|
base world, where we started | - | 0.990 | - |
sparse world: a tail of quiet machines | the shape of LANL windows | 0.99327 | 0 |
failure storms: 1.5% of machines, 90-100% failures, 1-4 hours | the top of the held-out set | 0.99379 | 3 |
a person moving to another machine | C3699 held 22 of the top 40 | 0.99636 | 0 |
new machines 0.6% -> 3% | the first hours of new machines | 0.99743 | 0 |
The relocation is worth telling separately, because I got it wrong the first time - and the mistake turned out to be a meaningful one.
On the hard day one machine, C3699, held twenty-two rows out of forty at the top of the false alarms. The diagnosis: a person had moved onto it from another machine. I added a “relocation” role: from some hour onwards, half of the machine’s events come from a stranger user. It did nothing at all.
I compared the features of the real C3699 against my synthetic one - and the difference came down to one thing: in the real move, 27 events out of 27 produce a new triple of “account, source, destination”, and in mine, zero. The reason is simple, and I should have used my head: a person who moves to another machine keeps going to their own usual servers, and for the new machine all of those are new. I had been making them walk the destinations of the machine’s owner.
A one-line fix. C3699 disappeared from the top forty entirely. After the third round, with the new machines, it came back in part - eight windows, starting from fifteenth place - but it no longer held the top.
What those three fixes look like in the config
So that “two lines” doesn’t sound like a figure of speech - here they are verbatim, out of the actual files.
Storms (world-storm): one role and three overrides in the places where an ordinary machine had values of its own (in the file they sit in three different sequences; here they are gathered together).
<!-- STORM. A service with a broken password hammering away without stopping: one or two users, a few destinations, 90-100% failures, hundreds of events an hour. In LANL there are 3-13 such windows a day, labelled normal. Earlier worlds contained none, and the networks took a storm for an attack. --> <sequence name="isStorm"><gen type="text" value="1,0" percent="1.5,98.5"/></sequence> <!-- and three overrides where an ordinary machine has its own values --> <gen if="isStorm == 1" type="number" value="1..2"/> <!-- accounts --> <gen if="isStorm == 1" type="number" value="1..4"/> <!-- hours: the whole storm in 1-4 --> <gen if="isStorm == 1" type="number" value="900..1000"/> <!-- failures, out of a thousand -->
The relocation (world-move) came out longer, because the newcomer has to be not merely let in but sent walking to their own places:
<!-- MOVE and NEWCOMER. From some hour onwards half of the machine's events come from a user it has never seen, and it stays that way to the end of the day, with no failures. A move is a user of another machine (known in the network), a newcomer is a fresh account (known nowhere). The arrival goes to THEIR OWN destinations, new both to the machine and to them: on C3699 every triple is new - that was the difference from version one. --> <sequence name="isMove"><gen type="text" value="1,0" percent="2,98"/></sequence> <sequence name="isFresh"><gen type="text" value="1,0" percent="1,99"/></sequence> <sequence name="moveHour"><gen type="number" value="8..20"/></sequence> <sequence name="Newcomer"> <gen type="formula" expr="Foreign == 0 && IsSvc == 0 && IsServ == 0 && Period == 1 && (W.isMove == 1 || W.isFresh == 1) && floor(Time / 3600) >= W.moveHour && hash(N, 21) < 0.5 ? 1 : 0"/> </sequence> <sequence name="NewUser"> <gen type="formula" expr="W.isMove == 1 ? ((W.hid * 37 + 11) % 2000) * 30 : 90000 + W.hid"/> </sequence>
The difference between the first version of the move and the second is not in those lines - they are identical in both. It is in the destination formula, which isn’t above. In the first version the newcomer walked the destinations of the machine’s owner, and the “new triple” feature never fired: the destinations were familiar to the machine. In the second they have a range of their own, new to the machine as well - one branch at the start of the Dst formula:
expr="Newcomer == 1 ? 5000 + (W.hid * 13) % 900 + floor(hash(N, 2) * 4) : (...)"
That is precisely what separated the real C3699 from my invented one.
The third fix, “new machines at 3%”, is one number: the share of the “new machine” role in the lookup table changes from 0.6 to 3. The base world was 135 lines, the final one is 210. Everything in between was mined by rounds like these.
In all of this the generator owes exactly one thing: to stay out of the way. It knows nothing about networks, storms and relocations, and it shouldn’t - its job is that describing them takes two minutes once you’ve worked out what was missing. The understanding is human work, and it isn’t going anywhere.
How much of this is honest
This needs its own bookkeeping, otherwise it turns into cheating.
A held-out set is valuable exactly up to the first look at it. The seven above was obtained before I first opened the top of the false alarms - that is an honest transfer. The storms I found having already looked there, and everything built on top inherited that knowledge. The difference is fundamental and the two must not be mixed:
result | AUC | false alarms to the 16th | how it was obtained |
|---|---|---|---|
one network, base world - the first half of the article | 0.909 | 188 | honestly |
six clean worlds, six networks | 0.938 | 7 | honestly |
six sparse worlds | 0.944 | 10 | honestly |
eighteen networks over three families: clean, with storms, final | 0.932 | 1 | after diagnosing errors |
The honest number to compare against the one we started with is seven instead of a hundred and eighty-eight. The one is real too, but it already involves peeking, and I quote it only with that note attached.
And one more line in the same ledger: I consulted the held-out set at least twenty-three times over the project. Each consultation reduces its value a little. A genuinely clean check would be fresh Los Alamos days we never touched at all, used once. It hasn’t been done, and saying so is more honest than staying quiet.
It also turned out how many networks are worth keeping. The saturation curve: one network gives 43 false alarms before the sixteenth attack, two give 10, four give 4, six give 3, twelve give 1, and past that nothing changes. Saturation is at roughly twelve networks and three families of worlds with different phenomena. A fourth family added nothing.
For scale: what the specialists get
This isn’t a league table, it’s a coordinate system. I couldn’t tell myself whether 0.909 was a lot or a little until I looked at what people who do this professionally get.
solution | AUC | what it was trained on |
|---|---|---|
LMTracker | ~0.95 | on labelled Los Alamos data |
six sparse worlds, six networks | 0.944 | synthetic only |
six clean worlds, six networks | 0.938 | synthetic only |
UGEA-LMD | 0.9254 | on labelled Los Alamos data |
my boosting | 0.918 | synthetic only |
one network, base world | 0.909 | synthetic only |
fully connected, 1333 parameters | 0.862 | synthetic only |
counter of “accounts from a machine” | 0.618 | - |
counter of “destinations” | 0.535 | - |
A caveat about the top row: some papers publish not the value itself but only their first place in their own comparison, so I take the 0.92-0.95 range from the ones that give a number. And comparing AUC across different evaluation protocols head-on is incorrect - the splits differ - so this is a reference scale, not a comparison.
Three honest conclusions.
First. Threshold rules lag far behind. On the held-out set they work barely better than a coin. By cost of search the difference is three orders of magnitude: to find sixteen windows out of 64, the counter needs about 161 thousand false alarms, and the recurrent network needs 188. That one I did not expect at all.
Second. The synthetic data did make it into the range of the research work, and that is nicer than I was counting on. Above UGEA-LMD, below LMTracker, six thousandths short of the upper bound. Both numbers are honest transfers, with no peeking at the held-out set. There isn’t much to celebrate here, for the reason in the previous paragraph: the protocols differ and a head-on comparison is incorrect. But previously I was below everyone who publishes an AUC, and now I’m in that range.
I include the boosting row deliberately, even though I explained above that it’s unfit: by AUC it formally holds its place in the same row, and by cost of search it is hundreds of times worse. The best illustration of why comparing methods by AUC alone is meaningless - including comparing against this very table. And nobody else’s work publishes a cost of search, by the way, so on the measure that matters most to an analyst there are simply no reference points.
Third, and the main one: our problem statements differ. Everything listed needs labelled examples of the attack in your network. And an ordinary organisation doesn’t have them: to get labels you first have to be compromised, discover it and label it. Commercial systems get around this with unsupervised profiling, but they require weeks of observation.
I found no work in which a lateral-movement detector was trained on synthetic data only and checked against LANL. But I’m not going to claim “I’m the first”: absence from search results is not proof.
Can this be deployed
As a finished product, no. As a first-pass filter, yes.
What works: four thousand parameters, features computed in a single streaming pass without loading the data into memory, without needing a single labelled incident from your network, and what comes out is not a verdict but a queue - “here are twenty windows, look at these first”.
Three conditions, each of them measured.
At least a week of network history. On the first day, where a machine has 464 events of history against 3500 on day 8, the method falls apart.
Refresh the baseline every two or three weeks - because of the scissors above.
This is a filter, not an auto-block. To find sixteen real windows an analyst looks through twenty-three windows out of 3.6 million - sixteen real and seven false. Acceptable for a review queue; still not acceptable for automatic response: half the labelled windows lie in the tail, in positions from three thousand to two million, and by the features of the hour they are indistinguishable from noise - one event, one new user, not a single failure. They can only be pulled out with other features, links between machines, and none of that is in this work.
A word about the anti-pattern. Allow-lists of the form “the machine passed a check, so we exclude it from analysis” are an accelerated version of poisoning the baseline. Remember the 58.3%.
What didn’t work
The negative results outnumbered the positive ones, and each came with an understood mechanism - which is worth more than the result itself.
Training on diluted windows. It sounded logical: teach the network to see one foreign event among thirty-seven. I got 1428 false alarms instead of 27. The mechanism was worked out: dilution teaches that any event with triple novelty means a compromise, and there are enough of those in 1330 normal windows.
Accumulating suspicion across neighbouring hours. An intrusion lasts hours, so it seemed one should accumulate. The result ranged from neutral to destructive, AUC 0.504. Diagnosis: the same self-poisoning mechanism repeated at the scale of hours - from the second hour onwards the activity enters the machine’s own baseline.
Ensembles of networks from one world. A stack over the committee loses to plain averaging in every composition, and the cause is that same spread of 0.0002. Weak learners did give the stack some disagreement to work with, and it beat averaging for the first time, but the absolute level stayed below a single network. The qualifier “from one world” is essential here: as soon as the worlds became different, the ensemble started working - that was above, and it is exactly the case where a negative result turned out to be only half right.
And my own mistakes. Twice, twelve features out of sixteen made it to the model’s input, the missing ones arrived empty, and the network output a constant. I caught it by its characteristic signature - an AUC of exactly 0.50000. Another time the threshold search turned out to be quadratic and simply never finished on 3.6 million values. And once I broke my own rule of “check the synthetic data before training” - the result was cut in half.
Where everything is, and how to check it
Everything above exists as real files, not as a retelling. It is all in the tdc-guard repository, with a step-by-step guide in README listing the expected numbers at every step (there is a README in Russian right beside it). The layout, so that you can look at it or run it yourself. In the repository’s file names the held-out set is called sealed: exam/sealed.mjs, results/sealed-windows.csv.
The worlds
The configs live in the repository in two copies: in gen/ with English comments and in the Russian copies with Russian ones. The lines of code are the same in both.
file | what’s in it |
|---|---|
the base world, 135 lines - the one quoted in full in the spoiler above | |
the clean world: the rich world without attack stages, family ns - six of these gave the honest seven | |
sparse: the quiet machines the coincidences were born from | |
plus failure storms | |
plus a person moving to another machine | |
the final one, 210 lines | |
the world with an explicit coincidence mechanism, and the control that refuted it | |
the |
The feature builder (the measurer) and the exam
file | what’s in it |
|---|---|
the single feature builder, shared by the synthetic data and reality | |
the loader that fails if not all the features arrived - added after my two AUCs of exactly 0.50000 | |
the |
Networks and experiments
file | what’s in it |
|---|---|
the model, feature encoding, AUC by ranks and the cost curve in one pass | |
the sweep over 31 fully connected architectures | |
the recurrent ones and the nine exotic ones | |
six networks on six worlds against the held-out set - the main result | |
the control: six seeds of one world, for the row in the attribution ladder | |
the top forty false alarms with their features - the instrument of the micro-loop | |
the saturation curve: how many networks are worth keeping |
The record
DIARY is the entries by date, every run in order, including the failed ones and the traps. FACTS is the set of numbers with notes on what was obtained honestly and what came after diagnosing errors. The second document is more useful for verification, the first is more honest: you can see how many times I was wrong along the way. The Russian versions sit next to them under a .ru suffix (DIARY in Russian, FACTS in Russian, README in Russian).
The data is open and freely downloadable from the Los Alamos portal. The worlds are deterministic given a seed and rebuild byte for byte, so the numbers above reproduce for you without asking me.
What this adds up to
You can get a working detector without a single labelled incident in your network. Twenty-five thousand parameters, six small networks, training measured in seconds on a laptop. Seven false alarms on the way to the sixteenth attack out of three and a half million windows - against a hundred and sixty-one thousand for the threshold counter, which works directly on your real data at that. The whole thing took two days: one for reconnaissance of the data, the generator and the first network, the second for experiments with worlds, ensembles and the micro-loop.
You can’t - and here I had to take my words back twice.
First I wrote that I had hit a fundamental limit: a generator produces regularities, reality consists of regularities plus coincidences, and coincidences cannot be synthesized. It sounded solid. It turned out coincidences appear by themselves if the windows have the right shape, and all that was missing was sparseness.
Then I wrote that ensembles are useless on this problem, because the networks make mistakes in the same places. Also true - right up until the networks stop learning from one world.
Both times the ceiling turned out to be not the method but something I hadn’t thought through about my own world. The real boundaries I am prepared to defend lie elsewhere: half the labelled windows are indistinguishable from noise by the features of the hour, and there is nothing here to pull them out with; and the held-out set I touched at least twenty-three times, so by rights it is long overdue for replacement with fresh days.
Five rules I took away
Synthetic data has to be right in meaning, not similar in statistics. The account must be real and stolen; an invented one breaks everything, and no network settings will fix that.
The world and the architecture are one decision, not two. Five times over the project, a change of world changed which architecture won. You cannot compare networks on one world and carry the conclusion to another.
AUC and the cost of search diverge. Not twice and not by accident, but systematically: the ordering of models by one measure doesn’t match the ordering by the other.
An ensemble cures disagreement, not a shared delusion. If every network is wrong in the same way there is nothing to combine - only a world closes that hole.
Variety merged into one set becomes contradiction; split across models, it becomes strength. The same six worlds: one network per world gives seven false alarms, merged into one set they give twenty-six thousand.
The conclusion about the generator itself is boring, and that’s the best thing that can be said about it. It survived more than a dozen versions of the world and never once became the bottleneck. The bottleneck was somewhere else the whole time: working out what exactly to produce. Reconnaissance of the data, diagnosing one specific machine at the top of the false alarms, guessing what my world doesn’t know about it - and only then one line of config.
In problems built on synthetic data, the bottleneck is never the model.
What I liked most in this work is the loop from the last sections but one. The detector’s error names a specific machine; you look at it and understand which phenomenon is missing from your invented world; you add two lines; the error is gone. Note that the interesting part is entirely human - and that is, perhaps, good news for anyone worried that data is about to start generating itself.
The tool is called TDCV2 and it is MIT licensed.
The Los Alamos data - open, freely downloadable
If you work in this field professionally and can see where I went wrong, write in the comments - I’m genuinely interested. Especially about the baseline scissors: I described them as a measured property, but I don’t know how commercial systems live with this, and I won’t pretend otherwise.