The original plan was simple. I prepared one Ubuntu Server VM under KVM, installed the packages needed for the lab, enabled SSH, configured cloud‑init, created a WireGuard interface for the private test network and installed a small monitoring agent. Once the system looked clean, I shut it down and used it as a template for 100 full clones.
The hypervisor did its part perfectly. Every clone received its own virtual NIC and MAC address. DHCP assigned different addresses, and provisioning changed the hostname according to the VM number. The resulting environment looked exactly as expected.
vm-001 10.20.0.101 vm-002 10.20.0.102 vm-003 10.20.0.103 ... vm-100 10.20.0.200
Ping worked, SSH worked, the monitoring dashboard showed 100 hosts, and every VM produced its own CPU and memory graphs. For a few minutes I considered the job finished.
The problem appeared when I started writing a small inventory script for an unrelated test. I wanted every result to contain something more stable than an IP address, so I added /etc/machine-id to the collected data. The first three rows were identical.
At first I thought the script was reading the local file instead of the remote one. That would have been an embarrassingly simple bug. I logged into several VMs manually and ran the same command.
cat /etc/machine-id
vm-001 returned:
8f75c476869c4b1cbcc8f03a714e1285
vm-018 returned exactly the same value. So did vm-064 and vm-100.
That was the point where the model in my head started to change. The network had created 100 distinct endpoints, but that did not mean the operating systems inside those endpoints had become 100 distinct installations.
The template had already been booted before cloning. systemd had generated an identity, SSH had created host keys, cloud‑init had finished initialization, the filesystem had been created, and WireGuard already had its private key. Shutting down the source VM did not return it to some neutral pre‑identity state. It simply froze all of that persistent state inside the disk image.
The clones were not being born from a blank template. They were waking up from copies of a machine that had already decided who it was.
The first collision was machine‑id, but SSH was much worse
The duplicated /etc/machine-id was the easiest problem to understand. On a systemd‑based Linux installation, this file contains a persistent identifier for the local system. It is small, boring and easy to forget, which is probably why it is such a good example of this entire problem.
Nothing dramatic happens when two machines share it. The kernel still boots, applications still start and the server remains reachable. That makes the collision easy to miss. At the same time, software can use machine‑id directly or derive stable identifiers from it, so duplicated values can confuse inventory tools, agents and services that assume the value distinguishes installations.
The SSH check was less harmless.
Each clone contained the same files under /etc/ssh that had been generated on the original VM:
ls -l /etc/ssh/ssh_host_*
The relevant files included the RSA, ECDSA and Ed25519 host keys. I compared the Ed25519 fingerprints on several machines.
ssh-keygen -lf /etc/ssh/ssh_host_ed25519_key.pub
Every host returned the same fingerprint.
To make sure this was not just some formatting oddity, I hashed the private key itself.
sudo sha256sum /etc/ssh/ssh_host_ed25519_key
Again, identical.
This changed the seriousness of the experiment. A duplicated machine‑id is mainly an identity‑management problem. A duplicated SSH host private key is a duplicated secret.
Suppose vm-023 is destroyed and another clone later receives the same network address. With unique SSH host keys, a client that previously connected to that address can notice that the server identity has changed. With a shared host key, the replacement presents exactly the key the client already trusts.
There is a second consequence that I initially overlooked. If one VM is compromised and its SSH host private key is stolen, the attacker does not obtain the identity of one server. They obtain the identity currently used by the entire cloned fleet.
That was enough to stop treating the problem as a funny lab artifact. The template had silently turned per‑host credentials into shared credentials.
The virtual disk was unique, but the filesystem inside it was not
The third collision was the root filesystem UUID. Once I thought about it properly, this one was almost embarrassingly predictable.
Each clone had its own virtual disk object. From the hypervisor point of view those disks were different. But a virtual disk is only a container for bytes, and the ext4 filesystem inside it had been copied byte for byte from the source machine.
I checked the root device first:
findmnt -n -o SOURCE /
Then the filesystem metadata:
sudo blkid
Every clone reported the same UUID for the root ext4 filesystem.
/dev/vda2: UUID=8e157db1-d42a-4d2c-a85d-d21db08353ad TYPE=ext4
While every disk remained isolated inside its own VM, nothing obviously failed. That made it tempting to ignore the duplication. The problem became much easier to see when I attached two cloned system disks to the same recovery machine.
Now one Linux system could see two filesystems claiming exactly the same UUID.
That is where the difference between disk identity and filesystem identity stops being theoretical. Recovery commands, mount configuration and tooling that expects a UUID to identify one filesystem can become ambiguous. The hypervisor had created another virtual disk, but it had not created another ext4 filesystem. It had copied an existing one.
I briefly considered adding a command to regenerate the UUID during first boot, but that felt too casual for the root filesystem. Boot configuration may refer to the UUID, /etc/fstab may refer to it, and changing it after the system is already assembled can create a much more annoying failure than the one being fixed.
So filesystem identity became a provisioning concern rather than a generic first‑boot cleanup step. In my setup, the disk image is now prepared so the filesystem identity is handled deliberately during image creation or disk provisioning.
That was the first useful design lesson from the experiment: not every identity should be reset in the same place. Different components own different identities, and they should ideally be allowed to create those identities at the correct stage of the machine lifecycle.
cloud‑init was not broken; my image lifecycle was
cloud‑init produced the strangest symptoms because initially it looked like the component that should have prevented exactly this problem.
Some per‑instance initialization did not behave consistently after cloning, so I started with the obvious checks.
cloud-init status --long cloud-init query instance_id ls -la /var/lib/cloud/
The answer was sitting in /var/lib/cloud.
The source VM had already completed a normal boot before it became a template. That meant cloud‑init had already stored local state describing the instance it had initialized. In the first version of the lab I had also preserved instance‑specific NoCloud seed data longer than I should have.
The clones therefore inherited not only software configuration but evidence that first‑boot initialization had already happened.
For a while I was mentally blaming cloud‑init because I expected it to make the cloned systems unique. That expectation was backwards. cloud‑init can initialize a new instance, but it cannot magically know that a filesystem containing completed initialization state is supposed to represent a completely different machine unless the image is prepared for that transition.
The source VM had already crossed the boundary between image and instance.
That distinction became the most useful way to think about the whole setup. Packages belong to the image. Base operating system configuration can belong to the image. A host identity belongs to the instance. Cloud metadata belongs to the instance. A secret generated to identify one particular server belongs to the instance.
Once that boundary is clear, a lot of cleanup decisions stop looking arbitrary.
The fifth duplicate was a WireGuard private identity
WireGuard was the most obvious mistake in retrospect.
The source VM already had a private key under its WireGuard configuration when it was converted into a template. Naturally, the private key was copied to every clone.
Checking the public identity was enough to prove it:
sudo wg show wg0 public-key
The result was identical on every VM.
At the network layer, the environment looked like 100 hosts. At the WireGuard layer, it looked like one cryptographic peer copied 100 times.
This one was entirely self‑inflicted. WireGuard had no way to know that its private key was sitting inside a reusable image, and the hypervisor had no way to know that one particular sequence of bytes inside the virtual disk represented a credential that should become unique.
That realization made me widen the search beyond the original five fields. What about a monitoring agent that generates an installation UUID on its first start? What about a backup client that enrolls once and stores a local identity? What about a service discovery agent that writes its node ID under /var/lib? What about locally generated TLS keys?
A VM image has no concept of identity semantics. It simply contains files and blocks. If some software generates a persistent identifier before the image is sealed, that identifier becomes part of the image unless something explicitly removes it.
The practical question I now ask for every service is simple: if I copy this state to another machine, can that machine appear to be the original one?
If the answer is yes, that state deserves attention before the image is reused.
Checking 100 machines manually was not going to happen
After finding the first few collisions manually, I wanted to know whether the problem existed on a handful of VMs or across the whole fleet. Logging into 100 systems and running five commands on each was not an attractive plan, so I wrote a collector.
The script connects to every VM, extracts the identities I cared about and stores them in a CSV file. It is intentionally simple because the interesting part here is not SSH automation. The interesting part is having a repeatable way to ask whether the supposedly independent machines are actually unique at the layers being tested.
#!/usr/bin/env bash set -u OUTPUT="vm-identities.csv" printf '%s\n' \ "host,machine_id,ssh_ed25519,root_uuid,cloud_instance_id,wireguard_public_key" \ > "$OUTPUT" for i in $(seq -w 1 100); do HOST="vm-$i" echo "Scanning $HOST" >&2 SSH_OPTIONS=( -o BatchMode=yes -o ConnectTimeout=3 -o StrictHostKeyChecking=accept-new ) MACHINE_ID=$( ssh "${SSH_OPTIONS[@]}" "$HOST" \ "cat /etc/machine-id 2>/dev/null" \ 2>/dev/null ) SSH_ED25519=$( ssh "${SSH_OPTIONS[@]}" "$HOST" \ "ssh-keygen -lf /etc/ssh/ssh_host_ed25519_key.pub 2>/dev/null | awk '{print \$2}'" \ 2>/dev/null ) ROOT_UUID=$( ssh "${SSH_OPTIONS[@]}" "$HOST" ' ROOT=$(findmnt -n -o SOURCE /) blkid -s UUID -o value "$ROOT" 2>/dev/null ' 2>/dev/null ) CLOUD_INSTANCE_ID=$( ssh "${SSH_OPTIONS[@]}" "$HOST" ' cloud-init query instance_id 2>/dev/null || cat /var/lib/cloud/data/instance-id 2>/dev/null ' 2>/dev/null ) WG_PUBLIC_KEY=$( ssh "${SSH_OPTIONS[@]}" "$HOST" \ "sudo wg show wg0 public-key 2>/dev/null" \ 2>/dev/null ) MACHINE_ID=${MACHINE_ID//$'\n'/} SSH_ED25519=${SSH_ED25519//$'\n'/} ROOT_UUID=${ROOT_UUID//$'\n'/} CLOUD_INSTANCE_ID=${CLOUD_INSTANCE_ID//$'\n'/} WG_PUBLIC_KEY=${WG_PUBLIC_KEY//$'\n'/} printf '%s,%s,%s,%s,%s,%s\n' \ "$HOST" \ "$MACHINE_ID" \ "$SSH_ED25519" \ "$ROOT_UUID" \ "$CLOUD_INSTANCE_ID" \ "$WG_PUBLIC_KEY" \ >> "$OUTPUT" done echo echo "Inventory written to $OUTPUT" for column in 2 3 4 5 6; do count=$( cut -d, -f"$column" "$OUTPUT" | tail -n +2 | grep -v '^$' | sort -u | wc -l ) echo "column $column: $count" done
The first complete run produced a wonderfully bad result.
Hosts scanned: 100 Unique machine IDs: 1 Unique SSH Ed25519 host keys: 1 Unique root filesystem UUIDs: 1 Unique cloud-init instance IDs: 1 Unique WireGuard public keys: 1
That output changed the problem from a collection of suspicious files into something measurable. The environment contained 100 running operating systems, but five independent mechanisms still saw essentially one identity.
I turned the collisions into a test that could fail the image build
A CSV file was useful while investigating the problem, but I wanted the check to survive after the experiment. If the template was rebuilt six months later, I did not want uniqueness to depend on someone remembering to inspect several files manually.
The next step was a small Python validator. It groups hosts by every collected identity, prints collisions and returns a non‑zero exit code when duplicated values are found. That makes the check useful not only for a lab but also as a gate in a template validation pipeline.
#!/usr/bin/env python3 from __future__ import annotations import csv import sys from collections import defaultdict from pathlib import Path INPUT_FILE = Path("vm-identities.csv") FIELDS = { "machine_id": "Linux machine ID", "ssh_ed25519": "SSH Ed25519 host key", "root_uuid": "Root filesystem UUID", "cloud_instance_id": "cloud-init instance ID", "wireguard_public_key": "WireGuard public key", } def load_inventory(path: Path) -> list[dict[str, str]]: if not path.exists(): print(f"Inventory file not found: {path}", file=sys.stderr) raise SystemExit(2) with path.open("r", encoding="utf-8", newline="") as file: rows = list(csv.DictReader(file)) if not rows: print("Inventory is empty", file=sys.stderr) raise SystemExit(2) return rows def group_hosts( rows: list[dict[str, str]], field: str, ) -> dict[str, list[str]]: groups: dict[str, list[str]] = defaultdict(list) for row in rows: host = row.get("host", "").strip() value = row.get(field, "").strip() if not host: continue groups[value or "<missing>"].append(host) return groups def check_field( rows: list[dict[str, str]], field: str, label: str, ) -> bool: groups = group_hosts(rows, field) missing = groups.pop("<missing>", []) collisions = { value: hosts for value, hosts in groups.items() if len(hosts) > 1 } print("=" * 72) print(label) print("=" * 72) print(f"Unique values: {len(groups)}") print(f"Missing values: {len(missing)}") if not collisions: print("Status: OK") print() return False print(f"Collisions: {len(collisions)}") print() for value, hosts in sorted( collisions.items(), key=lambda item: len(item[1]), reverse=True, ): print(f"Value: {value}") print(f"Used by: {len(hosts)} hosts") for host in hosts[:8]: print(f" {host}") if len(hosts) > 8: print(f" ... {len(hosts) - 8} more") print() return True def main() -> int: rows = load_inventory(INPUT_FILE) print(f"Hosts scanned: {len(rows)}") print() failed = False for field, label in FIELDS.items(): if check_field(rows, field, label): failed = True if failed: print("RESULT: identity collisions detected") return 1 print("RESULT: no identity collisions detected") return 0 if __name__ == "__main__": raise SystemExit(main())
After rebuilding the template, I ran the same validation again. This time the important host‑level identities were unique.
Hosts scanned: 100 Unique machine IDs: 100 Unique SSH Ed25519 host keys: 100 Unique cloud-init instance IDs: 100 Unique WireGuard public keys: 100
Filesystem identity remained a separate validation because I had moved that part into disk provisioning. That separation was intentional rather than an exception hidden in the script.
The rule that came out of this was useful far beyond this particular lab: duplicated values are not automatically bugs, but duplicated identities should always be deliberate.
Fixing the image was cleaner than repairing every clone
My first instinct was to create one large post‑clone script that removed IDs, deleted keys, reset cloud‑init and restarted services. It would probably have worked for this exact image.
It would also have become a trap.
A better solution was to change the template lifecycle so that the image was sealed before instance‑specific identity became permanent.
For systemd, the reusable image should not retain the finished machine ID of the source VM. Before sealing, the machine ID can be cleared so the new instance creates its own value during initialization.
sudo truncate -s 0 /etc/machine-id
I also check /var/lib/dbus/machine-id rather than assuming its layout, because depending on the system it may be linked or managed differently.
ls -l /etc/machine-id /var/lib/dbus/machine-id
For OpenSSH, the host keys are removed before sealing.
sudo rm -f /etc/ssh/ssh_host_*
The important part is not the deletion command itself. The important part is making sure the boot process generates new host keys before SSH is used.
cloud‑init is reset as part of the image sealing process rather than treated as something that will somehow repair an already initialized clone.
sudo cloud-init clean --logs --machine-id
Instance metadata is then supplied separately to each new VM.
WireGuard follows the same principle. Its private key does not belong in a generic template if the key is intended to identify one host. The key is generated during per‑instance provisioning.
sudo install -d -m 700 /etc/wireguard sudo sh -c ' umask 077 wg genkey > /etc/wireguard/private.key wg pubkey < /etc/wireguard/private.key \ > /etc/wireguard/public.key '
The resulting lifecycle is now much easier to understand.
build base image ↓ install software ↓ remove instance-specific identity ↓ seal image ↓ clone ↓ supply instance metadata ↓ first boot ↓ generate machine identity and credentials
Previously I had accidentally built this instead:
build machine ↓ boot it ↓ generate identity and credentials ↓ configure everything ↓ seal finished machine ↓ copy its identity 100 times
Those two pipelines look similar until something actually depends on the copied identity.
There is no single identity of a Linux VM
The experiment became more interesting once I stopped asking whether a clone was unique and started asking who considered it unique.
The hypervisor saw 100 VM objects. The virtual switch saw 100 MAC addresses. DHCP saw 100 clients. DNS saw 100 hostnames. Monitoring saw 100 metric sources.
systemd originally saw one machine identity copied 100 times. SSH saw one host key copied 100 times. ext4 saw 100 copies of one filesystem. WireGuard saw one cryptographic peer. cloud‑init inherited state from one initialized source.
None of those layers were behaving randomly. Each one was internally consistent. They simply used different definitions of machine identity.
That explains why this class of problem can stay invisible for a surprisingly long time. Normal health checks tend to focus on reachability, process state, CPU, memory, HTTP responses and network addresses. A VM can pass all of those checks while still carrying an identity copied from another machine.
The failure is not necessarily a crash. Sometimes it is two monitoring agents overwriting each other. Sometimes it is a strange SSH trust relationship. Sometimes it appears only during recovery, when two disks with the same filesystem UUID are attached to one host. Sometimes it is a security issue caused by a private key that was never supposed to leave the original machine.
The most dangerous cloned identity is often the one nobody remembered existed.
What I check before sealing a VM image now
The template process now has a small identity review before an image is considered reusable:
system identity, including machine‑id and instance‑specific hostnames
SSH, VPN and locally generated TLS private keys
cloud‑init state and first‑boot markers
filesystem and storage identifiers when the provisioning model requires them to be unique
monitoring, backup, service discovery and security‑agent identities
application UUIDs and enrollment state stored under persistent directories such as
/var/lib
The last category requires the most manual work because Linux has no universal command that can tell whether an arbitrary file represents application identity.
One technique that has worked well is to compare the filesystem before and after services are started for the first time. A file that appears once, survives reboots and is later used to identify the host deserves inspection.
I also keep coming back to one simple test. If copying a particular file to another machine allows that machine to impersonate the original one from the point of view of some service, that file probably should not be baked into a generic image unless the sharing is intentional.
Conclusion
The original job was supposed to be boring infrastructure work. Build one VM, clone it 100 times, run a test and delete everything afterwards.
Instead, it exposed a weakness in the way I thought about VM templates.
A powered‑off server is not automatically a golden image. If the machine has already completed first boot and generated host‑specific state, shutting it down does not make that state generic. It only makes the machine easier to copy.
The hypervisor will happily duplicate a filesystem containing machine IDs, SSH private keys, cloud‑init state, application enrollment IDs and VPN credentials because, at that layer, they are just bytes.
A reusable image therefore needs to be incomplete in a very specific way. It should contain the operating system, packages and common configuration, but it should still be waiting to become a particular machine.
After rebuilding the image around that idea, the 100 clones finally became independent not only from the point of view of KVM, DHCP and DNS, but also from the point of view of the services running inside them.
That was the part I had missed at the beginning. The hypervisor had created 100 virtual machines, while several layers inside the first version of those machines still believed they were copies of the same one.
