A few weeks ago I was sitting next to a programmer who has been dealing with production software for longer than I have been using computers.

We were looking at an annoying backend problem. Requests occasionally failed after deployment, but only under load. Restarting the process fixed everything for a while.

My automatic reaction was predictable.

Search the exception. Check Stack Overflow. Ask an AI tool. Search GitHub issues. Maybe paste the suspicious function somewhere and see what comes back.

He did none of that.

For maybe fifteen minutes he barely touched the code.

He checked the logs, wrote three possible causes in a text file, rejected one of them, added a tiny piece of instrumentation, ran the program twice, then opened the implementation of a library function.

At one point I asked whether it would be faster just to search for the error.

His answer was simple: first we need to know what we are searching for.

That sentence annoyed me a little, mostly because it was obviously correct.

A day later I decided to try something stupid: spend one working week debugging without Google, Stack Overflow, Reddit, GitHub issue searches or AI coding assistants.

Not permanently. I am not moving into a cabin with a ThinkPad and a printed POSIX manual.

Just seven days.

The only things allowed were local documentation, official documentation I already knew how to reach directly, source code, tests, logs, Git history and tools already installed on the machine.

The experiment quickly became less about living without search and more about noticing how often I normally skip the actual debugging part.

The first surprise: I was searching before I had a question

The first morning was uncomfortable.

A Python service threw an occasional JSON decoding error while reading messages from another process over a socket. Normally the exception text would have been copied into a search engine within thirty seconds.

Instead I opened the code.

The relevant part looked completely harmless:

header = sock.recv(4)
size = struct.unpack("!I", header)[0]
payload = sock.recv(size)
message = json.loads(payload)

A four-byte length prefix, followed by a JSON payload. Nothing exotic.

The logs pointed at json.loads, so the first hypothesis was bad JSON.

I dumped the received bytes.

They were indeed incomplete.

That seemed to confirm the hypothesis for about thirty seconds, until the next question appeared: why would the sender produce incomplete JSON?

The sender used sendall. The serialized object was correct before transmission. Checksums matched before the write.

So the JSON parser was probably innocent.

This is where the veteran-programmer habit started making sense. Instead of naming technologies, he had been naming assumptions.

My assumptions were:

The sender produces one complete frame.

The receiver reads four bytes of header.

The receiver then reads exactly the number of payload bytes stored in that header.

The first statement was easy to verify.

The second one was not guaranteed at all.

TCP gives us a byte stream. It does not preserve the boundaries of our application messages. A call to recv(4) can return four bytes, but it may also return one, two or three. The same problem applies to the payload.

On a local machine with tiny messages, the bug had hidden itself surprisingly well.

So instead of searching for Python JSON random error, I wrote a small reproducer.

Python — deterministic test for fragmented TCP reads

import json
import random
import socket
import struct
import threading
import time
from typing import Optional


def encode_message(data: dict) -> bytes:
    payload = json.dumps(
        data,
        separators=(",", ":"),
        ensure_ascii=False,
    ).encode("utf-8")

    return struct.pack("!I", len(payload)) + payload


def send_fragmented(
    sock: socket.socket,
    frame: bytes,
    min_chunk: int = 1,
    max_chunk: int = 5,
) -> None:
    """
    Deliberately split one application frame into many small writes.

    TCP does not promise that the receiver will observe the same
    boundaries, but this makes partial reads much easier to reproduce.
    """
    offset = 0

    while offset < len(frame):
        remaining = len(frame) - offset
        chunk_size = random.randint(
            min_chunk,
            min(max_chunk, remaining),
        )

        chunk = frame[offset:offset + chunk_size]
        sock.sendall(chunk)

        offset += chunk_size

        # Make fragmentation visible even on a fast local machine.
        time.sleep(0.002)

    sock.shutdown(socket.SHUT_WR)


def read_message_buggy(sock: socket.socket) -> Optional[dict]:
    """
    This version contains the assumption that caused the real bug:
    one recv() call is expected to fill the requested buffer.
    """
    header = sock.recv(4)

    if not header:
        return None

    if len(header) != 4:
        raise RuntimeError(
            f"Partial header: expected 4 bytes, got {len(header)}"
        )

    payload_size = struct.unpack("!I", header)[0]

    payload = sock.recv(payload_size)

    if len(payload) != payload_size:
        raise RuntimeError(
            f"Partial payload: expected {payload_size} bytes, "
            f"got {len(payload)}"
        )

    return json.loads(payload)


def recv_exactly(sock: socket.socket, size: int) -> bytes:
    """
    Read exactly size bytes unless the peer closes the connection.
    """
    buffer = bytearray()

    while len(buffer) < size:
        chunk = sock.recv(size - len(buffer))

        if not chunk:
            raise EOFError(
                f"Connection closed after {len(buffer)} "
                f"of {size} bytes"
            )

        buffer.extend(chunk)

    return bytes(buffer)


def read_message_fixed(sock: socket.socket) -> Optional[dict]:
    first_byte = sock.recv(1)

    if not first_byte:
        return None

    header = first_byte + recv_exactly(sock, 3)
    payload_size = struct.unpack("!I", header)[0]

    if payload_size > 10 * 1024 * 1024:
        raise ValueError(
            f"Refusing suspicious frame size: {payload_size}"
        )

    payload = recv_exactly(sock, payload_size)

    return json.loads(payload)


def run_once(reader) -> None:
    sender, receiver = socket.socketpair()

    message = {
        "type": "build_finished",
        "project": "demo-service",
        "duration_ms": 1847,
        "successful": True,
        "files": [
            "api.py",
            "worker.py",
            "storage.py",
        ],
    }

    frame = encode_message(message)

    thread = threading.Thread(
        target=send_fragmented,
        args=(sender, frame),
        daemon=True,
    )
    thread.start()

    try:
        decoded = reader(receiver)
        print("Decoded:", decoded)
    finally:
        receiver.close()
        sender.close()
        thread.join(timeout=1)


def main() -> None:
    print("Testing buggy reader")

    try:
        run_once(read_message_buggy)
    except Exception as exc:
        print(
            f"{type(exc).__name__}: {exc}"
        )

    print()
    print("Testing fixed reader")

    run_once(read_message_fixed)


if __name__ == "__main__":
    main()

The important part was not the recv_exactly function. That function is boring.

The useful part was finding the false assumption.

Before this experiment, I would probably have searched the exception first. Maybe the correct answer would have appeared immediately. Maybe not.

But the search phrase itself would already contain my interpretation of the bug.

And my interpretation was wrong.

The JSON parser was simply the place where corrupted assumptions finally became visible.

How often does that happen in normal debugging? The line that throws the exception gets accused simply because it happens to be holding the body when the murder is discovered.

Debugging became much easier when I started writing down what must be true

By the second day, one habit had changed.

Before touching code, I started writing a tiny list:

Observed fact
Possible explanation
Test that could make the explanation false

Nothing fancy.

No elaborate debugging template.

For example, another service was occasionally taking thirty seconds to return even though its normal response time was below a second.

The obvious suspect was the database.

So the first note looked like this:

Observed: HTTP request remains open for about 30 seconds.

Hypothesis: database query blocks.

Test: log timestamps immediately before and after the query.

The query took 14 milliseconds.

Good. One suspect gone.

Next hypothesis: connection pool exhaustion.

Test: record pool acquisition time separately from query execution.

No problem there either.

Next: DNS.

Then TCP connect time.

Then application semaphore.

The difference sounds small, but it changed the whole session. Usually debugging feels like walking through a dark room and touching random objects. Now every command was supposed to kill or strengthen one hypothesis.

That also changed how I used logs.

Previously, adding logs often meant dumping more state.

Now the question became more specific: what is the smallest piece of information that separates hypothesis A from hypothesis B?

That is much more useful than adding ten INFO lines and hoping the bug feels guilty enough to confess.

There was another unexpected effect.

Without a search tab waiting nearby, reading source code stopped feeling like the expensive option.

A library behaved strangely, so I followed the call.

Then another call.

Then one more.

Three minutes later the weird behaviour made complete sense.

This was probably the biggest psychological change during the experiment. Source code had always been available, but search made it feel slower than it actually was.

The nastiest bug of the week looked like a network problem

The second interesting failure happened in an asyncio worker.

After several hours, some requests simply stopped progressing. CPU usage was low. Memory looked normal. The database was fine.

Restarting the worker immediately fixed it.

That is exactly the kind of bug where searching feels irresistible.

Asyncio random hang after hours would have been a wonderfully terrible query.

Instead, the process got the same treatment as the socket bug.

First question: what resource do successful requests obtain that stuck requests are waiting for?

The service had a concurrency limit implemented with an asyncio.Semaphore.

That immediately became interesting.

The worker allowed three expensive external operations at the same time. Under normal conditions, a task acquired a permit, completed its request and released the permit.

Except sometimes tasks were cancelled.

A timeout higher in the stack could cancel a coroutine while it was suspended inside an await.

The release happened after that await.

Which meant it sometimes never happened.

One leaked permit was invisible.

Two made the service slower.

Three turned the semaphore into a tiny permanent parking lot.

The networking code had done nothing wrong.

Again.

I wrote another reduced example.

Python — reproducing a leaked asyncio semaphore

import asyncio
from collections.abc import Awaitable, Callable


class ExternalService:
    def __init__(self, concurrency: int = 3) -> None:
        self._limit = asyncio.Semaphore(concurrency)

    async def call_buggy(self, request_id: int) -> str:
        """
        BUG:
        If cancellation happens during the simulated I/O operation,
        release() is never reached.
        """
        await self._limit.acquire()

        print(
            f"[buggy] request {request_id} acquired a slot"
        )

        await asyncio.sleep(10)

        self._limit.release()

        return f"result-{request_id}"

    async def call_fixed(self, request_id: int) -> str:
        """
        The async context manager releases the semaphore
        when leaving the block, including cancellation paths.
        """
        async with self._limit:
            print(
                f"[fixed] request {request_id} acquired a slot"
            )

            await asyncio.sleep(0.1)

            return f"result-{request_id}"


async def cancel_after_acquire(
    operation: Callable[[int], Awaitable[str]],
    count: int,
) -> None:
    tasks = [
        asyncio.create_task(operation(i))
        for i in range(count)
    ]

    # Give every task enough time to acquire its semaphore slot.
    await asyncio.sleep(0.05)

    for task in tasks:
        task.cancel()

    results = await asyncio.gather(
        *tasks,
        return_exceptions=True,
    )

    cancelled = sum(
        isinstance(result, asyncio.CancelledError)
        for result in results
    )

    print(f"Cancelled tasks: {cancelled}")


async def probe(
    operation: Callable[[int], Awaitable[str]],
    label: str,
) -> None:
    try:
        result = await asyncio.wait_for(
            operation(999),
            timeout=0.5,
        )

        print(
            f"{label}: probe completed with {result}"
        )

    except asyncio.TimeoutError:
        print(
            f"{label}: probe timed out waiting for a semaphore slot"
        )


async def demonstrate_bug() -> None:
    print("=== Buggy implementation ===")

    service = ExternalService(concurrency=3)

    # All three tasks acquire one permit and are then cancelled.
    # Because release() lives after the await, all permits leak.
    await cancel_after_acquire(
        service.call_buggy,
        count=3,
    )

    # Nothing is performing network I/O now, but the new request
    # cannot acquire a permit.
    await probe(
        service.call_buggy,
        "buggy",
    )


async def demonstrate_fix() -> None:
    print()
    print("=== Fixed implementation ===")

    service = ExternalService(concurrency=3)

    await cancel_after_acquire(
        service.call_fixed,
        count=3,
    )

    # Cancellation exits the async-with block and returns permits.
    await probe(
        service.call_fixed,
        "fixed",
    )


async def main() -> None:
    await demonstrate_bug()
    await demonstrate_fix()


if __name__ == "__main__":
    asyncio.run(main())

This one bothered me more than the TCP issue because the original code looked reasonable during review.

Acquire.

Perform operation.

Release.

Anyone reading quickly understands the intention.

The missing piece is that async code has more exits than the happy-path indentation suggests. Cancellation can arrive at an await, and cleanup cannot depend on execution continuing normally afterward.

The fix was tiny.

The debugging session was not.

And that seems to be a recurring pattern in backend work: the size of a fix has almost no relationship with the amount of reasoning required to find it.

The veteran programmer was not avoiding tools

Halfway through the week, something clicked.

At first I had interpreted the older programmer's workflow as old-school minimalism.

That was wrong too.

He was using plenty of tools.

strace.

gdb.

grep.

git diff.

git log.

git bisect.

Packet captures.

Database clients.

Tiny reproduction programs.

He just used tools that exposed the system rather than tools that immediately proposed an explanation.

That distinction turned out to be important.

For example, when a process appeared frozen, the temptation was to stare at application logs. But if no application log was being produced, that itself told very little.

A quick strace -p PID could show that the process was blocked on a socket, waiting on a futex, repeatedly opening a missing file, or doing something completely different from what the application-level mental model suggested.

The same thing applied to /proc.

When memory looked suspicious, instead of searching for generic Python memory leak advice, checking /proc/<pid>/status, open file descriptors and process mappings gave a much narrower starting point.

Git became a debugging instrument too.

This sounds obvious, but git bisect is ridiculously effective when two facts are known:

the old revision works
the new revision fails

If there are 512 candidate commits, binary search needs only about nine decisions to reduce that history to one commit.

That is an absurdly good deal.

Yet in everyday work it is surprisingly easy to spend an hour reading code changed during the last month instead.

The week gradually became an exercise in asking one question:

Can the machine give me a fact before somebody else gives me an opinion?

Usually it could.

Removing search made the first ten minutes worse and the next hour better

There is a downside, and it would be silly to hide it.

Some problems became slower.

A strange compiler flag, an obscure library behaviour, an undocumented compatibility issue — search is excellent for these.

There were moments when the rule felt artificial.

One afternoon I spent twenty minutes discovering something that probably had a perfect three-line answer online.

That was not enlightening. It was just inefficient.

But another pattern appeared.

My normal debugging sessions often started very fast and then became messy.

Search error.

Open five tabs.

Read two GitHub issues.

Try one suggestion.

Change an environment variable.

Restart.

Find a similar exception from a different library version.

Try another suggestion.

Twenty minutes later there are six changes in the working tree and no clear idea which assumption is currently being tested.

Without search, the beginning was slower.

The middle was much cleaner.

By day four, the first few minutes of a bug usually looked like this:

Reproduce it.

Reduce it.

Identify the boundary where correct state becomes incorrect.

Write down the assumptions crossing that boundary.

Add one observation point.

Change one variable.

Repeat.

It sounds painfully basic when written down.

Maybe that is why it is so easy to ignore.

Have you ever noticed how quickly a debugging session turns into code editing? Sometimes the first modification happens before the bug has even been reproduced twice.

That happened to me constantly.

During this week, code changes became almost the last step.

I also started distrusting error messages in a healthier way

Error messages are useful, but they describe where a system noticed a contradiction.

They do not necessarily describe where the contradiction began.

The TCP bug produced a JSON exception.

The semaphore leak looked like network latency.

A bad configuration once looked like an authentication problem because the first component capable of rejecting the state happened to be the authentication layer.

This sounds obvious after the bug is solved.

Before it is solved, the exception has enormous psychological gravity.

It gives the problem a name.

And once a problem has a name, the brain starts collecting evidence that supports it.

JSON error becomes JSON problem.

Timeout becomes network problem.

Database exception becomes database problem.

The veteran programmer seemed much less impressed by that naming.

He kept asking about the last known good state.

That turned out to be one of the most useful questions of the week.

If the API response is wrong, was the database row correct?

If yes, was the object correct after deserialization?

If yes, was it correct before transformation?

If yes, was it correct before serialization?

Walk backward until the state stops being wrong.

Or start from the input and move forward.

Either direction works.

What matters is turning a giant system into a sequence of boundaries.

Once the faulty boundary is known, searching becomes much more powerful too.

Now the query is not Python JSON randomly broken.

It is closer to partial TCP reads with length-prefixed protocol.

That is a completely different quality of question.

The most useful tool was a stupid text file

No special debugging application won the week.

It was a plain text file.

For every non-trivial bug, I started writing something like this:

SYMPTOM
Worker stops accepting new jobs after several hours.

KNOWN
- Process is alive.
- Event loop still runs.
- CPU is near idle.
- Database responds normally.
- New tasks are created but do not reach external_call().

UNKNOWN
- What are they waiting for?
- Is the concurrency limiter exhausted?
- Can a permit disappear?

HYPOTHESIS
Cancelled tasks leak semaphore permits.

DISPROVE
Cancel a task after acquire and check whether another task
can acquire the same permit.

RESULT
Reproduced in 20 lines.

NEXT
Audit every manual acquire/release pair.

It feels almost embarrassingly primitive.

But it prevented the debugging process from becoming a pile of half-remembered ideas.

More importantly, it separated facts from guesses.

That distinction gets blurry surprisingly fast.

After forty minutes with a bug, a sentence such as Redis is probably fine can quietly turn into Redis is fine without anyone actually testing it.

Writing things down makes that harder.

It also gives debugging some persistence.

When interrupted by a message or meeting, returning to the bug no longer requires rebuilding the entire mental state from scratch.

For a student project this may seem unnecessary.

For a production incident at 2 AM, it suddenly looks much less silly.

What I kept after turning the internet back on

At the end of the seventh day, Google came back.

So did Stack Overflow.

So did AI tools.

There was no dramatic moment where modern software development had been exposed as a mistake.

Search is useful.

AI tools are useful.

A good Stack Overflow answer can save hours.

The experiment did not convince me to stop using any of them.

It changed the order.

Before the week, the workflow was often:

See error → search error → try plausible solution → investigate if necessary.

Now it is closer to:

See error → reproduce → identify boundary → write hypotheses → collect evidence → search with a precise question.

That small reordering matters.

AI is also much more useful at the end of that chain.

There is a huge difference between asking why is my asyncio service freezing and providing a minimal reproducer where cancelled coroutines acquire a semaphore and never return the permit.

The second question contains actual engineering work.

And sometimes, once the question becomes that precise, the answer is already sitting in front of you.

The biggest lesson from watching an experienced programmer was not that older developers know more obscure commands.

Of course experience matters. After enough years, certain failure patterns become familiar.

But the more interesting difference was behavioural.

He did not seem desperate to make the error disappear.

He wanted the system to explain itself first.

That is a subtle difference.

Fixing an error and understanding an error are often the same task, but not always.

Modern tools are extremely good at helping with the first one.

The second still requires someone to ask the machine the right questions.

So no, I am not planning another internet-free debugging week.

But one rule survived:

Before searching for the answer, spend ten minutes making sure you actually know the question.

I Spent a Week Debugging Without Google, Stack Overflow, or AI — What Changed Wasn’t My Speed