One of the most inspiring moments in microcontroller development is their first installation and launch within a local network. Yet, this very moment can turn into a living hell when the procedure has to be repeated dozens or hundreds of times. Deploying dozens of identical controllers simultaneously demands hours of monotonous configuration—time that any engineer would rather spend on more creative and useful tasks. To solve this problem, a lightweight open-source service codenamed "Esp monitor" was developed.
The Configuration Problem and Protocol Choice
One of the most popular (and importantly, cheap—costing around $2) microcontrollers is the Wemos D1 Mini based on the ESP8266 chip. It is fast, reasonably reliable, and boasts a vast ecosystem of add-on modules. These are the exact controllers widely used in household appliances and DIY projects due to their massive potential for integration into sensors and home automation systems.
However, they are also among the most inconvenient to configure: by default, the only wireless interface "onboard" is Wi-Fi. This means connecting to a new wireless access point turns into another "dance with a tambourine" (an uphill battle).
It might seem that this task could be solved by classic port forwarding out of the local Wi-Fi network, but in most scenarios, this is a dead end:
It requires administrative access to the Wi-Fi router;
Exclusive access must be provided for each device hidden behind NAT;
There is no single control center for the device family—you have to access each device individually by its own address;
Constant security concerns arise.
A much more elegant and practical solution is using an external MQTT broker outside the local network. The device itself connects to the broker, transmits its current state, and awaits executable commands.
MQTT (Message Queuing Telemetry Transport) is a very lightweight, bidirectional telemetry transport protocol. Unlike WebSockets, it does not require holding a heavy, persistent connection, making it the ideal standard for IoT. You can run an MQTT broker on any single-board computer (including a Raspberry Pi 3), rent one in the cloud, or deploy it using free public services.
Esp-Server Architecture
The management tool esp-server is a lightweight service written in Go with an embedded SQLite database. It can be deployed in the cloud on any system supporting Docker or compiled from source code wherever a Go runtime environment is available (Windows, Linux, macOS).
The server centrally manages devices, automating complex configurations that include Wi-Fi settings, mDNS, the standard SSDP stack (with an HTTP port for reading data), and connection parameters for the MQTT broker. Any other logic can be easily implemented by the engineer using standard MQTT methods.
Besides the server component, the project repository contains firmware for the ESP8266 and ESP32, which can be easily adapted to any task—from a simple sensor to a smart home automation hub.
First-Time Network Onboarding and API
Users interact with esp-server either through a public REST API or a web interface in a browser. While mobile apps are more convenient, the web interface wins on versatility.
To pair microcontrollers with a local network, you can use the native iOS app (the source code of which is also available in the repository). In this app, you only need to enter the configuration parameters once and then send them to the device operating in Access Point (AP) mode with a single button press. However, Apple's policies complicate life for engineers: building the app requires the Xcode development environment and a developer account.
For those using Android or preferring universal methods, an alternative and extremely simple approach is offered—sending parameters via a regular cURL request (POST REST):
Bash
curl -X POST 192.168.4.1:80/config \ -H "Content-Type: application/json" \ -d '{ "WIFI_SSID": "<wi-fi SSID>", "WIFI_PASS": "wi-fi pass", "MDNS": "<mdns identifier, e.g. local.esp>", "SSDP": "SSDP name", "HTTP_PORT": "80", "MQTT_HOST": "<HOST>", "MQTT_PORT": "1883", "MQTT_USER": "", "MQTT_PASS": "", "MQTT_ROOT": "<root topic>" }'
(Note: ) An example of this request can be found in the ./clients/curl directory of the project.
Important Nuances of Working with Devices:
First Launch: Upon its first boot, the ESP device spawns its own Access Point (AP) at the default address
192.168.4.1(under SSIDesp8266-setuporesp32-setupdepending on the chip). The engineer's job is to send a REST request to this address. Once the parameters are successfully received, the device reboots, connects to the local Wi-Fi network, and establishes communication with the MQTT broker.Emergency Reset: If something goes wrong, you can physically reset a faulty configuration by holding the button on pin
D4for more than 3 seconds.Identification (SSDP name): This is the key field by which the device is identified on the
esp-serverside. If left empty, the name is auto-generated based on the chip type and its MAC address. If necessary, you can override thecreateMacAddress()method directly in the firmware to use Unix time instead of a MAC address.MQTT_ROOT: If you host your own MQTT broker, simply specifying a short name (e.g.,
root) is sufficient. However, if you are using a public broker liketest.mosquitto.org, it is better to define a unique identifier as the root (such as a UUID generated via a dedicated website) to ensure your topics do not collide with others.
Project Evolution: From Static Imports to Interactivity
Initially, the baseline scenario was envisioned as follows: an engineer logs controller configurations in an Excel spreadsheet, exports it to .csv, uploads it to the server, and views a successful setup status once the device connects. However, real-world practice proved that simply monitoring a list is not enough—there arose a demand to alter device configurations on the fly.
It is important to understand that the ESP devices themselves have no awareness of esp-server—they interact exclusively with the MQTT broker. In turn, esp-server knows nothing about the internal circuitry of the ESP—it merely reads from and writes to topics defined in the server's .env configuration file.
Two primary topics are used for data exchange:
<root>/<ssdp_name>/state— sends the current state of pins from the ESP to the server<root>/<ssdp_name>/action— transmits control commands from the server to the ESP.
The firmware enforces a rate limit on sending data to the state topic—no more than once per second—to avoid overloading the network (this limit can be removed if necessary). The server considers a device active (online) if messages have been received from it within the last minute. Since the ESP32 is hardware-wise more stable and less prone to "contact bounce" under no load, it is crucial to ensure that the ESP32 firmware sends at least a minimal ping packet once per minute to maintain its network status.
Interface and Security
Adding new devices happens automatically: as soon as the server spots a new message in the <root> topic, it extracts the <ssdp_name> and registers the new device in the SQLite database. Unauthorized users can only view a basic registry of registered devices, their mDNS, and their uptime. That is where their privileges end.
Since the server must face the internet but administration rights must remain exclusive, a simple and reliable Linux-style authorization was implemented:
A secret parameter,
xtoken, is defined in the server's configuration file.Whoever has physical access to the server's configuration file ("access to the machine") has control over the system.
The user inputs this token on the login page, and the session is saved in the browser.
Web UI for not authorized user Authorized users have the ability to:
View the expanded registry (activation date, boot time, last update date).
Filter devices by SSDP or mDNS name (useful for administering different device families on a single server).
Import device databases via
.csvon the landing page.Navigate to a detailed device card: view pin states, the history of recent actions, and delete inactive devices (those in a timeout state).
Configuration ("Configure device"): The capability to override any device parameters directly from a mobile screen.
Deployment Configurations (Deploy Tiers)
This is where we just begin to descend into the "rabbit hole" of the project's architecture. Esp-Monitor supports 5 primary deployment tiers depending on the scale and demands of your network:
Tier | Infrastructure Composition and Description |
|---|---|
Tiny | Minimal tier. Only |
Small | A local private MQTT broker is added to the server. Data is transmitted within your security perimeter. |
Medium | A distributed event streaming platform, Apache Kafka, is introduced, along with the Kafdrop web UI for Kafka, and specialized workers: |
Large | Comprehensive monitoring. Adds the Vector metrics collector, VictoriaMetrics time-series database (compatible with Prometheus), and Grafana dashboards. |
Huge | Same as Large, but the entire infrastructure is deployed within the Kubernetes (k8s) orchestrator. |
Docker or Kubernetes: What to Choose?
In the repository, the deploy folder is split into two directories:
docker— contains configuration files for deploying tiers from Tiny to Large.k8s— contains scripts for setting up a local Minikube environment and manifests to launch a Kubernetes cluster.
Pros and Cons of Kubernetes:
Advantages: A self-healing and easily scalable platform. If one of the services crashes, K8s restarts it automatically. Components can be distributed across different physical servers, or dozens of worker replicas can be launched for different data segments. All of this remains packaged into unified manifests.
Disadvantages: A high barrier to entry. You will have to sacrifice a fair share of nerve cells to get the basic configuration working.
Note: To simplify things, pre-made automated installation scripts are included in the project. You won't have to plunge deep into the weeds of the Docker CLI or Kubernetes API—deployment and restarts take literally a couple of commands. Furthermore, Minikube provides a convenient web dashboard out of the box.
If you opt for classic Docker, you get other benefits: pre-built, lightweight images on Docker Hub are compiled for three architectures simultaneously (amd64, arm64, and arm). This allows you to run the system on PCs, Apple Silicon-powered Macs, and energy-efficient single-board computers like Raspberry Pi.
Under the Hood: Kafka and Workers (Medium+)
You might have noticed that esp-server itself only stores the last state of the pins and the latest command. However, when opting for the Medium tier or above, the full potential of the architecture is unlocked: esp-server streams absolutely all events to Kafka. Once inside, they can be stored and accumulated for weeks.
Currently, 5 topics are predefined in Kafka:
root— system events (e.g., the launch ofesp-serveritself).logs— diagnostic messages, errors, broker connection logs, and bus logs.pins— detailed history of microcontroller pin state transitions.actions— sent commands and configuration updates.anomalies— operational anomalies detected by external processors.
A specialized service, ai-worker, writes to the anomalies topic. Overall, three types of workers are shipped out of the box:
kworker — builds aggregated tables, showing changes in ESP pin states in real time.
ai-worker — streams aggregated data to an AI analysis engine to detect anomalies and writes the results back to Kafka.
hworker — fetches detected anomalies from Kafka and pipes them to the next level of business logic (for instance, to automation workflows in platforms like n8n).
Thanks to this distributed design, the workers can be scaled horizontally: you can spin up as many instances as there are partitions in the Kafka topics, distributing the workload evenly. The source code for all workers is located in the services directory and is fully open for customization.
Real-Time Pin Oscillogram (Large+)
For a detailed analysis of physical sensor data, the Vector + VictoriaMetrics + Grafana stack is used. The Vectoragent reads raw pin change data from Kafka, transforms it into time-series data, and writes it to the high-performance VictoriaMetrics DBMS. By hooking up Grafana, you can generate beautiful charts, track contact bounce, and design robust engineering dashboards to monitor your apartment or production line.
Quick Start
Requirements:
Docker installed (this is also required for running Kubernetes via Minikube).
Environment Setup:
In the deploy/docker and deploy/k8s directories, you will find a my.env.example file. Copy it, rename it to .env, and fill in the parameters (tokens, broker addresses) as shown in the file's examples.
Startup Commands:
Tiny:
docker compose -f deploy/docker/tiny.yaml up -dSmall:
docker compose -f deploy/docker/small.yaml up -dMedium:
docker compose -f deploy/docker/medium.yaml up -dLarge:
docker compose -f deploy/docker/large.yaml up -dHuge (Kubernetes):
Run the initialization script:
sh ./setup-k8s-cluster.shLaunch the cluster using the
en-start-clusterorru-start-clusterscript (these are simple bash scripts whose structure you can inspect in any text editor).
Please Note: When starting the Minikube tunnel, the system will prompt you for your
sudopassword. This is necessary to map (forward) the internal cluster ports to your host machine so you can open the web interfaces ofesp-server, Grafana, Kafdrop, and the Minikube Dashboard in a standard browser.
Documentation and Source Code
The project is documented in great detail: almost every folder contains its own README.md file. The docs/archdirectory contains highly comprehensive architectural documentation, spanning 11 chapters in both Russian and English.
The project is actively evolving, and its source code is completely open-source.