A pharmacy chain runs a temperature monitor on every vaccine fridge and refrigerated case it owns. Each one is a small STM32 board with a screen the staff glance at, sitting on the store network. There are a few hundred of them. Compliance wants an unbroken temperature record from every cabinet, and one morning the record from cabinet 212 has a gap: the monitor rebooted overnight, and nobody knows why.
With one board on your desk, that is a debugging problem: attach the debugger, reproduce the fault, read the stack. With a few hundred boards across a hundred stores, it is an operations problem. You will not visit cabinet 212, and if you did, the fault would not happen while you were standing there.
So the device has to explain itself. When it faults, it has to capture the evidence and report it off the board on its own, so the crash you could never reproduce becomes one you simply read in a browser. This article follows that path end to end on an STM32F746G-DISCO running Zephyr: from a fault nobody is there to see, to a decoded stack trace on your desk, for one board and for a fleet of them. Most of it is a few lines of configuration. The one genuinely STM32-specific part is keeping the crash report alive across the reboot, and I will flag it when we get there.
The backend here is Spotflow (an observability platform for embedded devices: it collects logs, metrics and crash dumps from a fleet and makes them searchable in one web app), which I work on. The device side is an open-source Zephyr module. The full example is on GitHub so everything below builds and flashes.
The board and the network
The STM32F746G-DISCO is a good stand-in for a wired field device: a Cortex-M7 at 216 MHz, 1 MB of internal flash, 320 KB of SRAM, 8 MB of external SDRAM, a 4.3” 480x272 LCD with capacitive touch, and 10/100 Ethernet through an onboard LAN8742A PHY.
Ethernet is how the crash report leaves the board. The SDK’s own samples default to Wi-Fi wherever the board has it, and a fridge monitor in a back room does not. The board comes up on the network and opens one outbound MQTT-over-TLS connection to Spotflow; nothing inbound, nothing to configure on the store’s router. Two cables get you going:
- The ST-LINK Micro-USB (CN14) on the top edge, to your PC. It powers the board and gives you flashing and a serial console.
- An Ethernet cable (CN9) to a network with internet access.
Turning it on
The application is a plain Zephyr project built with west. The Spotflow device SDK is a west module in west.yml, next to Zephyr pinned to v4.4.0. The board target is the in-tree stm32f746g_disco, so the LTDC display, the FT5336 touch controller and the Ethernet MAC and PHY are already in the devicetree.
The app models the fridge monitor. Every two seconds it reads the cabinet temperature (simulated here, 2 to 6 °C in normal operation), logs it, reports it as a metric, shows it on the LCD and checks it against a safe limit of 8 °C. A button, physical or on-screen, simulates a door left open: the cabinet warms past its 8 °C limit over a few seconds, and the LCD turns red with an alarm.
Turning on the device’s telemetry is configuration. In prj.conf:
CONFIG_SPOTFLOW=y CONFIG_SPOTFLOW_METRICS=y CONFIG_SPOTFLOW_METRICS_SYSTEM=y CONFIG_SPOTFLOW_COREDUMPS=y
The logging line is missing from that list on purpose: the Spotflow backend attaches to Zephyr’s own CONFIG_LOG, so existing LOG_INF and LOG_WRN calls are forwarded unchanged. Adding this to an existing product is a configuration change, not a code change: your existing logging and fault handling keep working as they are.
On first boot the console shows the link come up and the connection to Spotflow establish, within ten to twenty seconds of reset:
[00:00:01.753,000] [inf] phy_mii: check_autonegotiation_completion: PHY (0) Link speed 100 Mb, full duplex [00:00:01.758,000] [inf] spotflow_sample_eth: handler: Interface is up -> starting DHCPv4 [00:00:08.803,000] [inf] spotflow_device_id: spotflow_get_device_id: Using Spotflow device ID: cold-chain-monitor-001 [00:00:10.364,000] [inf] spotflow_net: spotflow_mqtt_establish_mqtt: MQTT connected! [00:00:11.060,000] [inf] fridge_monitor: fridge_monitor_step: Cabinet temperature: 2.6 C
The everyday view: logs and metrics across the fleet
Before the crash, the same connection carries the ordinary telemetry, and at fleet scale that is the point: one place for every device instead of one console at a time. Logs need no code. The temperature lines and the occasional probe warning arrive in the portal as they are, searchable across devices and filterable by unit, firmware version or severity:
[00:00:17.060,000] [inf] fridge_monitor: fridge_monitor_step: Cabinet temperature: 4.2 C [00:00:17.060,000] [wrn] fridge_monitor: fridge_monitor_step: Temperature probe I2C read timeout, retrying
The temperature metric is two calls: register it once, report it every cycle.
static struct spotflow_metric_float *g_temperature_metric; rc = spotflow_register_metric_float("temperature_celsius", SPOTFLOW_AGG_INTERVAL_1MIN, &g_temperature_metric); /* ... every 2 s ... */ rc = spotflow_report_metric_float(g_temperature_metric, temp);
The second argument matters more than it looks. SPOTFLOW_AGG_INTERVAL_1MIN makes the SDK aggregate on the device: it keeps min, max, sum and count for the minute and sends one message when the minute closes. That is what a cold-chain record wants anyway (the auditor asks for the peak, not thirty samples), and it keeps the device healthy when the network drops. An earlier version sent every reading, thirty messages a minute. The first time the board lost its connection, those messages had nowhere to go: the send queue filled, the SDK ran out of memory, and the log filled with errors. At one aggregated message a minute, the same outage is easy to ride out.
System metrics come with CONFIG_SPOTFLOW_METRICS_SYSTEM and need no application code: free heap, CPU utilization, per-thread stack usage, network bytes, connection state and the reset cause of the last boot.
There is one more fleet lever worth naming: the log level is set from the portal. When a single unit misbehaves you raise it to DEBUG remotely, watch, and drop it back to ERROR afterwards, without a firmware release to the whole fleet.
Of those system metrics, the reset cause is the signal cabinet 212 was missing: it records that the board rebooted, and why, instead of leaving you with silence. What it does not tell you is the fault itself; that is what the coredump is for.
The crash, and getting it off the board
Now the fault, and it is a realistic one. When the cabinet gets too warm, the monitor is supposed to raise an alarm through a callback function. That callback is set up at startup, but only on units wired to the alerting system. This one is a display-only unit, so it was never set. The code calls the callback anyway, without checking it exists:
#define SAFE_MAX_CELSIUS 8.0f static alarm_fn_t g_alarm_callback = NULL; /* not registered on this unit */ static void check_temperature(float temp_celsius) { if (temp_celsius > SAFE_MAX_CELSIUS) { LOG_WRN("Cabinet temperature above safe limit: %.1f C (limit: %.1f C)", (double)temp_celsius, (double)SAFE_MAX_CELSIUS); g_alarm_callback(temp_celsius); /* NULL: faults here */ } }
It passes every bench test, because on the bench the cabinet never warms up. It fires in the field, at the exact moment a fridge crosses its limit and the alarm is supposed to save the stock. Pressing the board’s USER button (or the on-screen SIMULATE EXCURSION button) plays that out: the temperature climbs on the LCD, crosses 8 °C, the panel turns red with an alarm, and then the monitor calls the alarm handler.
[inf] fridge_monitor: fridge_monitor_step: Cabinet temperature: 7.8 C [inf] fridge_monitor: fridge_monitor_step: Cabinet temperature: 9.3 C [wrn] fridge_monitor: check_temperature: Cabinet temperature above safe limit: 9.3 C (limit: 8.0 C) [err] os: usage_fault: ***** USAGE FAULT ***** [err] os: usage_fault: Illegal use of the EPSR [err] os: esf_dump: Faulting instruction address (r15/pc): 0x00000000 [err] os: z_fatal_error: >>> ZEPHYR FATAL ERROR 35: Unknown error on CPU 0
Those two lines are the signature of a call through a null function pointer. The callback is zero, so the CPU jumps to address 0x00000000, which leaves it in an invalid execution state; instead of running anything it raises a usage fault, reported as "illegal use of the EPSR", with the program counter stuck at 0x00000000. On the bench you would read that straight off the debugger. In the field there is no debugger, so the device has to save the evidence itself, and that is the one part the hardware really forces you to think about.
When Zephyr hits a fatal fault it captures a coredump: registers, thread metadata and stack memory. To survive the reboot, the dump has to be written from inside the fault handler and read back on the next boot, and that constrains where it can live. The board has 1 MB of internal flash and a 16 MB QSPI NOR, and the roomy QSPI is the wrong choice. Its driver moves data with interrupt-driven, blocking transfers, and inside a fault handler the scheduler is gone, so nothing can block. The internal flash controller programs synchronously and works in that context. So the coredump goes to internal flash.
STM32F7 internal flash is not uniformly sectored, and a fixed-partitions region has to land on a sector boundary. The layout that works keeps the application in the first 768 KB (sectors 0 to 6) and reserves the final 256 KB sector for the dump:
| Sector |
Size |
Offset |
|---|---|---|
| 0 to 3 | 4 × 32 KB | 0x00000 – 0x1FFFF |
| 4 | 128 KB | 0x20000 – 0x3FFFF |
| 5 to 6 | 2 × 256 KB | 0x40000 – 0xBFFFF |
| 7 | 256 KB | 0xC0000 – 0xFFFFF |
In boards/stm32f746g_disco.overlay:
/ { chosen { zephyr,code-partition = &slot0_partition; }; }; &flash0 { partitions { compatible = "fixed-partitions"; #address-cells = ; #size-cells = ; slot0_partition: partition@0 { label = "image-0"; reg = ; }; coredump_partition: partition@c0000 { label = "coredump-partition"; reg = ; }; }; };
One more choice keeps the dump small. Zephyr's default mode dumps all of RAM between _image_ram_start and _image_ram_end. For this image that region is 187 KB, and the dump fits the 256 KB partition. But most of those bytes are .bss, .data and heap buffers that tell you nothing about the fault. Dumping thread stacks and metadata instead captures the stacks and metadata of every thread, along with the faulting context's registers. That gives you the crashing call stack without the rest of RAM, and it stays a small fraction of the sector however the image grows. Two config lines switch it on:
CONFIG_DEBUG_COREDUMP_THREADS_METADATA=y CONFIG_DEBUG_COREDUMP_MEMORY_DUMP_THREADS=y
Two labels are load-bearing and easy to miss: Zephyr’s flash-partition backend looks for a partition named exactly coredump-partition, and CONFIG_USE_DT_CODE_PARTITION=y is what actually confines the linker to the first 768 KB. Get those right and the rest is automatic. The dump is written to sector 7, the SDK’s reboot handler resets the board, and about fifteen seconds later it has reconnected and finished the upload:
[00:00:14.402,000] [inf] spotflow_coredump: spotflow_poll_and_process_enqueued_coredump_chunks: Coredump successfully sent.
Reading the root cause in a browser
This is the payoff, and the part you could not do from a site visit. The dump arrives as raw addresses until Spotflow has your symbols. You upload the build’s zephyr.elf once, on the Firmware Management page, and those symbols decode the stack frames and variable names for every crash from that build.
Spotflow matches each crash to the exact firmware build it came from and uses that build’s symbols automatically, so you never pick the right file by hand. This is what makes it work across a fleet: with several firmware versions in the field at once, every dump still resolves against the build that produced it.
With the symbols in place, the dump decodes: the faulting thread, the stack running back through check_temperature() into fridge_monitor_step(), register values, and an AI-generated explanation of the root cause. You are reading why cabinet 212 rebooted, from your desk, on a board you never touched.
What to carry to your own STM32 fleet
- Let the device capture its own crash. On a fatal fault, write the coredump to flash and upload it on the next boot. You debug the fault that actually happened in the field, not one you had to reproduce first.
- On STM32, keep the crash path on internal flash. You cannot run the QSPI driver from a fault handler, so the dump goes to internal flash, on a sector boundary, and dumps threads rather than all of RAM so it fits one sector.
- Archive the ELF for every release. Spotflow matches each crash to the build that produced it, so even a fleet on several firmware versions resolves each dump to the right symbols on its own.
- Size the pipe for a device that goes offline. Aggregate metrics on the device; a field unit loses connectivity, and a queue sized for a connected one fills in minutes.
- Put the whole fleet in one place. Logs, metrics and crashes searchable across devices, with the log level switchable per unit from the portal, beats tailing one board at a time.
None of the device-side work depends on the backend. Getting a crash off an STM32F7 in one piece takes the same effort whether you send it to Spotflow or to your own server. Zephyr gives you the coredump and the log line, and nothing more. Collecting them, storing them, matching each dump to its symbols, searching across the fleet, and changing a device’s log level remotely: that is the part you would otherwise build and run yourself. With Spotflow it is a west module, a config file, and the handful of decisions above.
Resources
- Example code, board bring-up through crash: firmware-observability-examples/stm32-cold-chain-monitor. To run it on your own board, follow the README: it walks through the Spotflow credentials and the build.
- Spotflow device SDK (Zephyr module): github.com/spotflow-io/device-sdk
- Spotflow docs: crash reports and coredumps, crash reports with Zephyr, metrics
- Zephyr: coredump subsystem, STM32F746G Discovery board
- STMicroelectronics: STM32F746G-DISCO, RM0385 reference manual (flash sector layout, section 3.3)
Michael Mikuš works on Spotflow, where the team builds tooling for getting logs, metrics and crash dumps off constrained devices and into something engineers can query. He writes about remote diagnostics and firmware monitoring for device fleets in the field.