Keep ESP32 OTA reachable whatever the firmware does

- Rollback: override verifyRollbackLater() so a new OTA image stays
  pending until it ran 60 s with WiFi; any earlier restart rolls back
  (the Arduino core marked every image valid at boot before)
- Task watchdog (30 s) on loop(), paused during OTA uploads
- Crash counter in RTC memory: OTA-only window after a crash, 10 min
  safe mode (WiFi + OTA only) after 3 crashes in a row
- Restart when WiFi stays down for 10 min
- Home Assistant sensor 'FlipDot Display Firmware' with version/state
- logic/boot_policy with tests (60 tests total)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

CLAUDE.md

@@ -91,6 +91,21 @@ See README.md for the overview.
   The unique ids `FlipDotDisplayClock` / `FlipDotDisplayPower` and `HA_DEVICE_ID` must stay
   the same, or Home Assistant creates new entities.
 
+## Keeping ESP32 OTA alive (most important requirement)
+
+- The user never wants to need USB for the ESP32 again. Any change must keep OTA working,
+  also when the new code crashes or hangs.
+- `system/firmware_guard.cpp` overrides the Arduino core's weak `verifyRollbackLater()`
+  (C linkage) to return true. A new OTA image stays `PENDING_VERIFY` until it has run
+  `FIRMWARE_VALID_TIME` with WiFi. Any restart before that makes the bootloader roll back.
+  Check with `nm` that the strong symbol is linked after toolchain or core updates.
+- The task watchdog (`WATCHDOG_TIMEOUT_S`) covers `loop()`. ArduinoOTA's `onStart` stops
+  it, because an upload blocks `loop()`. Never block `loop()` for 30 s or longer.
+- The crash counter lives in `RTC_NOINIT_ATTR` memory. Decisions are in
+  `logic/boot_policy.*` (tested). Safe mode runs only WiFi + OTA.
+- Keep `otaLoop()` and `firmwareGuardLoop()` at the top of `loop()`, before anything that
+  can return early or block.
+
 ## Git
 
 - Remote: https://mangavalk.com/mangavalk/Flipdot-display.git, branch `main`.

README.md

@@ -170,9 +170,10 @@ The ESP32 talks to the Nano on **UART2** (`Serial2`, 115200 baud). The ESP32's o
 
 ## How it works
 
-1. **ESP32 boot.** The ESP32 connects to WiFi and starts ArduinoOTA. After a crash or
+1. **ESP32 boot.** The ESP32 connects to WiFi and starts ArduinoOTA. After a crash it
-   unexpected reset it stays in an *OTA panic window* for 15 seconds, so you can flash a
+   runs only OTA for 15 seconds, and after repeated crashes it goes to safe mode. Bad OTA
-   fix before the main code runs again. It then switches the display relay on, waits for
+   firmware is rolled back automatically, see
+   [Keeping OTA alive](src/esp32/README.md#keeping-ota-alive). It then switches the display relay on, waits for
    the Nano's `READY` message (at most 3 s), shows "Connecting...", starts NTP and
    connects to MQTT.
 2. **Clock.** Once the time is synced, the ESP32 sends a new frame whenever the screen

src/esp32/README.md

@@ -19,6 +19,9 @@ debug output. Wiring is described in the [main README](../../README.md#wiring-be
 | `logic/schedule.*` | When the clock is shown (night, clock, standby) |
 | `logic/power_timer.*` | Countdown to switching the display off |
 | `logic/status_led.h` | Blink pattern of the built-in LED |
+| `logic/boot_policy.*` | How to start after a reset: normal, OTA-only window, safe mode |
+| **`system/`** | **Keeping OTA alive** |
+| `system/firmware_guard.*` | Rollback of bad OTA firmware, watchdog, crash counter, status for Home Assistant |
 | **`display/`** | **What is drawn** |
 | `display/frames.*` | The screens: clock, logo, blank, text (`FlipdotFrames`) |
 | `display/bitmaps.h` | 1-bit bitmaps (Pikachu, thought bubble, Nintendo logo, ...) |
@@ -26,22 +29,54 @@ debug output. Wiring is described in the [main README](../../README.md#wiring-be
 | `nano_link/nano_link.*` | Relay and serial link: power on (waits for `READY`), power off, bootloader start (`NanoLink`) |
 | `nano_link/nano_flash_bridge.*` | WiFi to serial bridge for flashing the Nano over WiFi (TCP port 2323) |
 | **`net/`** | **Network** |
-| `net/wifi_ota.*` | WiFi, ESP32 OTA updates and the OTA panic window |
+| `net/wifi_ota.*` | WiFi (with a restart when it stays down), ESP32 OTA updates, OTA-only mode |
 | `net/home_assistant.*` | MQTT and the two Home Assistant switches |
 
 `DisplayController` only talks to two small interfaces, `DisplayPower` (implemented by
 `NanoLink`) and `Frames` (implemented by `FlipdotFrames`). The tests replace both with
 fakes.
 
+## Keeping OTA alive
+
+The ESP32 must stay reachable over OTA whatever the firmware does, so it never needs USB
+again. Several layers take care of that:
+
+| Problem | What happens |
+|---------|--------------|
+| A new OTA firmware crashes, hangs or can't connect to WiFi | **Rollback.** A new firmware is only marked as good after it has run for 60 s with WiFi. If it restarts before that, the bootloader starts the previous firmware again. |
+| The firmware hangs in `loop()` | **Watchdog.** If `loop()` doesn't run for 30 s, the ESP32 restarts. That counts as a crash. |
+| The firmware crashes once or twice | **OTA window.** After a crash the ESP32 runs only WiFi and OTA for 15 s (LED blinks fast), then starts the main code again. |
+| The firmware keeps crashing (3 times in a row) | **Safe mode.** Only WiFi and OTA run, for 10 minutes. Then the main code gets one more try; one more crash brings safe mode back. Running 5 minutes without a crash clears the counter. |
+| WiFi stays down | After 10 minutes without WiFi, the ESP32 restarts, in case the WiFi stack is stuck. |
+
+So in every case you can flash a fix with `pio run -e esp32 -t upload`, at the latest
+after waiting for a safe-mode cycle.
+
+The **FlipDot Display Firmware** sensor in Home Assistant shows the state, e.g.
+`2.1.0, ok, 0 crashes`:
+
+| State | Meaning |
+|-------|---------|
+| `testing` | New OTA firmware, not yet marked as good (first 60 s) |
+| `ok` | Firmware marked as good |
+| `ok, no rollback in bootloader` | The bootloader on the ESP32 can't roll back. The other layers still work. |
+| `usb` | Flashed over USB, so there is no previous firmware to roll back to |
+
+Rules for changing the code:
+
+- Keep `otaLoop()` and `firmwareGuardLoop()` at the top of `loop()`.
+- Never block `loop()` for 30 s or longer.
+- Change `FIRMWARE_VERSION` in `config.h` with each release, so the sensor shows which
+  version is running.
+
 ## Features
 
 - **Frames only when something changes:** a new frame is sent when the minute changes, the
   screen changes (clock, logo, blank) or the Nano was just powered on. The serial line is
   quiet in between, which also gives the Nano time for its periodic full redraw.
 - **OTA updates** through ArduinoOTA. Hostname: `FlipDotDisplay`.
-- **OTA panic window:** after a crash or watchdog reset, the ESP32 runs only the OTA
+- **Robust OTA:** rollback, watchdog, OTA window after a crash and safe mode, see
-  handler for 15 seconds before it starts the main code, with the LED blinking fast. A
+  [Keeping OTA alive](#keeping-ota-alive).
-  firmware that crashes on boot can still be replaced over the air.
 - **NTP time** with timezone `CET-1CEST,M3.5.0,M10.5.0/3` (Europe/Amsterdam). Until the
   time is synced, the display shows "Connecting...".
 - **Home Assistant** through MQTT (`home-assistant-integration` library) with two switches.
@@ -76,7 +111,8 @@ The times are in `logic/schedule.cpp`.
 
 `pio test -e test_esp32` runs [test/test_esp32](../../test/test_esp32/test_main.cpp) on
 the PC: the schedule boundaries, the power timer (including `millis()` overflow), the LED
-pattern and the display controller (power on/off, redraws, both Home Assistant switches).
+pattern, the boot policy (crash counting, safe mode, rollback timing) and the display
+controller (power on/off, redraws, both Home Assistant switches).
 
 ## Adding bitmaps
 

src/esp32/config.h

@@ -59,11 +59,28 @@ const uint32_t NANO_RESET_PULSE = 50;
 #define TIMEZONE "CET-1CEST,M3.5.0,M10.5.0/3"
 #define NTP_SERVER "pool.ntp.org"
 
-// After a crash or unexpected reset, only run OTA for this long before starting the
+// ---------------------------------------------------------------------------
-// main code, so a firmware that crashes at boot can still be replaced
+// Keeping OTA alive (see logic/boot_policy.h and system/firmware_guard.h)
+// ---------------------------------------------------------------------------
+
+// After a crash, only run OTA for this long before starting the main code
 const uint32_t OTA_PANIC_WINDOW = 15000;
+// This many crashes in a row: safe mode (only WiFi and OTA) for SAFE_MODE_TIME
+const uint8_t SAFE_MODE_CRASHES = 3;
+const uint32_t SAFE_MODE_TIME = 10UL * 60 * 1000;
+// A new OTA firmware must run this long with WiFi, or the previous firmware comes back
+const uint32_t FIRMWARE_VALID_TIME = 60UL * 1000;
+// Running this long without a crash clears the crash counter
+const uint32_t CRASH_FREE_TIME = 5UL * 60 * 1000;
+// loop() must not block longer than this, or the ESP32 restarts (counts as a crash)
+const uint32_t WATCHDOG_TIMEOUT_S = 30;
+// WiFi: first connection at boot, and restart when WiFi stays down this long
+const uint32_t WIFI_CONNECT_TIMEOUT = 30000;
+const uint32_t WIFI_RESTART_TIME = 10UL * 60 * 1000;
+
+#define FIRMWARE_VERSION "2.1.0"
 
 // Home Assistant device id (do not change: Home Assistant's entities are linked to it)
 const uint8_t HA_DEVICE_ID[] = {0x34, 0x86, 0x5d, 0x3a, 0x08, 0x28};
 #define HA_DEVICE_NAME "FlipDotDisplay"
-#define HA_SOFTWARE_VERSION "2.0.0"
+#define HA_SOFTWARE_VERSION FIRMWARE_VERSION

src/esp32/logic/boot_policy.cpp

@@ -0,0 +1,37 @@
+#include "logic/boot_policy.h"
+
+#include "config.h"
+
+BootDecision decideBoot(ResetKind reset, uint8_t crash_count)
+{
+	switch (reset)
+	{
+	case ResetKind::PowerOn:
+		return {BootMode::Normal, 0};
+	case ResetKind::Crash:
+		if (crash_count < 255)
+			crash_count++;
+		if (crash_count >= SAFE_MODE_CRASHES)
+			return {BootMode::SafeMode, crash_count};
+		return {BootMode::PanicWindow, crash_count};
+	case ResetKind::Software:
+	case ResetKind::Other:
+	default:
+		return {BootMode::Normal, crash_count};
+	}
+}
+
+uint8_t crashCountAfterSafeMode()
+{
+	return SAFE_MODE_CRASHES - 1;
+}
+
+bool shouldMarkFirmwareValid(uint32_t running_ms, bool wifi_connected)
+{
+	return running_ms >= FIRMWARE_VALID_TIME && wifi_connected;
+}
+
+bool shouldClearCrashCount(uint32_t running_ms)
+{
+	return running_ms >= CRASH_FREE_TIME;
+}

src/esp32/logic/boot_policy.h

@@ -0,0 +1,45 @@
+#pragma once
+
+#include <stdint.h>
+
+// Decides how the ESP32 starts after a reset, so OTA keeps working even when the firmware
+// crashes. Pure logic, tested in test/test_esp32.
+//
+//   Power on                 normal start, crash counter cleared
+//   Software reset           normal start (e.g. after an OTA update)
+//   Crash (panic, watchdog)  crash counter + 1, then:
+//                              < SAFE_MODE_CRASHES  OTA-only for OTA_PANIC_WINDOW, then normal
+//                              >= SAFE_MODE_CRASHES safe mode: OTA only for SAFE_MODE_TIME
+//   Other (brownout, ...)    normal start, counter kept
+
+enum class ResetKind
+{
+	PowerOn,
+	Software,
+	Crash,
+	Other,
+};
+
+enum class BootMode
+{
+	Normal,
+	PanicWindow, // OTA only for a short time, then normal
+	SafeMode,    // OTA only for a long time, then restart
+};
+
+struct BootDecision
+{
+	BootMode mode;
+	uint8_t crash_count;
+};
+
+BootDecision decideBoot(ResetKind reset, uint8_t crash_count);
+
+// Crash counter after safe mode ends: one more crash brings safe mode back
+uint8_t crashCountAfterSafeMode();
+
+// A new OTA firmware is accepted (no rollback) once it ran this long with WiFi
+bool shouldMarkFirmwareValid(uint32_t running_ms, bool wifi_connected);
+
+// Running this long without a crash clears the crash counter
+bool shouldClearCrashCount(uint32_t running_ms);

src/esp32/main.cpp

@@ -13,6 +13,7 @@
 #include "nano_link/nano_link.h"
 #include "net/home_assistant.h"
 #include "net/wifi_ota.h"
+#include "system/firmware_guard.h"
 
 NanoLink nano(Serial2, NANO_RX_PIN, NANO_TX_PIN, RELAY_PIN, NANO_RESET_PIN);
 BROSE9323 display(DISPLAY_WIDTH, DISPLAY_HEIGHT, PANEL_WIDTH);
@@ -30,12 +31,25 @@ void setup()
 {
 	Serial.begin(115200);
 	pinMode(LED_PIN, OUTPUT);
-	nano.begin();
+	nano.begin(); // display off until the main code runs
+
+	// Decided first, before anything that could crash
+	BootMode boot_mode = firmwareGuardBegin();
 
 	wifiBegin();
 	otaBegin();
+
+	if (boot_mode == BootMode::SafeMode)
+	{
+		// Crashed too often: only WiFi and OTA, then try the main code once more
+		otaOnlyMode(SAFE_MODE_TIME, LED_PIN);
+		firmwareGuardSafeModeEnded();
+		ESP.restart();
+	}
+	if (boot_mode == BootMode::PanicWindow)
+		otaOnlyMode(OTA_PANIC_WINDOW, LED_PIN);
+
 	flashBridge.begin();
-	otaPanicWindow(LED_PIN);
 
 	nano.on();
 	display.begin(&nano.stream());
@@ -45,11 +59,18 @@ void setup()
 
 	configTzTime(TIMEZONE, NTP_SERVER);
 	homeAssistantBegin(controller);
+
+	firmwareGuardStartWatchdog();
 }
 
 void loop()
 {
+	// OTA and the firmware guard come first, so they keep working whatever happens below
 	otaLoop();
+	firmwareGuardLoop(wifiConnected());
+	wifiWatchdog();
+
+	homeAssistantSetFirmwareStatus(firmwareGuardStatus());
 	homeAssistantLoop();
 
 	// While the Nano is being flashed, nothing else may talk to it

src/esp32/net/home_assistant.cpp

@@ -2,6 +2,7 @@
 
 #include <ArduinoHA.h>
 #include <WiFi.h>
+#include <string.h>
 
 #include "config.h"
 #include "credentials.h"
@@ -13,6 +14,10 @@ static HAMqtt mqtt(client, device);
 // Unique ids: do not change, Home Assistant's entities are linked to them
 static HASwitch clockSwitch("FlipDotDisplayClock");
 static HASwitch powerSwitch("FlipDotDisplayPower");
+static HASensor firmwareSensor("FlipDotDisplayFirmware");
+
+static char firmware_status[64] = "";
+static bool firmware_status_sent = false;
 
 static DisplayController *displayController = nullptr;
 
@@ -46,10 +51,25 @@ void homeAssistantBegin(DisplayController &controller)
 	powerSwitch.setCurrentState(controller.powerEnabled());
 	powerSwitch.onCommand(onPowerCommand);
 
+	firmwareSensor.setName("FlipDot Display Firmware");
+	firmwareSensor.setIcon("mdi:chip");
+	// The sensor doesn't resend its value on reconnect by itself
+	mqtt.onConnected([]() { firmware_status_sent = false; });
+
 	mqtt.begin(BROKER_ADDR, MQTT_USER, MQTT_PASSWORD);
 }
 
 void homeAssistantLoop()
 {
 	mqtt.loop();
+	if (!firmware_status_sent && firmware_status[0] && mqtt.isConnected())
+		firmware_status_sent = firmwareSensor.setValue(firmware_status);
+}
+
+void homeAssistantSetFirmwareStatus(const char *status)
+{
+	if (strcmp(status, firmware_status) == 0)
+		return;
+	strncpy(firmware_status, status, sizeof(firmware_status) - 1);
+	firmware_status_sent = false;
 }

src/esp32/net/home_assistant.h

@@ -2,8 +2,12 @@
 
 #include "logic/display_controller.h"
 
-// Home Assistant integration over MQTT (with discovery). Two switches:
+// Home Assistant integration over MQTT (with discovery):
-//   FlipDot Display Clock  on = clock, off = Nintendo logo
+//   FlipDot Display Clock     switch: on = clock, off = Nintendo logo
-//   FlipDot Display Power  off = clear the display and switch it off
+//   FlipDot Display Power     switch: off = clear the display and switch it off
+//   FlipDot Display Firmware  sensor: version and health, e.g. "2.1.0, ok, 0 crashes"
 void homeAssistantBegin(DisplayController &controller);
 void homeAssistantLoop();
+
+// Publish the firmware status (only sent when it changes, and again after a reconnect)
+void homeAssistantSetFirmwareStatus(const char *status);

src/esp32/net/wifi_ota.cpp

@@ -2,24 +2,26 @@
 
 #include <ArduinoOTA.h>
 #include <WiFi.h>
-#include <rom/rtc.h>
 
 #include "config.h"
 #include "credentials.h"
+#include "system/firmware_guard.h"
+
+static uint32_t wifi_lost_at = 0;
+static bool wifi_lost = false;
 
 void wifiBegin()
 {
 	WiFi.setHostname(HOSTNAME);
 	WiFi.mode(WIFI_STA);
+	WiFi.setAutoReconnect(true);
 	WiFi.begin(WIFI_SSID, WIFI_PASSWORD);
 	Serial.println("Connecting to WiFi");
-	if (WiFi.waitForConnectResult() != WL_CONNECTED)
+	if (WiFi.waitForConnectResult(WIFI_CONNECT_TIMEOUT) != WL_CONNECTED)
 	{
 		Serial.println("WiFi connection failed, restarting");
-		delay(5000);
 		ESP.restart();
 	}
-	// Later disconnects are handled by the WiFi driver's auto reconnect
 	Serial.print("Connected, IP ");
 	Serial.println(WiFi.localIP());
 }
@@ -29,12 +31,38 @@ bool wifiConnected()
 	return WiFi.status() == WL_CONNECTED;
 }
 
+void wifiWatchdog()
+{
+	if (wifiConnected())
+	{
+		wifi_lost = false;
+		return;
+	}
+	if (!wifi_lost)
+	{
+		wifi_lost = true;
+		wifi_lost_at = millis();
+	}
+	else if (millis() - wifi_lost_at >= WIFI_RESTART_TIME)
+	{
+		Serial.println("WiFi down for too long, restarting");
+		ESP.restart();
+	}
+}
+
 void otaBegin()
 {
 	ArduinoOTA.setHostname(HOSTNAME);
-	ArduinoOTA.onStart([]() { Serial.println("OTA update started"); });
+	ArduinoOTA.onStart([]() {
+		// The update blocks loop() until it is done
+		firmwareGuardStopWatchdog();
+		Serial.println("OTA update started");
+	});
 	ArduinoOTA.onEnd([]() { Serial.println("OTA update done"); });
-	ArduinoOTA.onError([](ota_error_t error) { Serial.printf("OTA error %u\n", error); });
+	ArduinoOTA.onError([](ota_error_t error) {
+		Serial.printf("OTA error %u, restarting\n", error);
+		ESP.restart();
+	});
 	ArduinoOTA.begin();
 }
 
@@ -43,27 +71,15 @@ void otaLoop()
 	ArduinoOTA.handle();
 }
 
-// A normal start is a power on of CPU 0 or a start of CPU 1 by CPU 0
+void otaOnlyMode(uint32_t duration, int led_pin)
-static bool unexpectedReset()
 {
-	RESET_REASON cpu0 = rtc_get_reset_reason(0);
+	Serial.printf("OTA only for %lu s\n", (unsigned long)(duration / 1000));
-	RESET_REASON cpu1 = rtc_get_reset_reason(1);
-	Serial.printf("Reset reason: CPU0 %d, CPU1 %d\n", cpu0, cpu1);
-	return cpu0 != POWERON_RESET && cpu1 != EXT_CPU_RESET;
-}
-
-void otaPanicWindow(int led_pin)
-{
-	if (!unexpectedReset())
-		return;
-
-	Serial.println("Unexpected reset: waiting for an OTA update");
 	uint32_t start = millis();
-	while (millis() - start < OTA_PANIC_WINDOW)
+	while (millis() - start < duration)
 	{
 		ArduinoOTA.handle();
-		// fast blink
+		wifiWatchdog();
-		digitalWrite(led_pin, (millis() / 50) % 2);
+		digitalWrite(led_pin, (millis() / 50) % 2); // fast blink
 		delay(10);
 	}
 	digitalWrite(led_pin, LOW);

src/esp32/net/wifi_ota.h

@@ -1,14 +1,19 @@
 #pragma once
 
+#include <stdint.h>
+
 // Connect to WiFi. Restarts the ESP32 when the first connection fails.
 void wifiBegin();
 
 bool wifiConnected();
 
+// Restarts the ESP32 when WiFi has been down for WIFI_RESTART_TIME (a stuck WiFi stack
+// would otherwise make OTA unreachable). Call every loop.
+void wifiWatchdog();
+
 // Over-the-air updates of the ESP32 firmware (pio run -e esp32 -t upload)
 void otaBegin();
 void otaLoop();
 
-// After a crash or unexpected reset, run only OTA for OTA_PANIC_WINDOW, so a firmware that
+// Run only WiFi and OTA for `duration` ms (after a crash / safe mode), LED blinking fast
-// crashes at boot can still be replaced. Does nothing after a normal power on.
+void otaOnlyMode(uint32_t duration, int led_pin);
-void otaPanicWindow(int led_pin);

src/esp32/system/firmware_guard.cpp

@@ -0,0 +1,140 @@
+#include "system/firmware_guard.h"
+
+#include <Arduino.h>
+#include <esp_ota_ops.h>
+#include <esp_system.h>
+#include <esp_task_wdt.h>
+
+#include "config.h"
+
+// Survives crashes and software resets, not a power loss
+static const uint32_t RTC_MAGIC = 0xF11D0C0A;
+RTC_NOINIT_ATTR static uint32_t rtc_magic;
+RTC_NOINIT_ATTR static uint8_t rtc_crash_count;
+
+static uint32_t main_start = 0; // millis() when the main code started
+static bool watchdog_running = false;
+static bool firmware_pending = false; // new OTA firmware, not yet marked as good
+static bool crash_count_cleared = false;
+static char status[64];
+
+// The Arduino core marks a new OTA firmware as good right at boot. Returning true here
+// leaves that to us, so the bootloader can roll back a firmware that fails early.
+extern "C" bool verifyRollbackLater()
+{
+	return true;
+}
+
+static ResetKind resetKind()
+{
+	switch (esp_reset_reason())
+	{
+	case ESP_RST_POWERON:
+		return ResetKind::PowerOn;
+	case ESP_RST_SW:
+		return ResetKind::Software;
+	case ESP_RST_PANIC:
+	case ESP_RST_INT_WDT:
+	case ESP_RST_TASK_WDT:
+	case ESP_RST_WDT:
+		return ResetKind::Crash;
+	default:
+		return ResetKind::Other;
+	}
+}
+
+static void setCrashCount(uint8_t count)
+{
+	rtc_magic = RTC_MAGIC;
+	rtc_crash_count = count;
+}
+
+static const char *firmwareState()
+{
+	esp_ota_img_states_t state;
+	if (esp_ota_get_state_partition(esp_ota_get_running_partition(), &state) != ESP_OK)
+		return "usb";
+	switch (state)
+	{
+	case ESP_OTA_IMG_PENDING_VERIFY:
+		return "testing";
+	case ESP_OTA_IMG_VALID:
+		return "ok";
+	case ESP_OTA_IMG_NEW:
+		// The bootloader turns NEW into PENDING_VERIFY when it supports rollback
+		return "ok, no rollback in bootloader";
+	default:
+		return "usb";
+	}
+}
+
+static void updateStatus()
+{
+	snprintf(status, sizeof(status), "%s, %s, %u crashes", FIRMWARE_VERSION, firmwareState(),
+			 rtc_magic == RTC_MAGIC ? rtc_crash_count : 0);
+}
+
+BootMode firmwareGuardBegin()
+{
+	uint8_t count = rtc_magic == RTC_MAGIC ? rtc_crash_count : 0;
+	BootDecision decision = decideBoot(resetKind(), count);
+	setCrashCount(decision.crash_count);
+
+	esp_ota_img_states_t state;
+	firmware_pending = esp_ota_get_state_partition(esp_ota_get_running_partition(), &state) == ESP_OK &&
+					   state == ESP_OTA_IMG_PENDING_VERIFY;
+
+	updateStatus();
+	Serial.printf("Firmware %s, reset reason %d, crash count %u\n", status, esp_reset_reason(),
+				  decision.crash_count);
+	return decision.mode;
+}
+
+void firmwareGuardSafeModeEnded()
+{
+	setCrashCount(crashCountAfterSafeMode());
+}
+
+void firmwareGuardStartWatchdog()
+{
+	// Reconfigures the watchdog the ESP-IDF already started (default 5 s)
+	esp_task_wdt_init(WATCHDOG_TIMEOUT_S, true);
+	esp_task_wdt_add(NULL);
+	watchdog_running = true;
+	main_start = millis();
+}
+
+void firmwareGuardStopWatchdog()
+{
+	if (watchdog_running)
+		esp_task_wdt_delete(NULL);
+	watchdog_running = false;
+}
+
+void firmwareGuardLoop(bool wifi_connected)
+{
+	if (watchdog_running)
+		esp_task_wdt_reset();
+
+	uint32_t running = millis() - main_start;
+
+	if (firmware_pending && shouldMarkFirmwareValid(running, wifi_connected))
+	{
+		esp_ota_mark_app_valid_cancel_rollback();
+		firmware_pending = false;
+		updateStatus();
+		Serial.println("New firmware marked as good");
+	}
+
+	if (!crash_count_cleared && shouldClearCrashCount(running))
+	{
+		setCrashCount(0);
+		crash_count_cleared = true;
+		updateStatus();
+	}
+}
+
+const char *firmwareGuardStatus()
+{
+	return status;
+}

src/esp32/system/firmware_guard.h

@@ -0,0 +1,32 @@
+#pragma once
+
+#include "logic/boot_policy.h"
+
+// Keeps the ESP32 reachable over OTA, whatever the firmware does:
+//
+// - Rollback: a new OTA firmware is only marked as good after it ran FIRMWARE_VALID_TIME
+//   with WiFi. If it crashes, hangs or restarts before that, the bootloader starts the
+//   previous firmware again.
+// - Watchdog: loop() must keep running; a hang longer than WATCHDOG_TIMEOUT_S restarts
+//   the ESP32 (and counts as a crash).
+// - Crash counter (kept in RTC memory over resets): after a crash the ESP32 first runs
+//   only OTA for a while, after SAFE_MODE_CRASHES crashes in a row it goes to safe mode.
+//   The decisions are in logic/boot_policy.
+
+// Call first in setup(): reads the reset reason and updates the crash counter
+BootMode firmwareGuardBegin();
+
+// Safe mode is over: allow one more try of the main code
+void firmwareGuardSafeModeEnded();
+
+// Start the watchdog for loop(), at the end of setup()
+void firmwareGuardStartWatchdog();
+
+// Stop the watchdog, e.g. during an OTA update (the update blocks loop())
+void firmwareGuardStopWatchdog();
+
+// Call every loop(): feeds the watchdog, marks a new firmware as good, clears the counter
+void firmwareGuardLoop(bool wifi_connected);
+
+// Short status for Home Assistant, e.g. "2.1.0, ok, 0 crashes"
+const char *firmwareGuardStatus();

test/test_esp32/test_main.cpp

@@ -11,6 +11,7 @@
 #include <vector>
 
 #include "config.h"
+#include "logic/boot_policy.h"
 #include "logic/display_controller.h"
 #include "logic/power_timer.h"
 #include "logic/schedule.h"
@@ -171,6 +172,81 @@ void test_timer_across_millis_overflow()
 	TEST_ASSERT_TRUE(t.expired(0x00000100u));
 }
 
+// ---------------------------------------------------------------------------
+// Boot policy (keeping OTA alive)
+// ---------------------------------------------------------------------------
+
+void test_power_on_starts_normally_and_clears_crashes()
+{
+	BootDecision d = decideBoot(ResetKind::PowerOn, 2);
+	TEST_ASSERT_TRUE(d.mode == BootMode::Normal);
+	TEST_ASSERT_EQUAL(0, d.crash_count);
+}
+
+void test_software_reset_starts_normally_and_keeps_count()
+{
+	// e.g. the restart after an OTA update
+	BootDecision d = decideBoot(ResetKind::Software, 1);
+	TEST_ASSERT_TRUE(d.mode == BootMode::Normal);
+	TEST_ASSERT_EQUAL(1, d.crash_count);
+}
+
+void test_other_reset_starts_normally_and_keeps_count()
+{
+	BootDecision d = decideBoot(ResetKind::Other, 2);
+	TEST_ASSERT_TRUE(d.mode == BootMode::Normal);
+	TEST_ASSERT_EQUAL(2, d.crash_count);
+}
+
+void test_first_crashes_give_a_panic_window()
+{
+	BootDecision d = decideBoot(ResetKind::Crash, 0);
+	TEST_ASSERT_TRUE(d.mode == BootMode::PanicWindow);
+	TEST_ASSERT_EQUAL(1, d.crash_count);
+	d = decideBoot(ResetKind::Crash, d.crash_count);
+	TEST_ASSERT_TRUE(d.mode == BootMode::PanicWindow);
+	TEST_ASSERT_EQUAL(2, d.crash_count);
+}
+
+void test_repeated_crashes_give_safe_mode()
+{
+	uint8_t count = 0;
+	BootDecision d = {BootMode::Normal, 0};
+	for (int i = 0; i < SAFE_MODE_CRASHES; i++)
+	{
+		d = decideBoot(ResetKind::Crash, count);
+		count = d.crash_count;
+	}
+	TEST_ASSERT_TRUE(d.mode == BootMode::SafeMode);
+}
+
+void test_after_safe_mode_one_more_crash_gives_safe_mode_again()
+{
+	uint8_t count = crashCountAfterSafeMode();
+	TEST_ASSERT_TRUE(decideBoot(ResetKind::Software, count).mode == BootMode::Normal);
+	TEST_ASSERT_TRUE(decideBoot(ResetKind::Crash, count).mode == BootMode::SafeMode);
+}
+
+void test_crash_count_does_not_overflow()
+{
+	BootDecision d = decideBoot(ResetKind::Crash, 255);
+	TEST_ASSERT_EQUAL(255, d.crash_count);
+	TEST_ASSERT_TRUE(d.mode == BootMode::SafeMode);
+}
+
+void test_new_firmware_needs_time_and_wifi_to_be_accepted()
+{
+	TEST_ASSERT_FALSE(shouldMarkFirmwareValid(FIRMWARE_VALID_TIME - 1, true));
+	TEST_ASSERT_FALSE(shouldMarkFirmwareValid(FIRMWARE_VALID_TIME, false));
+	TEST_ASSERT_TRUE(shouldMarkFirmwareValid(FIRMWARE_VALID_TIME, true));
+}
+
+void test_crash_count_cleared_after_running_long_enough()
+{
+	TEST_ASSERT_FALSE(shouldClearCrashCount(CRASH_FREE_TIME - 1));
+	TEST_ASSERT_TRUE(shouldClearCrashCount(CRASH_FREE_TIME));
+}
+
 // ---------------------------------------------------------------------------
 // Status LED
 // ---------------------------------------------------------------------------
@@ -348,6 +424,16 @@ int main()
 	RUN_TEST(test_timer_cancel);
 	RUN_TEST(test_timer_across_millis_overflow);
 
+	RUN_TEST(test_power_on_starts_normally_and_clears_crashes);
+	RUN_TEST(test_software_reset_starts_normally_and_keeps_count);
+	RUN_TEST(test_other_reset_starts_normally_and_keeps_count);
+	RUN_TEST(test_first_crashes_give_a_panic_window);
+	RUN_TEST(test_repeated_crashes_give_safe_mode);
+	RUN_TEST(test_after_safe_mode_one_more_crash_gives_safe_mode_again);
+	RUN_TEST(test_crash_count_does_not_overflow);
+	RUN_TEST(test_new_firmware_needs_time_and_wifi_to_be_accepted);
+	RUN_TEST(test_crash_count_cleared_after_running_long_enough);
+
 	RUN_TEST(test_led_off_while_display_off);
 	RUN_TEST(test_led_blinks_slow_with_wifi_and_fast_without);