From ddbdf7e65ff16a188069641fa3723f303e916a1c Mon Sep 17 00:00:00 2001 From: MangaValk Date: Tue, 13 Oct 2026 01:05:06 +0200 Subject: [PATCH] Keep ESP32 OTA reachable whatever the firmware does - Rollback: override verifyRollbackLater() so a new OTA image stays pending until it ran 60 s with WiFi; any earlier restart rolls back (the Arduino core marked every image valid at boot before) - Task watchdog (30 s) on loop(), paused during OTA uploads - Crash counter in RTC memory: OTA-only window after a crash, 10 min safe mode (WiFi + OTA only) after 3 crashes in a row - Restart when WiFi stays down for 10 min - Home Assistant sensor 'FlipDot Display Firmware' with version/state - logic/boot_policy with tests (60 tests total) Co-Authored-By: Claude Opus 5.5 --- CLAUDE.md | 15 +++ README.md | 7 +- src/esp32/README.md | 46 ++++++++- src/esp32/config.h | 23 ++++- src/esp32/logic/boot_policy.cpp | 37 ++++++++ src/esp32/logic/boot_policy.h | 45 +++++++++ src/esp32/main.cpp | 25 ++++- src/esp32/net/home_assistant.cpp | 20 ++++ src/esp32/net/home_assistant.h | 10 +- src/esp32/net/wifi_ota.cpp | 62 +++++++----- src/esp32/net/wifi_ota.h | 11 ++- src/esp32/system/firmware_guard.cpp | 140 ++++++++++++++++++++++++++++ src/esp32/system/firmware_guard.h | 32 +++++++ test/test_esp32/test_main.cpp | 86 +++++++++++++++++ 14 files changed, 517 insertions(+), 42 deletions(-) create mode 100644 src/esp32/logic/boot_policy.cpp create mode 100644 src/esp32/logic/boot_policy.h create mode 100644 src/esp32/system/firmware_guard.cpp create mode 100644 src/esp32/system/firmware_guard.h diff --git a/CLAUDE.md b/CLAUDE.md index 111e420..75b0e93 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -91,6 +91,21 @@ See README.md for the overview. The unique ids `FlipDotDisplayClock` / `FlipDotDisplayPower` and `HA_DEVICE_ID` must stay the same, or Home Assistant creates new entities. +## Keeping ESP32 OTA alive (most important requirement) + +- The user never wants to need USB for the ESP32 again. Any change must keep OTA working, + also when the new code crashes or hangs. +- `system/firmware_guard.cpp` overrides the Arduino core's weak `verifyRollbackLater()` + (C linkage) to return true. A new OTA image stays `PENDING_VERIFY` until it has run + `FIRMWARE_VALID_TIME` with WiFi. Any restart before that makes the bootloader roll back. + Check with `nm` that the strong symbol is linked after toolchain or core updates. +- The task watchdog (`WATCHDOG_TIMEOUT_S`) covers `loop()`. ArduinoOTA's `onStart` stops + it, because an upload blocks `loop()`. Never block `loop()` for 30 s or longer. +- The crash counter lives in `RTC_NOINIT_ATTR` memory. Decisions are in + `logic/boot_policy.*` (tested). Safe mode runs only WiFi + OTA. +- Keep `otaLoop()` and `firmwareGuardLoop()` at the top of `loop()`, before anything that + can return early or block. + ## Git - Remote: https://mangavalk.com/mangavalk/Flipdot-display.git, branch `main`. diff --git a/README.md b/README.md index 969d650..ee53d67 100644 --- a/README.md +++ b/README.md @@ -170,9 +170,10 @@ The ESP32 talks to the Nano on **UART2** (`Serial2`, 115200 baud). The ESP32's o ## How it works -1. **ESP32 boot.** The ESP32 connects to WiFi and starts ArduinoOTA. After a crash or - unexpected reset it stays in an *OTA panic window* for 15 seconds, so you can flash a - fix before the main code runs again. It then switches the display relay on, waits for +1. **ESP32 boot.** The ESP32 connects to WiFi and starts ArduinoOTA. After a crash it + runs only OTA for 15 seconds, and after repeated crashes it goes to safe mode. Bad OTA + firmware is rolled back automatically, see + [Keeping OTA alive](src/esp32/README.md#keeping-ota-alive). It then switches the display relay on, waits for the Nano's `READY` message (at most 3 s), shows "Connecting...", starts NTP and connects to MQTT. 2. **Clock.** Once the time is synced, the ESP32 sends a new frame whenever the screen diff --git a/src/esp32/README.md b/src/esp32/README.md index 8eba9e4..ff1bff4 100644 --- a/src/esp32/README.md +++ b/src/esp32/README.md @@ -19,6 +19,9 @@ debug output. Wiring is described in the [main README](../../README.md#wiring-be | `logic/schedule.*` | When the clock is shown (night, clock, standby) | | `logic/power_timer.*` | Countdown to switching the display off | | `logic/status_led.h` | Blink pattern of the built-in LED | +| `logic/boot_policy.*` | How to start after a reset: normal, OTA-only window, safe mode | +| **`system/`** | **Keeping OTA alive** | +| `system/firmware_guard.*` | Rollback of bad OTA firmware, watchdog, crash counter, status for Home Assistant | | **`display/`** | **What is drawn** | | `display/frames.*` | The screens: clock, logo, blank, text (`FlipdotFrames`) | | `display/bitmaps.h` | 1-bit bitmaps (Pikachu, thought bubble, Nintendo logo, ...) | @@ -26,22 +29,54 @@ debug output. Wiring is described in the [main README](../../README.md#wiring-be | `nano_link/nano_link.*` | Relay and serial link: power on (waits for `READY`), power off, bootloader start (`NanoLink`) | | `nano_link/nano_flash_bridge.*` | WiFi to serial bridge for flashing the Nano over WiFi (TCP port 2323) | | **`net/`** | **Network** | -| `net/wifi_ota.*` | WiFi, ESP32 OTA updates and the OTA panic window | +| `net/wifi_ota.*` | WiFi (with a restart when it stays down), ESP32 OTA updates, OTA-only mode | | `net/home_assistant.*` | MQTT and the two Home Assistant switches | `DisplayController` only talks to two small interfaces, `DisplayPower` (implemented by `NanoLink`) and `Frames` (implemented by `FlipdotFrames`). The tests replace both with fakes. +## Keeping OTA alive + +The ESP32 must stay reachable over OTA whatever the firmware does, so it never needs USB +again. Several layers take care of that: + +| Problem | What happens | +|---------|--------------| +| A new OTA firmware crashes, hangs or can't connect to WiFi | **Rollback.** A new firmware is only marked as good after it has run for 60 s with WiFi. If it restarts before that, the bootloader starts the previous firmware again. | +| The firmware hangs in `loop()` | **Watchdog.** If `loop()` doesn't run for 30 s, the ESP32 restarts. That counts as a crash. | +| The firmware crashes once or twice | **OTA window.** After a crash the ESP32 runs only WiFi and OTA for 15 s (LED blinks fast), then starts the main code again. | +| The firmware keeps crashing (3 times in a row) | **Safe mode.** Only WiFi and OTA run, for 10 minutes. Then the main code gets one more try; one more crash brings safe mode back. Running 5 minutes without a crash clears the counter. | +| WiFi stays down | After 10 minutes without WiFi, the ESP32 restarts, in case the WiFi stack is stuck. | + +So in every case you can flash a fix with `pio run -e esp32 -t upload`, at the latest +after waiting for a safe-mode cycle. + +The **FlipDot Display Firmware** sensor in Home Assistant shows the state, e.g. +`2.1.0, ok, 0 crashes`: + +| State | Meaning | +|-------|---------| +| `testing` | New OTA firmware, not yet marked as good (first 60 s) | +| `ok` | Firmware marked as good | +| `ok, no rollback in bootloader` | The bootloader on the ESP32 can't roll back. The other layers still work. | +| `usb` | Flashed over USB, so there is no previous firmware to roll back to | + +Rules for changing the code: + +- Keep `otaLoop()` and `firmwareGuardLoop()` at the top of `loop()`. +- Never block `loop()` for 30 s or longer. +- Change `FIRMWARE_VERSION` in `config.h` with each release, so the sensor shows which + version is running. + ## Features - **Frames only when something changes:** a new frame is sent when the minute changes, the screen changes (clock, logo, blank) or the Nano was just powered on. The serial line is quiet in between, which also gives the Nano time for its periodic full redraw. - **OTA updates** through ArduinoOTA. Hostname: `FlipDotDisplay`. -- **OTA panic window:** after a crash or watchdog reset, the ESP32 runs only the OTA - handler for 15 seconds before it starts the main code, with the LED blinking fast. A - firmware that crashes on boot can still be replaced over the air. +- **Robust OTA:** rollback, watchdog, OTA window after a crash and safe mode, see + [Keeping OTA alive](#keeping-ota-alive). - **NTP time** with timezone `CET-1CEST,M3.5.0,M10.5.0/3` (Europe/Amsterdam). Until the time is synced, the display shows "Connecting...". - **Home Assistant** through MQTT (`home-assistant-integration` library) with two switches. @@ -76,7 +111,8 @@ The times are in `logic/schedule.cpp`. `pio test -e test_esp32` runs [test/test_esp32](../../test/test_esp32/test_main.cpp) on the PC: the schedule boundaries, the power timer (including `millis()` overflow), the LED -pattern and the display controller (power on/off, redraws, both Home Assistant switches). +pattern, the boot policy (crash counting, safe mode, rollback timing) and the display +controller (power on/off, redraws, both Home Assistant switches). ## Adding bitmaps diff --git a/src/esp32/config.h b/src/esp32/config.h index 43eb1d9..2545644 100644 --- a/src/esp32/config.h +++ b/src/esp32/config.h @@ -59,11 +59,28 @@ const uint32_t NANO_RESET_PULSE = 50; #define TIMEZONE "CET-1CEST,M3.5.0,M10.5.0/3" #define NTP_SERVER "pool.ntp.org" -// After a crash or unexpected reset, only run OTA for this long before starting the -// main code, so a firmware that crashes at boot can still be replaced +// --------------------------------------------------------------------------- +// Keeping OTA alive (see logic/boot_policy.h and system/firmware_guard.h) +// --------------------------------------------------------------------------- + +// After a crash, only run OTA for this long before starting the main code const uint32_t OTA_PANIC_WINDOW = 15000; +// This many crashes in a row: safe mode (only WiFi and OTA) for SAFE_MODE_TIME +const uint8_t SAFE_MODE_CRASHES = 3; +const uint32_t SAFE_MODE_TIME = 10UL * 60 * 1000; +// A new OTA firmware must run this long with WiFi, or the previous firmware comes back +const uint32_t FIRMWARE_VALID_TIME = 60UL * 1000; +// Running this long without a crash clears the crash counter +const uint32_t CRASH_FREE_TIME = 5UL * 60 * 1000; +// loop() must not block longer than this, or the ESP32 restarts (counts as a crash) +const uint32_t WATCHDOG_TIMEOUT_S = 30; +// WiFi: first connection at boot, and restart when WiFi stays down this long +const uint32_t WIFI_CONNECT_TIMEOUT = 30000; +const uint32_t WIFI_RESTART_TIME = 10UL * 60 * 1000; + +#define FIRMWARE_VERSION "2.1.0" // Home Assistant device id (do not change: Home Assistant's entities are linked to it) const uint8_t HA_DEVICE_ID[] = {0x34, 0x86, 0x5d, 0x3a, 0x08, 0x28}; #define HA_DEVICE_NAME "FlipDotDisplay" -#define HA_SOFTWARE_VERSION "2.0.0" +#define HA_SOFTWARE_VERSION FIRMWARE_VERSION diff --git a/src/esp32/logic/boot_policy.cpp b/src/esp32/logic/boot_policy.cpp new file mode 100644 index 0000000..21a824c --- /dev/null +++ b/src/esp32/logic/boot_policy.cpp @@ -0,0 +1,37 @@ +#include "logic/boot_policy.h" + +#include "config.h" + +BootDecision decideBoot(ResetKind reset, uint8_t crash_count) +{ + switch (reset) + { + case ResetKind::PowerOn: + return {BootMode::Normal, 0}; + case ResetKind::Crash: + if (crash_count < 255) + crash_count++; + if (crash_count >= SAFE_MODE_CRASHES) + return {BootMode::SafeMode, crash_count}; + return {BootMode::PanicWindow, crash_count}; + case ResetKind::Software: + case ResetKind::Other: + default: + return {BootMode::Normal, crash_count}; + } +} + +uint8_t crashCountAfterSafeMode() +{ + return SAFE_MODE_CRASHES - 1; +} + +bool shouldMarkFirmwareValid(uint32_t running_ms, bool wifi_connected) +{ + return running_ms >= FIRMWARE_VALID_TIME && wifi_connected; +} + +bool shouldClearCrashCount(uint32_t running_ms) +{ + return running_ms >= CRASH_FREE_TIME; +} diff --git a/src/esp32/logic/boot_policy.h b/src/esp32/logic/boot_policy.h new file mode 100644 index 0000000..b2b17e7 --- /dev/null +++ b/src/esp32/logic/boot_policy.h @@ -0,0 +1,45 @@ +#pragma once + +#include + +// Decides how the ESP32 starts after a reset, so OTA keeps working even when the firmware +// crashes. Pure logic, tested in test/test_esp32. +// +// Power on normal start, crash counter cleared +// Software reset normal start (e.g. after an OTA update) +// Crash (panic, watchdog) crash counter + 1, then: +// < SAFE_MODE_CRASHES OTA-only for OTA_PANIC_WINDOW, then normal +// >= SAFE_MODE_CRASHES safe mode: OTA only for SAFE_MODE_TIME +// Other (brownout, ...) normal start, counter kept + +enum class ResetKind +{ + PowerOn, + Software, + Crash, + Other, +}; + +enum class BootMode +{ + Normal, + PanicWindow, // OTA only for a short time, then normal + SafeMode, // OTA only for a long time, then restart +}; + +struct BootDecision +{ + BootMode mode; + uint8_t crash_count; +}; + +BootDecision decideBoot(ResetKind reset, uint8_t crash_count); + +// Crash counter after safe mode ends: one more crash brings safe mode back +uint8_t crashCountAfterSafeMode(); + +// A new OTA firmware is accepted (no rollback) once it ran this long with WiFi +bool shouldMarkFirmwareValid(uint32_t running_ms, bool wifi_connected); + +// Running this long without a crash clears the crash counter +bool shouldClearCrashCount(uint32_t running_ms); diff --git a/src/esp32/main.cpp b/src/esp32/main.cpp index c75dc1b..e0deb66 100644 --- a/src/esp32/main.cpp +++ b/src/esp32/main.cpp @@ -13,6 +13,7 @@ #include "nano_link/nano_link.h" #include "net/home_assistant.h" #include "net/wifi_ota.h" +#include "system/firmware_guard.h" NanoLink nano(Serial2, NANO_RX_PIN, NANO_TX_PIN, RELAY_PIN, NANO_RESET_PIN); BROSE9323 display(DISPLAY_WIDTH, DISPLAY_HEIGHT, PANEL_WIDTH); @@ -30,12 +31,25 @@ void setup() { Serial.begin(115200); pinMode(LED_PIN, OUTPUT); - nano.begin(); + nano.begin(); // display off until the main code runs + + // Decided first, before anything that could crash + BootMode boot_mode = firmwareGuardBegin(); wifiBegin(); otaBegin(); + + if (boot_mode == BootMode::SafeMode) + { + // Crashed too often: only WiFi and OTA, then try the main code once more + otaOnlyMode(SAFE_MODE_TIME, LED_PIN); + firmwareGuardSafeModeEnded(); + ESP.restart(); + } + if (boot_mode == BootMode::PanicWindow) + otaOnlyMode(OTA_PANIC_WINDOW, LED_PIN); + flashBridge.begin(); - otaPanicWindow(LED_PIN); nano.on(); display.begin(&nano.stream()); @@ -45,11 +59,18 @@ void setup() configTzTime(TIMEZONE, NTP_SERVER); homeAssistantBegin(controller); + + firmwareGuardStartWatchdog(); } void loop() { + // OTA and the firmware guard come first, so they keep working whatever happens below otaLoop(); + firmwareGuardLoop(wifiConnected()); + wifiWatchdog(); + + homeAssistantSetFirmwareStatus(firmwareGuardStatus()); homeAssistantLoop(); // While the Nano is being flashed, nothing else may talk to it diff --git a/src/esp32/net/home_assistant.cpp b/src/esp32/net/home_assistant.cpp index cbe6c15..df136b9 100644 --- a/src/esp32/net/home_assistant.cpp +++ b/src/esp32/net/home_assistant.cpp @@ -2,6 +2,7 @@ #include #include +#include #include "config.h" #include "credentials.h" @@ -13,6 +14,10 @@ static HAMqtt mqtt(client, device); // Unique ids: do not change, Home Assistant's entities are linked to them static HASwitch clockSwitch("FlipDotDisplayClock"); static HASwitch powerSwitch("FlipDotDisplayPower"); +static HASensor firmwareSensor("FlipDotDisplayFirmware"); + +static char firmware_status[64] = ""; +static bool firmware_status_sent = false; static DisplayController *displayController = nullptr; @@ -46,10 +51,25 @@ void homeAssistantBegin(DisplayController &controller) powerSwitch.setCurrentState(controller.powerEnabled()); powerSwitch.onCommand(onPowerCommand); + firmwareSensor.setName("FlipDot Display Firmware"); + firmwareSensor.setIcon("mdi:chip"); + // The sensor doesn't resend its value on reconnect by itself + mqtt.onConnected([]() { firmware_status_sent = false; }); + mqtt.begin(BROKER_ADDR, MQTT_USER, MQTT_PASSWORD); } void homeAssistantLoop() { mqtt.loop(); + if (!firmware_status_sent && firmware_status[0] && mqtt.isConnected()) + firmware_status_sent = firmwareSensor.setValue(firmware_status); +} + +void homeAssistantSetFirmwareStatus(const char *status) +{ + if (strcmp(status, firmware_status) == 0) + return; + strncpy(firmware_status, status, sizeof(firmware_status) - 1); + firmware_status_sent = false; } diff --git a/src/esp32/net/home_assistant.h b/src/esp32/net/home_assistant.h index 9a6130d..c2ce8aa 100644 --- a/src/esp32/net/home_assistant.h +++ b/src/esp32/net/home_assistant.h @@ -2,8 +2,12 @@ #include "logic/display_controller.h" -// Home Assistant integration over MQTT (with discovery). Two switches: -// FlipDot Display Clock on = clock, off = Nintendo logo -// FlipDot Display Power off = clear the display and switch it off +// Home Assistant integration over MQTT (with discovery): +// FlipDot Display Clock switch: on = clock, off = Nintendo logo +// FlipDot Display Power switch: off = clear the display and switch it off +// FlipDot Display Firmware sensor: version and health, e.g. "2.1.0, ok, 0 crashes" void homeAssistantBegin(DisplayController &controller); void homeAssistantLoop(); + +// Publish the firmware status (only sent when it changes, and again after a reconnect) +void homeAssistantSetFirmwareStatus(const char *status); diff --git a/src/esp32/net/wifi_ota.cpp b/src/esp32/net/wifi_ota.cpp index 2d85258..ca0fc42 100644 --- a/src/esp32/net/wifi_ota.cpp +++ b/src/esp32/net/wifi_ota.cpp @@ -2,24 +2,26 @@ #include #include -#include #include "config.h" #include "credentials.h" +#include "system/firmware_guard.h" + +static uint32_t wifi_lost_at = 0; +static bool wifi_lost = false; void wifiBegin() { WiFi.setHostname(HOSTNAME); WiFi.mode(WIFI_STA); + WiFi.setAutoReconnect(true); WiFi.begin(WIFI_SSID, WIFI_PASSWORD); Serial.println("Connecting to WiFi"); - if (WiFi.waitForConnectResult() != WL_CONNECTED) + if (WiFi.waitForConnectResult(WIFI_CONNECT_TIMEOUT) != WL_CONNECTED) { Serial.println("WiFi connection failed, restarting"); - delay(5000); ESP.restart(); } - // Later disconnects are handled by the WiFi driver's auto reconnect Serial.print("Connected, IP "); Serial.println(WiFi.localIP()); } @@ -29,12 +31,38 @@ bool wifiConnected() return WiFi.status() == WL_CONNECTED; } +void wifiWatchdog() +{ + if (wifiConnected()) + { + wifi_lost = false; + return; + } + if (!wifi_lost) + { + wifi_lost = true; + wifi_lost_at = millis(); + } + else if (millis() - wifi_lost_at >= WIFI_RESTART_TIME) + { + Serial.println("WiFi down for too long, restarting"); + ESP.restart(); + } +} + void otaBegin() { ArduinoOTA.setHostname(HOSTNAME); - ArduinoOTA.onStart([]() { Serial.println("OTA update started"); }); + ArduinoOTA.onStart([]() { + // The update blocks loop() until it is done + firmwareGuardStopWatchdog(); + Serial.println("OTA update started"); + }); ArduinoOTA.onEnd([]() { Serial.println("OTA update done"); }); - ArduinoOTA.onError([](ota_error_t error) { Serial.printf("OTA error %u\n", error); }); + ArduinoOTA.onError([](ota_error_t error) { + Serial.printf("OTA error %u, restarting\n", error); + ESP.restart(); + }); ArduinoOTA.begin(); } @@ -43,27 +71,15 @@ void otaLoop() ArduinoOTA.handle(); } -// A normal start is a power on of CPU 0 or a start of CPU 1 by CPU 0 -static bool unexpectedReset() +void otaOnlyMode(uint32_t duration, int led_pin) { - RESET_REASON cpu0 = rtc_get_reset_reason(0); - RESET_REASON cpu1 = rtc_get_reset_reason(1); - Serial.printf("Reset reason: CPU0 %d, CPU1 %d\n", cpu0, cpu1); - return cpu0 != POWERON_RESET && cpu1 != EXT_CPU_RESET; -} - -void otaPanicWindow(int led_pin) -{ - if (!unexpectedReset()) - return; - - Serial.println("Unexpected reset: waiting for an OTA update"); + Serial.printf("OTA only for %lu s\n", (unsigned long)(duration / 1000)); uint32_t start = millis(); - while (millis() - start < OTA_PANIC_WINDOW) + while (millis() - start < duration) { ArduinoOTA.handle(); - // fast blink - digitalWrite(led_pin, (millis() / 50) % 2); + wifiWatchdog(); + digitalWrite(led_pin, (millis() / 50) % 2); // fast blink delay(10); } digitalWrite(led_pin, LOW); diff --git a/src/esp32/net/wifi_ota.h b/src/esp32/net/wifi_ota.h index 82de5f2..15d1135 100644 --- a/src/esp32/net/wifi_ota.h +++ b/src/esp32/net/wifi_ota.h @@ -1,14 +1,19 @@ #pragma once +#include + // Connect to WiFi. Restarts the ESP32 when the first connection fails. void wifiBegin(); bool wifiConnected(); +// Restarts the ESP32 when WiFi has been down for WIFI_RESTART_TIME (a stuck WiFi stack +// would otherwise make OTA unreachable). Call every loop. +void wifiWatchdog(); + // Over-the-air updates of the ESP32 firmware (pio run -e esp32 -t upload) void otaBegin(); void otaLoop(); -// After a crash or unexpected reset, run only OTA for OTA_PANIC_WINDOW, so a firmware that -// crashes at boot can still be replaced. Does nothing after a normal power on. -void otaPanicWindow(int led_pin); +// Run only WiFi and OTA for `duration` ms (after a crash / safe mode), LED blinking fast +void otaOnlyMode(uint32_t duration, int led_pin); diff --git a/src/esp32/system/firmware_guard.cpp b/src/esp32/system/firmware_guard.cpp new file mode 100644 index 0000000..74b82d2 --- /dev/null +++ b/src/esp32/system/firmware_guard.cpp @@ -0,0 +1,140 @@ +#include "system/firmware_guard.h" + +#include +#include +#include +#include + +#include "config.h" + +// Survives crashes and software resets, not a power loss +static const uint32_t RTC_MAGIC = 0xF11D0C0A; +RTC_NOINIT_ATTR static uint32_t rtc_magic; +RTC_NOINIT_ATTR static uint8_t rtc_crash_count; + +static uint32_t main_start = 0; // millis() when the main code started +static bool watchdog_running = false; +static bool firmware_pending = false; // new OTA firmware, not yet marked as good +static bool crash_count_cleared = false; +static char status[64]; + +// The Arduino core marks a new OTA firmware as good right at boot. Returning true here +// leaves that to us, so the bootloader can roll back a firmware that fails early. +extern "C" bool verifyRollbackLater() +{ + return true; +} + +static ResetKind resetKind() +{ + switch (esp_reset_reason()) + { + case ESP_RST_POWERON: + return ResetKind::PowerOn; + case ESP_RST_SW: + return ResetKind::Software; + case ESP_RST_PANIC: + case ESP_RST_INT_WDT: + case ESP_RST_TASK_WDT: + case ESP_RST_WDT: + return ResetKind::Crash; + default: + return ResetKind::Other; + } +} + +static void setCrashCount(uint8_t count) +{ + rtc_magic = RTC_MAGIC; + rtc_crash_count = count; +} + +static const char *firmwareState() +{ + esp_ota_img_states_t state; + if (esp_ota_get_state_partition(esp_ota_get_running_partition(), &state) != ESP_OK) + return "usb"; + switch (state) + { + case ESP_OTA_IMG_PENDING_VERIFY: + return "testing"; + case ESP_OTA_IMG_VALID: + return "ok"; + case ESP_OTA_IMG_NEW: + // The bootloader turns NEW into PENDING_VERIFY when it supports rollback + return "ok, no rollback in bootloader"; + default: + return "usb"; + } +} + +static void updateStatus() +{ + snprintf(status, sizeof(status), "%s, %s, %u crashes", FIRMWARE_VERSION, firmwareState(), + rtc_magic == RTC_MAGIC ? rtc_crash_count : 0); +} + +BootMode firmwareGuardBegin() +{ + uint8_t count = rtc_magic == RTC_MAGIC ? rtc_crash_count : 0; + BootDecision decision = decideBoot(resetKind(), count); + setCrashCount(decision.crash_count); + + esp_ota_img_states_t state; + firmware_pending = esp_ota_get_state_partition(esp_ota_get_running_partition(), &state) == ESP_OK && + state == ESP_OTA_IMG_PENDING_VERIFY; + + updateStatus(); + Serial.printf("Firmware %s, reset reason %d, crash count %u\n", status, esp_reset_reason(), + decision.crash_count); + return decision.mode; +} + +void firmwareGuardSafeModeEnded() +{ + setCrashCount(crashCountAfterSafeMode()); +} + +void firmwareGuardStartWatchdog() +{ + // Reconfigures the watchdog the ESP-IDF already started (default 5 s) + esp_task_wdt_init(WATCHDOG_TIMEOUT_S, true); + esp_task_wdt_add(NULL); + watchdog_running = true; + main_start = millis(); +} + +void firmwareGuardStopWatchdog() +{ + if (watchdog_running) + esp_task_wdt_delete(NULL); + watchdog_running = false; +} + +void firmwareGuardLoop(bool wifi_connected) +{ + if (watchdog_running) + esp_task_wdt_reset(); + + uint32_t running = millis() - main_start; + + if (firmware_pending && shouldMarkFirmwareValid(running, wifi_connected)) + { + esp_ota_mark_app_valid_cancel_rollback(); + firmware_pending = false; + updateStatus(); + Serial.println("New firmware marked as good"); + } + + if (!crash_count_cleared && shouldClearCrashCount(running)) + { + setCrashCount(0); + crash_count_cleared = true; + updateStatus(); + } +} + +const char *firmwareGuardStatus() +{ + return status; +} diff --git a/src/esp32/system/firmware_guard.h b/src/esp32/system/firmware_guard.h new file mode 100644 index 0000000..35eac50 --- /dev/null +++ b/src/esp32/system/firmware_guard.h @@ -0,0 +1,32 @@ +#pragma once + +#include "logic/boot_policy.h" + +// Keeps the ESP32 reachable over OTA, whatever the firmware does: +// +// - Rollback: a new OTA firmware is only marked as good after it ran FIRMWARE_VALID_TIME +// with WiFi. If it crashes, hangs or restarts before that, the bootloader starts the +// previous firmware again. +// - Watchdog: loop() must keep running; a hang longer than WATCHDOG_TIMEOUT_S restarts +// the ESP32 (and counts as a crash). +// - Crash counter (kept in RTC memory over resets): after a crash the ESP32 first runs +// only OTA for a while, after SAFE_MODE_CRASHES crashes in a row it goes to safe mode. +// The decisions are in logic/boot_policy. + +// Call first in setup(): reads the reset reason and updates the crash counter +BootMode firmwareGuardBegin(); + +// Safe mode is over: allow one more try of the main code +void firmwareGuardSafeModeEnded(); + +// Start the watchdog for loop(), at the end of setup() +void firmwareGuardStartWatchdog(); + +// Stop the watchdog, e.g. during an OTA update (the update blocks loop()) +void firmwareGuardStopWatchdog(); + +// Call every loop(): feeds the watchdog, marks a new firmware as good, clears the counter +void firmwareGuardLoop(bool wifi_connected); + +// Short status for Home Assistant, e.g. "2.1.0, ok, 0 crashes" +const char *firmwareGuardStatus(); diff --git a/test/test_esp32/test_main.cpp b/test/test_esp32/test_main.cpp index 813f318..582c1b3 100644 --- a/test/test_esp32/test_main.cpp +++ b/test/test_esp32/test_main.cpp @@ -11,6 +11,7 @@ #include #include "config.h" +#include "logic/boot_policy.h" #include "logic/display_controller.h" #include "logic/power_timer.h" #include "logic/schedule.h" @@ -171,6 +172,81 @@ void test_timer_across_millis_overflow() TEST_ASSERT_TRUE(t.expired(0x00000100u)); } +// --------------------------------------------------------------------------- +// Boot policy (keeping OTA alive) +// --------------------------------------------------------------------------- + +void test_power_on_starts_normally_and_clears_crashes() +{ + BootDecision d = decideBoot(ResetKind::PowerOn, 2); + TEST_ASSERT_TRUE(d.mode == BootMode::Normal); + TEST_ASSERT_EQUAL(0, d.crash_count); +} + +void test_software_reset_starts_normally_and_keeps_count() +{ + // e.g. the restart after an OTA update + BootDecision d = decideBoot(ResetKind::Software, 1); + TEST_ASSERT_TRUE(d.mode == BootMode::Normal); + TEST_ASSERT_EQUAL(1, d.crash_count); +} + +void test_other_reset_starts_normally_and_keeps_count() +{ + BootDecision d = decideBoot(ResetKind::Other, 2); + TEST_ASSERT_TRUE(d.mode == BootMode::Normal); + TEST_ASSERT_EQUAL(2, d.crash_count); +} + +void test_first_crashes_give_a_panic_window() +{ + BootDecision d = decideBoot(ResetKind::Crash, 0); + TEST_ASSERT_TRUE(d.mode == BootMode::PanicWindow); + TEST_ASSERT_EQUAL(1, d.crash_count); + d = decideBoot(ResetKind::Crash, d.crash_count); + TEST_ASSERT_TRUE(d.mode == BootMode::PanicWindow); + TEST_ASSERT_EQUAL(2, d.crash_count); +} + +void test_repeated_crashes_give_safe_mode() +{ + uint8_t count = 0; + BootDecision d = {BootMode::Normal, 0}; + for (int i = 0; i < SAFE_MODE_CRASHES; i++) + { + d = decideBoot(ResetKind::Crash, count); + count = d.crash_count; + } + TEST_ASSERT_TRUE(d.mode == BootMode::SafeMode); +} + +void test_after_safe_mode_one_more_crash_gives_safe_mode_again() +{ + uint8_t count = crashCountAfterSafeMode(); + TEST_ASSERT_TRUE(decideBoot(ResetKind::Software, count).mode == BootMode::Normal); + TEST_ASSERT_TRUE(decideBoot(ResetKind::Crash, count).mode == BootMode::SafeMode); +} + +void test_crash_count_does_not_overflow() +{ + BootDecision d = decideBoot(ResetKind::Crash, 255); + TEST_ASSERT_EQUAL(255, d.crash_count); + TEST_ASSERT_TRUE(d.mode == BootMode::SafeMode); +} + +void test_new_firmware_needs_time_and_wifi_to_be_accepted() +{ + TEST_ASSERT_FALSE(shouldMarkFirmwareValid(FIRMWARE_VALID_TIME - 1, true)); + TEST_ASSERT_FALSE(shouldMarkFirmwareValid(FIRMWARE_VALID_TIME, false)); + TEST_ASSERT_TRUE(shouldMarkFirmwareValid(FIRMWARE_VALID_TIME, true)); +} + +void test_crash_count_cleared_after_running_long_enough() +{ + TEST_ASSERT_FALSE(shouldClearCrashCount(CRASH_FREE_TIME - 1)); + TEST_ASSERT_TRUE(shouldClearCrashCount(CRASH_FREE_TIME)); +} + // --------------------------------------------------------------------------- // Status LED // --------------------------------------------------------------------------- @@ -348,6 +424,16 @@ int main() RUN_TEST(test_timer_cancel); RUN_TEST(test_timer_across_millis_overflow); + RUN_TEST(test_power_on_starts_normally_and_clears_crashes); + RUN_TEST(test_software_reset_starts_normally_and_keeps_count); + RUN_TEST(test_other_reset_starts_normally_and_keeps_count); + RUN_TEST(test_first_crashes_give_a_panic_window); + RUN_TEST(test_repeated_crashes_give_safe_mode); + RUN_TEST(test_after_safe_mode_one_more_crash_gives_safe_mode_again); + RUN_TEST(test_crash_count_does_not_overflow); + RUN_TEST(test_new_firmware_needs_time_and_wifi_to_be_accepted); + RUN_TEST(test_crash_count_cleared_after_running_long_enough); + RUN_TEST(test_led_off_while_display_off); RUN_TEST(test_led_blinks_slow_with_wifi_and_fast_without);