Add ESP32 recovery app with WiFi setup portal

- New flash layout (partitions.csv): factory = recovery app, ota_0/ota_1 = clock firmware; esp32_usb uploads to ota_0 so it never overwrites it - src/recovery: WiFi + OTA + web page only. Opens the password protected 'FlipDot-Setup' AP with a captive portal when WiFi fails, lets you pick another WiFi (stored in NVS, used by both apps), start the clock or stay - Recovery is started when the clock firmware can't connect to WiFi, crashes 3 times, after 3 quick power cycles, from a new Home Assistant button, or by the bootloader when no clock firmware works - Shared NVS store and boot switching in src/esp32/system - Clock firmware keeps working on the old flash layout (falls back to OTA-only safe mode) - README: recovery app, one-time USB installation; tests: 65 total Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 01:21:04 +02:00
1 parent ddbdf7e65f
commit 4749f9e596
@@ -104,7 +104,24 @@ See README.md for the overview.
 - The crash counter lives in `RTC_NOINIT_ATTR` memory. Decisions are in
  `logic/boot_policy.*` (tested). Safe mode runs only WiFi + OTA.
 - Keep `otaLoop()` and `firmwareGuardLoop()` at the top of `loop()`, before anything that
-  can return early or block.
+  can return early or block. Keep `firmwareGuardBegin()` first in `setup()`: it holds
+  the power-cycle trigger for the recovery app.
+- Flash layout `partitions.csv`: factory = recovery app (`src/recovery`, env `recovery`,
+  USB only), ota_0/ota_1 = clock firmware. `esp32_usb` uploads to 0x110000
+  (`board_upload.offset_address`) so it never overwrites the recovery app. Check with
+  `pio run -e <env> -t envdump` (`ESP32_APP_OFFSET`) after changing upload settings.
+- The recovery app compiles `esp32/logic/boot_policy.cpp`, `esp32/system/system_store.cpp`
+  and `esp32/system/boot_target.cpp` (see its `build_src_filter`). Keep those free of
+  clock-firmware dependencies. Both apps share NVS namespace `flipdot`. RTC memory is
+  not shared between the apps (different layouts).
+- `rebootToRecovery()` selects the factory partition, which erases otadata. The recovery
+  app later selects the clock slot again (`startMainFirmware()`), so the clock firmware
+  is `PENDING_VERIFY` again and must pass the 60 s check, or the bootloader falls back to
+  the recovery app.
+- The clock firmware still works on the old flash layout (no factory partition):
+  `rebootToRecovery()` returns, and the callers fall back to OTA-only mode or a restart.
+- Arduino's `esp_app_desc_t` version and date come from the precompiled core, so the
+  clock firmware stores `FIRMWARE_VERSION` per slot in NVS for the recovery page.

 ## Git

@@ -36,8 +36,10 @@ Nano and drawn there.
 platformio.ini          one PlatformIO project, multiple environments
 lib/BROSE9323/          shared flipdot driver (used by both targets)
 src/nano/               Arduino Nano firmware
-src/esp32/              ESP32 firmware
+src/esp32/              ESP32 clock firmware
  secrets.example.h     template for WiFi / MQTT credentials
+src/recovery/           ESP32 recovery app: WiFi setup page + OTA
+partitions.csv          ESP32 flash layout (recovery app + two clock firmware slots)
 test/test_nano/         unit tests for the Nano firmware (run on the PC)
 test/test_esp32/        unit tests for the ESP32 logic (run on the PC)
 test/mocks/             fake Arduino, EEPROM and display driver for the Nano tests
@@ -61,7 +63,8 @@ this folder.
 | `nano` | Arduino Nano (ATmega328P) | USB (`COM10`) |
 | `nano_ota` | Arduino Nano (ATmega328P) | Over WiFi through the ESP32 (`10.1.1.198`). See [Flashing the Nano over WiFi](#flashing-the-nano-over-wifi). |
 | `esp32` | ESP32 DevKit v1 | OTA over WiFi (`10.1.1.198`) |
-| `esp32_usb` | ESP32 DevKit v1 | USB (`COM11`). Use it for the first flash or for recovery. |
+| `esp32_usb` | ESP32 DevKit v1 | USB (`COM11`), into `ota_0`. Normally not needed. |
+| `recovery` | ESP32 DevKit v1 | USB (`COM11`), one-time install of the [recovery app](#recovery-app) |

 ```sh
 pio run                          # build nano + esp32
@@ -155,6 +158,69 @@ even if the uploader is interrupted.
 > **Security:** anyone on your network can connect to port 2323 and flash the Nano, just
 > like the ESP32's own OTA has no password. Keep the display on a trusted network.

+## Recovery app
+
+The ESP32 has a small second app, the **recovery app** ([src/recovery](src/recovery)). It
+has its own place on the flash and is never changed by OTA updates. It contains only WiFi,
+a setup web page and OTA, so the ESP32 can always be reached, even when the clock firmware
+is broken. The display stays off while it runs, and the built-in LED double-blinks.
+
+```
+flash (4 MB):  bootloader | nvs | otadata | factory: recovery app | ota_0: clock | ota_1: clock
+```
+
+**When it starts**
+
+| Trigger | Back to the clock |
+|---------|-------------------|
+| The clock firmware can't connect to WiFi (30 s at boot, or 10 min later on) | As soon as the WiFi works again (after at least 1 minute) |
+| The clock firmware crashed 3 times in a row | After 10 minutes |
+| **FlipDot Display Recovery** button in Home Assistant | After 10 minutes |
+| Switching the power on and off 3 times, each time within 10 s | After 10 minutes |
+| There is no working clock firmware (the bootloader falls back to it) | Never by itself |
+
+The setup page can also start the clock right away, or keep the recovery app running
+("Stay in recovery").
+
+**WiFi setup.** The recovery app keeps trying the stored WiFi. If that fails for 20 s, it
+also opens its own WiFi: **FlipDot-Setup**, with the password `SETUP_AP_PASSWORD` from
+`src/esp32/secrets.h`. Connect a phone to it; the setup page opens by itself (otherwise go
+to <http://192.168.4.1>). Pick a network, enter the password and save. The new WiFi is
+stored in the ESP32's NVS and used by both apps from then on (instead of `WIFI_SSID` from
+`secrets.h`). The setup page is also reachable on your normal WiFi, at the ESP32's IP
+address.
+
+**Updating the clock firmware from the recovery app.** Upload as usual,
+`pio run -e esp32 -t upload`. On the setup WiFi, add `--upload-port 192.168.4.1`. The
+recovery app writes the firmware to `ota_0` and starts it.
+
+**Updating the recovery app itself** needs USB (`pio run -e recovery -t upload`). It
+should rarely be necessary.
+
+### One-time installation (USB)
+
+This changes the flash layout, so it can only be done over USB. It is the last USB flash
+the ESP32 needs. Best done together with the wiring TODOs, while the display is open
+anyway.
+
+1. Connect the ESP32 over USB (`COM11`, see `platformio.ini`).
+2. Erase the flash and install the recovery app:
+   ```sh
+   pio run -e recovery -t erase
+   pio run -e recovery -t upload
+   ```
+   The recovery app starts (LED double-blinks), connects to the WiFi from `secrets.h`, and
+   waits, because there is no clock firmware yet.
+3. Install the clock firmware **over WiFi**. This also tests that the recovery app works:
+   ```sh
+   pio run -e esp32 -t upload
+   ```
+4. Check Home Assistant: the **FlipDot Display Firmware** sensor should show
+   `2.1.0, testing, ...`, and after a minute `2.1.0, ok, 0 crashes, recovery ok`.
+
+Erasing the flash also clears the NVS: WiFi settings made on the setup page are reset to
+the ones in `secrets.h`.
+
 ## Wiring between ESP32 and Nano

 The ESP32 talks to the Nano on **UART2** (`Serial2`, 115200 baud). The ESP32's own
@@ -194,6 +260,13 @@ The serial protocol is described in [src/nano/README.md](src/nano/README.md).

 ## TODO

+- [ ] **Install the recovery app** (one-time USB flash, see
+  [One-time installation](#one-time-installation-usb)). Until then the ESP32 runs the old
+  flash layout: the clock firmware works, but without the recovery app; the Home
+  Assistant sensor shows "no recovery app".
+- [ ] **Test the recovery app on the real hardware**: the Home Assistant button, the setup
+  WiFi with a wrong WiFi password, 3 quick power cycles, and an OTA upload into it.
+
 - [ ] **Rewire the Nano link to UART2.** The firmware now uses `Serial2` (GPIO 16/17)
  instead of `Serial` (GPIO 1/3). Move the ESP32 TX wire from GPIO 1 to **GPIO 17** before
  you flash this firmware, or the display gets no data. See
@@ -0,0 +1,12 @@
+# ESP32 flash layout (4 MB). See README.md "Recovery app".
+#
+#   factory  recovery app (src/recovery): WiFi setup portal + OTA, never changed by OTA
+#   ota_0/1  clock firmware (src/esp32), updated over OTA in turns, with rollback
+#
+# Name,   Type, SubType,  Offset,   Size
+nvs,      data, nvs,      0x9000,   0x5000
+otadata,  data, ota,      0xe000,   0x2000
+factory,  app,  factory,  0x10000,  0x100000
+ota_0,    app,  ota_0,    0x110000, 0x170000
+ota_1,    app,  ota_1,    0x280000, 0x170000
+coredump, data, coredump, 0x3F0000, 0x10000
@@ -3,7 +3,8 @@
 ;   nano       Arduino Nano that drives the Brose flipdot panels directly (USB upload)
 ;   nano_ota   Same Nano firmware, flashed over WiFi through the ESP32
 ;   esp32     ESP32 "brain" (WiFi, NTP clock, Home Assistant) - OTA upload
-;   esp32_usb  Same ESP32 firmware, flashed over USB (first flash / recovery)
+;   esp32_usb  Same ESP32 firmware, flashed over USB into ota_0
+;   recovery   ESP32 recovery app (WiFi setup + OTA), USB only, see README.md
 ;   test_nano  Unit tests for the Nano firmware, run on the PC (needs gcc on PATH)
 ;   test_esp32 Unit tests for the ESP32 logic, run on the PC
 ;
@@ -53,26 +54,50 @@ upload_command = "$PYTHONEXE" "$PROJECT_DIR/tools/nano_net_upload.py" "$SOURCE"
 ; ---------------------------------------------------------------------------
 ; ESP32 - controller, talks to the Nano over serial
 ; ---------------------------------------------------------------------------
-[env:esp32]
+[esp32_common]
 platform = espressif32
 board = esp32doit-devkit-v1
+; factory = recovery app, ota_0 / ota_1 = clock firmware (see partitions.csv)
+board_build.partitions = partitions.csv
+monitor_port = COM11
+
+[env:esp32]
+extends = esp32_common
 build_src_filter = +<esp32/>
+; Size check against the ota slots, not the (smaller) factory partition
+board_build.app_partition_name = ota_0
 ; ESPSERIAL turns the BROSE9323 driver into a serial client: every draw call is
 ; sent to the Nano as a text command instead of toggling GPIO pins.
 build_flags = -D ESPSERIAL -I src/esp32
 lib_deps =
 	${env.lib_deps}
-	https://github.com/robjen/GFX_fonts.git
 	dawidchyrzynski/home-assistant-integration@^2.1.0
 	knolleary/PubSubClient@^2.8.0
 upload_protocol = espota
 upload_port = 10.1.1.198
-monitor_port = COM11

+; USB upload of the clock firmware: written to ota_0, never over the recovery app
 [env:esp32_usb]
 extends = env:esp32
 upload_protocol = esptool
 upload_port = COM11
+board_upload.offset_address = 0x110000
+
+; ---------------------------------------------------------------------------
+; ESP32 recovery app (src/recovery) in the factory partition: WiFi setup page + OTA.
+; One-time install over USB, see README.md "Recovery app".
+; ---------------------------------------------------------------------------
+[env:recovery]
+extends = esp32_common
+build_src_filter =
+	+<recovery/>
+	+<esp32/logic/boot_policy.cpp>
+	+<esp32/system/system_store.cpp>
+	+<esp32/system/boot_target.cpp>
+build_flags = -I src/esp32
+lib_deps =
+upload_protocol = esptool
+upload_port = COM11

 ; ---------------------------------------------------------------------------
 ; Unit tests on the PC (pio test -e test_nano -e test_esp32)
@@ -20,8 +20,10 @@ debug output. Wiring is described in the [main README](../../README.md#wiring-be
 | `logic/power_timer.*` | Countdown to switching the display off |
 | `logic/status_led.h` | Blink pattern of the built-in LED |
 | `logic/boot_policy.*` | How to start after a reset: normal, OTA-only window, safe mode |
-| **`system/`** | **Keeping OTA alive** |
-| `system/firmware_guard.*` | Rollback of bad OTA firmware, watchdog, crash counter, status for Home Assistant |
+| **`system/`** | **Keeping OTA alive** (`system_store` and `boot_target` are shared with the recovery app) |
+| `system/firmware_guard.*` | Rollback of bad OTA firmware, watchdog, crash counter, power-cycle trigger, status for Home Assistant |
+| `system/boot_target.*` | Switching between the clock firmware and the recovery app |
+| `system/system_store.*` | Settings in NVS: WiFi, recovery reason, firmware slot and version, power-on count |
 | **`display/`** | **What is drawn** |
 | `display/frames.*` | The screens: clock, logo, blank, text (`FlipdotFrames`) |
 | `display/bitmaps.h` | 1-bit bitmaps (Pikachu, thought bubble, Nintendo logo, ...) |
@@ -30,7 +32,7 @@ debug output. Wiring is described in the [main README](../../README.md#wiring-be
 | `nano_link/nano_flash_bridge.*` | WiFi to serial bridge for flashing the Nano over WiFi (TCP port 2323) |
 | **`net/`** | **Network** |
 | `net/wifi_ota.*` | WiFi (with a restart when it stays down), ESP32 OTA updates, OTA-only mode |
-| `net/home_assistant.*` | MQTT and the two Home Assistant switches |
+| `net/home_assistant.*` | MQTT and the Home Assistant entities (two switches, firmware sensor, recovery button) |

 `DisplayController` only talks to two small interfaces, `DisplayPower` (implemented by
 `NanoLink`) and `Frames` (implemented by `FlipdotFrames`). The tests replace both with
@@ -46,8 +48,9 @@ again. Several layers take care of that:
 | A new OTA firmware crashes, hangs or can't connect to WiFi | **Rollback.** A new firmware is only marked as good after it has run for 60 s with WiFi. If it restarts before that, the bootloader starts the previous firmware again. |
 | The firmware hangs in `loop()` | **Watchdog.** If `loop()` doesn't run for 30 s, the ESP32 restarts. That counts as a crash. |
 | The firmware crashes once or twice | **OTA window.** After a crash the ESP32 runs only WiFi and OTA for 15 s (LED blinks fast), then starts the main code again. |
-| The firmware keeps crashing (3 times in a row) | **Safe mode.** Only WiFi and OTA run, for 10 minutes. Then the main code gets one more try; one more crash brings safe mode back. Running 5 minutes without a crash clears the counter. |
-| WiFi stays down | After 10 minutes without WiFi, the ESP32 restarts, in case the WiFi stack is stuck. |
+| The firmware keeps crashing (3 times in a row) | **Recovery app** for 10 minutes (see the [main README](../../README.md#recovery-app)). Then the main code gets one more try; one more crash brings the recovery app back. Running 5 minutes without a crash clears the counter. Without the recovery app installed: only WiFi and OTA run, for 10 minutes. |
+| WiFi doesn't connect at boot (30 s), or stays down for 10 minutes | **Recovery app**, which keeps trying and opens the **FlipDot-Setup** WiFi with a setup page to choose another WiFi. Without the recovery app: restart. |
+| The firmware runs, but its OTA is broken | **Recovery app** through the Home Assistant button, or by switching the power on and off 3 times quickly. |

 So in every case you can flash a fix with `pio run -e esp32 -t upload`, at the latest
 after waiting for a safe-mode cycle.
@@ -59,7 +62,7 @@ The **FlipDot Display Firmware** sensor in Home Assistant shows the state, e.g.
 |-------|---------|
 | `testing` | New OTA firmware, not yet marked as good (first 60 s) |
 | `ok` | Firmware marked as good |
-| `ok, no rollback in bootloader` | The bootloader on the ESP32 can't roll back. The other layers still work. |
+| `ok, no rollback in bootloader` | The bootloader on the ESP32 can't roll back. The other layers still work. The one-time USB install of the recovery app also installs a bootloader that can. |
 | `usb` | Flashed over USB, so there is no previous firmware to roll back to |

 Rules for changing the code:
@@ -84,6 +87,10 @@ Rules for changing the code:
  - `FlipDot Display Clock`: on = clock; off = Nintendo logo while the display is on,
    then power off after 30 s
  - `FlipDot Display Power`: off = clear the display and power it off after 1 s
+
+  Plus the **FlipDot Display Firmware** sensor (see
+  [Keeping OTA alive](#keeping-ota-alive)) and the **FlipDot Display Recovery** button,
+  which restarts into the recovery app for 10 minutes.
 - **Relay on GPIO 13** switches the power for both the panels and the Nano. `LOW` turns it
  on.
  - **On:** the ESP waits for the Nano's `READY` message, at most 3 s
@@ -74,6 +74,16 @@ const uint32_t FIRMWARE_VALID_TIME = 60UL * 1000;
 const uint32_t CRASH_FREE_TIME = 5UL * 60 * 1000;
 // loop() must not block longer than this, or the ESP32 restarts (counts as a crash)
 const uint32_t WATCHDOG_TIMEOUT_S = 30;
+// Recovery app (src/recovery): started by the triggers in logic/boot_policy.h
+const uint8_t POWER_CYCLES_FOR_RECOVERY = 3;  // power ons in a row ...
+const uint32_t POWER_CYCLE_WINDOW = 10000;    // ... each switched off within this time
+const uint32_t RECOVERY_TIME = 10UL * 60 * 1000; // then back to the clock firmware
+// Stay at least this long, also when the WiFi works right away (limits switching back and
+// forth, which wears the otadata flash sector)
+const uint32_t RECOVERY_MIN_TIME = 60UL * 1000;
+#define SETUP_AP_SSID "FlipDot-Setup"          // password: SETUP_AP_PASSWORD in secrets.h
+const uint32_t SETUP_AP_DELAY = 20000;        // open the AP when WiFi fails this long
+
 // WiFi: first connection at boot, and restart when WiFi stays down this long
 const uint32_t WIFI_CONNECT_TIMEOUT = 30000;
 const uint32_t WIFI_RESTART_TIME = 10UL * 60 * 1000;
@@ -6,3 +6,7 @@
 #else
 #error "Missing src/esp32/secrets.h - copy src/esp32/secrets.example.h to secrets.h and fill it in"
 #endif
+
+#ifndef SETUP_AP_PASSWORD
+#error "Add SETUP_AP_PASSWORD to src/esp32/secrets.h (see secrets.example.h)"
+#endif
@@ -35,3 +35,36 @@ bool shouldClearCrashCount(uint32_t running_ms)
 {
 	return running_ms >= CRASH_FREE_TIME;
 }
+
+const char *recoveryReasonText(RecoveryReason reason)
+{
+	switch (reason)
+	{
+	case RecoveryReason::Wifi:
+		return "the clock firmware could not connect to WiFi";
+	case RecoveryReason::Crash:
+		return "the clock firmware crashed repeatedly";
+	case RecoveryReason::Manual:
+		return "requested from Home Assistant";
+	case RecoveryReason::PowerCycle:
+		return "the power was switched on and off quickly";
+	case RecoveryReason::None:
+	default:
+		return "no working clock firmware";
+	}
+}
+
+bool powerCyclesRequestRecovery(uint8_t power_on_count)
+{
+	return power_on_count >= POWER_CYCLES_FOR_RECOVERY;
+}
+
+bool recoveryShouldStartMain(RecoveryReason reason, uint32_t running_ms, bool wifi_connected,
+							 bool main_available)
+{
+	if (!main_available)
+		return false;
+	if (reason == RecoveryReason::Wifi)
+		return wifi_connected && running_ms >= RECOVERY_MIN_TIME;
+	return running_ms >= RECOVERY_TIME;
+}
@@ -9,8 +9,12 @@
 //   Software reset           normal start (e.g. after an OTA update)
 //   Crash (panic, watchdog)  crash counter + 1, then:
 //                              < SAFE_MODE_CRASHES  OTA-only for OTA_PANIC_WINDOW, then normal
-//                              >= SAFE_MODE_CRASHES safe mode: OTA only for SAFE_MODE_TIME
+//                              >= SAFE_MODE_CRASHES safe mode: the recovery app, or OTA
+//                                                   only for SAFE_MODE_TIME without it
 //   Other (brownout, ...)    normal start, counter kept
+//
+// The recovery app (src/recovery) uses the same file to decide when to start the clock
+// firmware again.

 enum class ResetKind
 {
@@ -43,3 +47,26 @@ bool shouldMarkFirmwareValid(uint32_t running_ms, bool wifi_connected);

 // Running this long without a crash clears the crash counter
 bool shouldClearCrashCount(uint32_t running_ms);
+
+// Why the recovery app was started (stored in NVS, shared by both apps)
+enum class RecoveryReason : uint8_t
+{
+	None = 0,       // no firmware in ota_0/ota_1, or the bootloader fell back to recovery
+	Wifi = 1,       // the clock firmware couldn't connect to WiFi
+	Crash = 2,      // the clock firmware crashed SAFE_MODE_CRASHES times in a row
+	Manual = 3,     // Home Assistant button
+	PowerCycle = 4, // switched on POWER_CYCLES_FOR_RECOVERY times quickly
+};
+
+const char *recoveryReasonText(RecoveryReason reason);
+
+// Power-on count: each power on within POWER_CYCLE_WINDOW of the previous one counts.
+// True when the next start should be the recovery app.
+bool powerCyclesRequestRecovery(uint8_t power_on_count);
+
+// Recovery app: time to start the clock firmware again?
+//   Wifi reason:   as soon as WiFi is connected again (but not before RECOVERY_MIN_TIME)
+//   other reasons: after RECOVERY_TIME (so a manual visit has time for an OTA upload)
+//   never when there is no clock firmware
+bool recoveryShouldStartMain(RecoveryReason reason, uint32_t running_ms, bool wifi_connected,
+							 bool main_available);
@@ -13,6 +13,7 @@
 #include "nano_link/nano_link.h"
 #include "net/home_assistant.h"
 #include "net/wifi_ota.h"
+#include "system/boot_target.h"
 #include "system/firmware_guard.h"

 NanoLink nano(Serial2, NANO_RX_PIN, NANO_TX_PIN, RELAY_PIN, NANO_RESET_PIN);
@@ -41,9 +42,11 @@ void setup()

 	if (boot_mode == BootMode::SafeMode)
 	{
-		// Crashed too often: only WiFi and OTA, then try the main code once more
-		otaOnlyMode(SAFE_MODE_TIME, LED_PIN);
+		// Crashed too often: the recovery app, or without it only WiFi and OTA here,
+		// then try the main code once more
 		firmwareGuardSafeModeEnded();
+		rebootToRecovery(RecoveryReason::Crash);
+		otaOnlyMode(SAFE_MODE_TIME, LED_PIN);
 		ESP.restart();
 	}
 	if (boot_mode == BootMode::PanicWindow)
@@ -6,6 +6,7 @@

 #include "config.h"
 #include "credentials.h"
+#include "system/boot_target.h"

 static WiFiClient client;
 static HADevice device(HA_DEVICE_ID, sizeof(HA_DEVICE_ID));
@@ -15,6 +16,7 @@ static HAMqtt mqtt(client, device);
 static HASwitch clockSwitch("FlipDotDisplayClock");
 static HASwitch powerSwitch("FlipDotDisplayPower");
 static HASensor firmwareSensor("FlipDotDisplayFirmware");
+static HAButton recoveryButton("FlipDotDisplayRecovery");

 static char firmware_status[64] = "";
 static bool firmware_status_sent = false;
@@ -51,6 +53,10 @@ void homeAssistantBegin(DisplayController &controller)
 	powerSwitch.setCurrentState(controller.powerEnabled());
 	powerSwitch.onCommand(onPowerCommand);

+	recoveryButton.setName("FlipDot Display Recovery");
+	recoveryButton.setIcon("mdi:lifebuoy");
+	recoveryButton.onCommand([](HAButton *) { rebootToRecovery(RecoveryReason::Manual); });
+
 	firmwareSensor.setName("FlipDot Display Firmware");
 	firmwareSensor.setIcon("mdi:chip");
 	// The sensor doesn't resend its value on reconnect by itself
@@ -5,7 +5,8 @@
 // Home Assistant integration over MQTT (with discovery):
 //   FlipDot Display Clock     switch: on = clock, off = Nintendo logo
 //   FlipDot Display Power     switch: off = clear the display and switch it off
-//   FlipDot Display Firmware  sensor: version and health, e.g. "2.1.0, ok, 0 crashes"
+//   FlipDot Display Firmware  sensor: version and health, e.g. "2.1.0, ok, 0 crashes, ..."
+//   FlipDot Display Recovery  button: restart into the recovery app (for 10 minutes)
 void homeAssistantBegin(DisplayController &controller);
 void homeAssistantLoop();

@@ -5,7 +5,9 @@

 #include "config.h"
 #include "credentials.h"
+#include "system/boot_target.h"
 #include "system/firmware_guard.h"
+#include "system/system_store.h"

 static uint32_t wifi_lost_at = 0;
 static bool wifi_lost = false;
@@ -15,11 +17,15 @@ void wifiBegin()
 	WiFi.setHostname(HOSTNAME);
 	WiFi.mode(WIFI_STA);
 	WiFi.setAutoReconnect(true);
-	WiFi.begin(WIFI_SSID, WIFI_PASSWORD);
-	Serial.println("Connecting to WiFi");
+	String ssid, password;
+	loadWifiCredentials(ssid, password);
+	WiFi.begin(ssid.c_str(), password.c_str());
+	Serial.printf("Connecting to WiFi %s\n", ssid.c_str());
 	if (WiFi.waitForConnectResult(WIFI_CONNECT_TIMEOUT) != WL_CONNECTED)
 	{
-		Serial.println("WiFi connection failed, restarting");
+		// The recovery app keeps trying and offers a setup page to pick another WiFi
+		Serial.println("WiFi connection failed");
+		rebootToRecovery(RecoveryReason::Wifi);
 		ESP.restart();
 	}
 	Serial.print("Connected, IP ");
@@ -45,7 +51,8 @@ void wifiWatchdog()
 	}
 	else if (millis() - wifi_lost_at >= WIFI_RESTART_TIME)
 	{
-		Serial.println("WiFi down for too long, restarting");
+		Serial.println("WiFi down for too long");
+		rebootToRecovery(RecoveryReason::Wifi);
 		ESP.restart();
 	}
 }
@@ -9,3 +9,7 @@
 #define BROKER_ADDR IPAddress(192, 168, 1, 10)
 #define MQTT_USER "mqtt-user"
 #define MQTT_PASSWORD "mqtt-password"
+
+// Password of the "FlipDot-Setup" WiFi that the recovery app opens when it can't connect
+// (WPA2, at least 8 characters)
+#define SETUP_AP_PASSWORD "change-me-please"
@@ -0,0 +1,92 @@
+#include "system/boot_target.h"
+
+#include <Arduino.h>
+#include <esp_ota_ops.h>
+#include <esp_partition.h>
+
+#include "system/system_store.h"
+
+static const esp_partition_t *appPartition(esp_partition_subtype_t subtype)
+{
+	return esp_partition_find_first(ESP_PARTITION_TYPE_APP, subtype, NULL);
+}
+
+// The partition starts with a firmware image (full check happens in esp_ota_set_boot_partition)
+static bool holdsFirmware(const esp_partition_t *partition)
+{
+	esp_app_desc_t desc;
+	return partition && esp_ota_get_partition_description(partition, &desc) == ESP_OK;
+}
+
+bool recoveryAvailable()
+{
+	return holdsFirmware(appPartition(ESP_PARTITION_SUBTYPE_APP_FACTORY));
+}
+
+void rebootToRecovery(RecoveryReason reason)
+{
+	const esp_partition_t *recovery = appPartition(ESP_PARTITION_SUBTYPE_APP_FACTORY);
+	if (!holdsFirmware(recovery))
+	{
+		Serial.println("No recovery app on this flash");
+		return;
+	}
+	Serial.printf("Restarting into the recovery app: %s\n", recoveryReasonText(reason));
+	setRecoveryReason(reason);
+	// Selecting factory clears otadata; the recovery app sets the clock firmware again later
+	if (esp_ota_set_boot_partition(recovery) != ESP_OK)
+	{
+		Serial.println("Could not select the recovery app");
+		return;
+	}
+	delay(100);
+	ESP.restart();
+}
+
+const char *runningPartitionLabel()
+{
+	return esp_ota_get_running_partition()->label;
+}
+
+bool mainFirmwareAvailable()
+{
+	return holdsFirmware(appPartition(ESP_PARTITION_SUBTYPE_APP_OTA_0)) ||
+		   holdsFirmware(appPartition(ESP_PARTITION_SUBTYPE_APP_OTA_1));
+}
+
+static bool startSlot(const esp_partition_t *partition)
+{
+	if (!holdsFirmware(partition))
+		return false;
+	// Verifies the whole image; fails for a broken or half written firmware
+	if (esp_ota_set_boot_partition(partition) != ESP_OK)
+		return false;
+	Serial.printf("Starting the clock firmware in %s\n", partition->label);
+	delay(100);
+	ESP.restart();
+	return true;
+}
+
+void startMainFirmware()
+{
+	const esp_partition_t *ota0 = appPartition(ESP_PARTITION_SUBTYPE_APP_OTA_0);
+	const esp_partition_t *ota1 = appPartition(ESP_PARTITION_SUBTYPE_APP_OTA_1);
+	bool last_was_ota1 = loadMainSlot() == "ota_1";
+
+	if (last_was_ota1)
+		startSlot(ota1) || startSlot(ota0);
+	else
+		startSlot(ota0) || startSlot(ota1);
+	Serial.println("No valid clock firmware found");
+}
+
+String slotDescription(const char *label)
+{
+	const esp_partition_t *partition = esp_partition_find_first(ESP_PARTITION_TYPE_APP, ESP_PARTITION_SUBTYPE_ANY, label);
+	if (!holdsFirmware(partition))
+		return "empty";
+	// The app description holds the Arduino core's version, so use the one the clock
+	// firmware stored when it started
+	String version = loadSlotVersion(label);
+	return version.isEmpty() ? "firmware (version unknown)" : "version " + version;
+}
@@ -0,0 +1,27 @@
+#pragma once
+
+#include <Arduino.h>
+
+#include "logic/boot_policy.h"
+
+// Switching between the clock firmware (ota_0 / ota_1) and the recovery app (factory).
+// See partitions.csv.
+
+// The flash has a recovery app (only with partitions.csv, after the one-time USB install)
+bool recoveryAvailable();
+
+// Restart into the recovery app. Returns only when there is no recovery app.
+void rebootToRecovery(RecoveryReason reason);
+
+// Label of the partition this firmware runs from ("factory", "ota_0", "ota_1")
+const char *runningPartitionLabel();
+
+// Recovery app: an ota slot holds a clock firmware
+bool mainFirmwareAvailable();
+
+// Recovery app: restart into the clock firmware (the last one that ran, else the other
+// slot). Returns only when no slot holds a valid firmware.
+void startMainFirmware();
+
+// The firmware in an ota slot for the recovery page, e.g. "version 2.1.0" or "empty"
+String slotDescription(const char *label);
@@ -6,6 +6,8 @@
 #include <esp_task_wdt.h>

 #include "config.h"
+#include "system/boot_target.h"
+#include "system/system_store.h"

 // Survives crashes and software resets, not a power loss
 static const uint32_t RTC_MAGIC = 0xF11D0C0A;
@@ -16,7 +18,8 @@ static uint32_t main_start = 0; // millis() when the main code started
 static bool watchdog_running = false;
 static bool firmware_pending = false; // new OTA firmware, not yet marked as good
 static bool crash_count_cleared = false;
-static char status[64];
+static bool power_on_count_cleared = false;
+static char status[80];

 // The Arduino core marks a new OTA firmware as good right at boot. Returning true here
 // leaves that to us, so the bootloader can roll back a firmware that fails early.
@@ -70,16 +73,40 @@ static const char *firmwareState()

 static void updateStatus()
 {
-	snprintf(status, sizeof(status), "%s, %s, %u crashes", FIRMWARE_VERSION, firmwareState(),
-			 rtc_magic == RTC_MAGIC ? rtc_crash_count : 0);
+	snprintf(status, sizeof(status), "%s, %s, %u crashes, %s", FIRMWARE_VERSION, firmwareState(),
+			 rtc_magic == RTC_MAGIC ? rtc_crash_count : 0,
+			 recoveryAvailable() ? "recovery ok" : "no recovery app");
+}
+
+// Switching the power on and off quickly a few times starts the recovery app. Checked
+// before anything else, so it works even when the rest of this firmware is broken.
+static void checkPowerCycles(ResetKind reset)
+{
+	if (reset != ResetKind::PowerOn)
+		return;
+	uint8_t count = loadPowerOnCount() + 1;
+	if (powerCyclesRequestRecovery(count) && recoveryAvailable())
+	{
+		savePowerOnCount(0);
+		rebootToRecovery(RecoveryReason::PowerCycle);
+	}
+	savePowerOnCount(count);
 }

 BootMode firmwareGuardBegin()
 {
+	ResetKind reset = resetKind();
+	checkPowerCycles(reset);
+
 	uint8_t count = rtc_magic == RTC_MAGIC ? rtc_crash_count : 0;
-	BootDecision decision = decideBoot(resetKind(), count);
+	BootDecision decision = decideBoot(reset, count);
 	setCrashCount(decision.crash_count);

+	// Tells the recovery app which slot to start and which version is in it
+	const char *label = runningPartitionLabel();
+	if (strncmp(label, "ota_", 4) == 0)
+		saveMainSlot(label, FIRMWARE_VERSION);
+
 	esp_ota_img_states_t state;
 	firmware_pending = esp_ota_get_state_partition(esp_ota_get_running_partition(), &state) == ESP_OK &&
 					   state == ESP_OTA_IMG_PENDING_VERIFY;
@@ -126,6 +153,13 @@ void firmwareGuardLoop(bool wifi_connected)
 		Serial.println("New firmware marked as good");
 	}

+	// Running long enough after a power on: it was not a quick power cycle
+	if (!power_on_count_cleared && millis() >= POWER_CYCLE_WINDOW)
+	{
+		savePowerOnCount(0);
+		power_on_count_cleared = true;
+	}
+
 	if (!crash_count_cleared && shouldClearCrashCount(running))
 	{
 		setCrashCount(0);
@@ -10,10 +10,13 @@
 // - Watchdog: loop() must keep running; a hang longer than WATCHDOG_TIMEOUT_S restarts
 //   the ESP32 (and counts as a crash).
 // - Crash counter (kept in RTC memory over resets): after a crash the ESP32 first runs
-//   only OTA for a while, after SAFE_MODE_CRASHES crashes in a row it goes to safe mode.
-//   The decisions are in logic/boot_policy.
+//   only OTA for a while, after SAFE_MODE_CRASHES crashes in a row it goes to safe mode
+//   (the recovery app). The decisions are in logic/boot_policy.
+// - Power cycles: switching the power on POWER_CYCLES_FOR_RECOVERY times, each time for
+//   less than POWER_CYCLE_WINDOW, starts the recovery app.

-// Call first in setup(): reads the reset reason and updates the crash counter
+// Call first in setup(): reads the reset reason, updates the crash counter, and restarts
+// into the recovery app after quick power cycles
 BootMode firmwareGuardBegin();

 // Safe mode is over: allow one more try of the main code
@@ -28,5 +31,5 @@ void firmwareGuardStopWatchdog();
 // Call every loop(): feeds the watchdog, marks a new firmware as good, clears the counter
 void firmwareGuardLoop(bool wifi_connected);

-// Short status for Home Assistant, e.g. "2.1.0, ok, 0 crashes"
+// Short status for Home Assistant, e.g. "2.1.0, ok, 0 crashes, recovery ok"
 const char *firmwareGuardStatus();
@@ -0,0 +1,98 @@
+#include "system/system_store.h"
+
+#include <Preferences.h>
+
+#include "credentials.h"
+
+static const char *NAMESPACE = "flipdot";
+
+void loadWifiCredentials(String &ssid, String &password)
+{
+	Preferences prefs;
+	prefs.begin(NAMESPACE, true);
+	ssid = prefs.getString("wifi_ssid", "");
+	password = prefs.getString("wifi_pass", "");
+	prefs.end();
+	if (ssid.isEmpty())
+	{
+		ssid = WIFI_SSID;
+		password = WIFI_PASSWORD;
+	}
+}
+
+void saveWifiCredentials(const String &ssid, const String &password)
+{
+	Preferences prefs;
+	prefs.begin(NAMESPACE, false);
+	prefs.putString("wifi_ssid", ssid);
+	prefs.putString("wifi_pass", password);
+	prefs.end();
+}
+
+void setRecoveryReason(RecoveryReason reason)
+{
+	Preferences prefs;
+	prefs.begin(NAMESPACE, false);
+	prefs.putUChar("rec_reason", (uint8_t)reason);
+	prefs.end();
+}
+
+RecoveryReason takeRecoveryReason()
+{
+	Preferences prefs;
+	prefs.begin(NAMESPACE, false);
+	uint8_t reason = prefs.getUChar("rec_reason", 0);
+	if (reason)
+		prefs.putUChar("rec_reason", 0);
+	prefs.end();
+	return reason <= (uint8_t)RecoveryReason::PowerCycle ? (RecoveryReason)reason : RecoveryReason::None;
+}
+
+void saveMainSlot(const char *label, const char *version)
+{
+	String version_key = String("ver_") + label;
+	Preferences prefs;
+	prefs.begin(NAMESPACE, false);
+	if (prefs.getString("main_slot", "") != label)
+		prefs.putString("main_slot", label);
+	if (prefs.getString(version_key.c_str(), "") != version)
+		prefs.putString(version_key.c_str(), version);
+	prefs.end();
+}
+
+String loadMainSlot()
+{
+	Preferences prefs;
+	prefs.begin(NAMESPACE, true);
+	String label = prefs.getString("main_slot", "");
+	prefs.end();
+	return label;
+}
+
+String loadSlotVersion(const char *label)
+{
+	String version_key = String("ver_") + label;
+	Preferences prefs;
+	prefs.begin(NAMESPACE, true);
+	String version = prefs.getString(version_key.c_str(), "");
+	prefs.end();
+	return version;
+}
+
+uint8_t loadPowerOnCount()
+{
+	Preferences prefs;
+	prefs.begin(NAMESPACE, true);
+	uint8_t count = prefs.getUChar("power_ons", 0);
+	prefs.end();
+	return count;
+}
+
+void savePowerOnCount(uint8_t count)
+{
+	Preferences prefs;
+	prefs.begin(NAMESPACE, false);
+	if (prefs.getUChar("power_ons", 0) != count)
+		prefs.putUChar("power_ons", count);
+	prefs.end();
+}
@@ -0,0 +1,26 @@
+#pragma once
+
+#include <Arduino.h>
+
+#include "logic/boot_policy.h"
+
+// Settings in NVS, shared by the clock firmware and the recovery app (both read the same
+// NVS partition). Survives power loss and firmware updates.
+
+// WiFi set in the recovery web page, or WIFI_SSID / WIFI_PASSWORD from secrets.h
+void loadWifiCredentials(String &ssid, String &password);
+void saveWifiCredentials(const String &ssid, const String &password);
+
+// Why the recovery app is started; take...() reads and clears it
+void setRecoveryReason(RecoveryReason reason);
+RecoveryReason takeRecoveryReason();
+
+// The clock firmware that started last: its partition label ("ota_0" / "ota_1") and
+// version, stored per slot for the recovery page
+void saveMainSlot(const char *label, const char *version);
+String loadMainSlot();
+String loadSlotVersion(const char *label);
+
+// Quick power ons in a row (see powerCyclesRequestRecovery)
+uint8_t loadPowerOnCount();
+void savePowerOnCount(uint8_t count);
@@ -0,0 +1,153 @@
+// Recovery app (factory partition): WiFi setup portal and OTA, nothing else. Started when
+// the clock firmware can't connect to WiFi, crashes repeatedly, after quick power cycles,
+// from the Home Assistant button, or by the bootloader when no clock firmware works.
+// It never drives the display. See README.md "Recovery app".
+
+#include <Arduino.h>
+#include <ArduinoOTA.h>
+#include <WiFi.h>
+#include <esp_task_wdt.h>
+
+#include "config.h"
+#include "credentials.h"
+#include "logic/boot_policy.h"
+#include "recovery_state.h"
+#include "system/boot_target.h"
+#include "system/system_store.h"
+#include "web_portal.h"
+
+// Retry the stored WiFi this often while it is not connected
+static const uint32_t STATION_RETRY_TIME = 30000;
+// Close the setup AP this long after the stored WiFi works again
+static const uint32_t AP_CLOSE_DELAY = 120000;
+static const uint32_t RECOVERY_WATCHDOG_S = 60;
+
+RecoveryState recovery;
+
+static uint32_t station_lost_since = 0;
+static uint32_t station_connected_since = 0;
+static bool watchdog_running = false;
+static bool main_available = false;
+
+void connectStation()
+{
+	String ssid, password;
+	loadWifiCredentials(ssid, password);
+	Serial.printf("Connecting to WiFi %s\n", ssid.c_str());
+	WiFi.disconnect();
+	WiFi.begin(ssid.c_str(), password.c_str());
+	recovery.station_attempt = millis();
+}
+
+static void openAccessPoint()
+{
+	WiFi.mode(WIFI_AP_STA);
+	WiFi.softAP(SETUP_AP_SSID, SETUP_AP_PASSWORD);
+	portalStartCaptiveDns();
+	recovery.ap_running = true;
+	Serial.printf("Setup WiFi \"%s\" open at %s\n", SETUP_AP_SSID, WiFi.softAPIP().toString().c_str());
+}
+
+static void closeAccessPoint()
+{
+	portalStopCaptiveDns();
+	WiFi.softAPdisconnect(true);
+	WiFi.mode(WIFI_STA);
+	recovery.ap_running = false;
+	Serial.println("Setup WiFi closed");
+}
+
+static void updateWifi()
+{
+	uint32_t now = millis();
+	if (WiFi.status() == WL_CONNECTED)
+	{
+		station_lost_since = now;
+		if (recovery.ap_running && now - station_connected_since >= AP_CLOSE_DELAY &&
+			WiFi.softAPgetStationNum() == 0)
+			closeAccessPoint();
+		return;
+	}
+	station_connected_since = now;
+
+	if (!recovery.ap_running && now - station_lost_since >= SETUP_AP_DELAY)
+		openAccessPoint();
+
+	// Connection attempts disturb the access point, so don't retry while someone uses it
+	bool ap_in_use = recovery.ap_running && WiFi.softAPgetStationNum() > 0;
+	if (!ap_in_use && now - recovery.station_attempt >= STATION_RETRY_TIME)
+		connectStation();
+}
+
+// Double blink: recovery app is running
+static void updateLed()
+{
+	uint32_t t = millis() % 1500;
+	digitalWrite(LED_PIN, t < 100 || (t >= 250 && t < 350));
+}
+
+void setup()
+{
+	Serial.begin(115200);
+	Serial.println("FlipDot recovery app");
+
+	// Display off: relay off, and no high TX line that would power the Nano
+	digitalWrite(RELAY_PIN, HIGH);
+	pinMode(RELAY_PIN, OUTPUT);
+	pinMode(NANO_TX_PIN, INPUT);
+	pinMode(LED_PIN, OUTPUT);
+
+	recovery.reason = takeRecoveryReason();
+	main_available = mainFirmwareAvailable();
+	Serial.printf("Reason: %s\n", recoveryReasonText(recovery.reason));
+
+	WiFi.persistent(false);
+	WiFi.setAutoReconnect(false); // retries are done in updateWifi()
+	WiFi.setHostname(HOSTNAME);
+	WiFi.mode(WIFI_STA);
+	connectStation();
+
+	ArduinoOTA.setHostname(HOSTNAME);
+	ArduinoOTA.onStart([]() {
+		// The update blocks loop() until it is done
+		if (watchdog_running)
+			esp_task_wdt_delete(NULL);
+		watchdog_running = false;
+		Serial.println("OTA update started");
+	});
+	ArduinoOTA.onEnd([]() { Serial.println("OTA update done, starting the new firmware"); });
+	ArduinoOTA.onError([](ota_error_t error) {
+		Serial.printf("OTA error %u, restarting\n", error);
+		ESP.restart();
+	});
+	ArduinoOTA.begin();
+
+	portalBegin();
+
+	// A hang restarts the ESP32; the bootloader then starts this recovery app again
+	esp_task_wdt_init(RECOVERY_WATCHDOG_S, true);
+	esp_task_wdt_add(NULL);
+	watchdog_running = true;
+}
+
+void loop()
+{
+	if (watchdog_running)
+		esp_task_wdt_reset();
+
+	ArduinoOTA.handle();
+	portalLoop();
+	updateWifi();
+	updateLed();
+
+	bool wifi_connected = WiFi.status() == WL_CONNECTED;
+	bool automatic = !recovery.stay &&
+					 recoveryShouldStartMain(recovery.reason, millis(), wifi_connected, main_available);
+	if (recovery.start_main || automatic)
+	{
+		recovery.start_main = false;
+		startMainFirmware(); // returns only when no slot holds a valid firmware
+		recovery.stay = true;
+	}
+	delay(2);
+}
@@ -0,0 +1,20 @@
+#pragma once
+
+#include <stdint.h>
+
+#include "logic/boot_policy.h"
+
+// State of the recovery app, shared by main.cpp and the web portal
+struct RecoveryState
+{
+	RecoveryReason reason = RecoveryReason::None;
+	bool stay = false;       // "Stay in recovery" pressed: never start the clock firmware automatically
+	bool ap_running = false; // the FlipDot-Setup access point is open
+	bool start_main = false; // "Start clock firmware" pressed
+	uint32_t station_attempt = 0; // millis() of the last WiFi.begin()
+};
+
+extern RecoveryState recovery;
+
+// Connect (again) to the WiFi stored in NVS
+void connectStation();
@@ -0,0 +1,204 @@
+#include "web_portal.h"
+
+#include <DNSServer.h>
+#include <WebServer.h>
+#include <WiFi.h>
+
+#include "config.h"
+#include "recovery_state.h"
+#include "system/boot_target.h"
+#include "system/system_store.h"
+
+static WebServer server(80);
+static DNSServer dns;
+static bool dns_running = false;
+
+// Network scan, refreshed when the page is opened and the last scan is old
+static const uint32_t SCAN_MAX_AGE = 30000;
+static uint32_t last_scan = 0;
+static bool scan_started = false;
+
+static String htmlEscape(const String &text)
+{
+	String out;
+	out.reserve(text.length());
+	for (char c : text)
+	{
+		switch (c)
+		{
+		case '&': out += "&amp;"; break;
+		case '<': out += "&lt;"; break;
+		case '>': out += "&gt;"; break;
+		case '"': out += "&quot;"; break;
+		case '\'': out += "&#39;"; break;
+		default: out += c;
+		}
+	}
+	return out;
+}
+
+static void startScanIfOld()
+{
+	if (WiFi.scanComplete() == WIFI_SCAN_RUNNING)
+		return;
+	if (!scan_started || millis() - last_scan >= SCAN_MAX_AGE)
+	{
+		WiFi.scanDelete();
+		WiFi.scanNetworks(true); // async: the page shows the previous result meanwhile
+		scan_started = true;
+		last_scan = millis();
+	}
+}
+
+static String networkOptions()
+{
+	String options;
+	int count = WiFi.scanComplete();
+	for (int i = 0; i < count; i++)
+	{
+		String ssid = htmlEscape(WiFi.SSID(i));
+		if (ssid.isEmpty())
+			continue;
+		options += "<option value=\"" + ssid + "\">" + ssid + " (" + WiFi.RSSI(i) + " dBm)</option>";
+	}
+	return options;
+}
+
+static String page(const String &title, const String &body)
+{
+	return "<!doctype html><html><head><meta charset=\"utf-8\">"
+		   "<meta name=\"viewport\" content=\"width=device-width,initial-scale=1\">"
+		   "<title>" + title + "</title><style>"
+		   "body{font-family:sans-serif;max-width:32em;margin:1em auto;padding:0 1em;line-height:1.4}"
+		   "h1{font-size:1.4em}h2{font-size:1.1em;margin-top:1.5em}"
+		   "input,select,button{font-size:1em;padding:.4em;margin:.2em 0;width:100%;box-sizing:border-box}"
+		   "button{cursor:pointer}td{padding:.1em .6em .1em 0;vertical-align:top}code{background:#eee;padding:0 .2em}"
+		   "</style></head><body><h1>" + title + "</h1>" + body + "</body></html>";
+}
+
+static String row(const String &label, const String &value)
+{
+	return "<tr><td>" + label + "</td><td>" + value + "</td></tr>";
+}
+
+static void handleRoot()
+{
+	startScanIfOld();
+
+	String ssid, password;
+	loadWifiCredentials(ssid, password);
+	bool connected = WiFi.status() == WL_CONNECTED;
+	String main_slot = loadMainSlot();
+
+	String body = "<p>The clock is paused. Reason: <b>" + String(recoveryReasonText(recovery.reason)) + "</b>.</p><table>";
+	body += row("WiFi", connected ? "connected to " + htmlEscape(WiFi.SSID()) + ", " + WiFi.localIP().toString()
+								  : "not connected (trying " + htmlEscape(ssid) + ")");
+	if (recovery.ap_running)
+		body += row("Setup WiFi", String(SETUP_AP_SSID) + ", " + WiFi.softAPIP().toString());
+	body += row("ota_0", slotDescription("ota_0") + (main_slot == "ota_0" ? " (last used)" : ""));
+	body += row("ota_1", slotDescription("ota_1") + (main_slot == "ota_1" ? " (last used)" : ""));
+
+	String next;
+	if (recovery.stay)
+		next = "stays in recovery until you start it";
+	else if (recovery.reason == RecoveryReason::Wifi)
+		next = "starts as soon as the WiFi works";
+	else
+		next = "starts automatically in " + String((RECOVERY_TIME - min<uint32_t>(millis(), RECOVERY_TIME)) / 60000 + 1) + " min";
+	body += row("Clock", next);
+	body += "</table>";
+
+	body += "<form method=\"post\" action=\"/start\"><button>Start the clock firmware now</button></form>";
+	if (!recovery.stay)
+		body += "<form method=\"post\" action=\"/stay\"><button>Stay in recovery</button></form>";
+
+	body += "<h2>WiFi</h2><form method=\"post\" action=\"/wifi\">"
+			"<select onchange=\"document.getElementById('s').value=this.value\">"
+			"<option value=\"\">Networks nearby...</option>" + networkOptions() + "</select>"
+			"<input id=\"s\" name=\"ssid\" placeholder=\"WiFi name\" value=\"" + htmlEscape(ssid) + "\" required>"
+			"<input name=\"password\" type=\"password\" placeholder=\"Password\">"
+			"<button>Save and connect</button></form>";
+
+	String ip = connected ? WiFi.localIP().toString() : WiFi.softAPIP().toString();
+	body += "<h2>Firmware update</h2><p>Upload a new clock firmware from the PC:<br>"
+			"<code>pio run -e esp32 -t upload --upload-port " + ip + "</code><br>"
+			"It is written to ota_0 and starts right after the upload.</p>";
+
+	server.send(200, "text/html", page("FlipDot recovery", body));
+}
+
+static void handleWifi()
+{
+	String ssid = server.arg("ssid");
+	String password = server.arg("password");
+	ssid.trim();
+	if (ssid.isEmpty())
+	{
+		server.send(400, "text/html", page("FlipDot recovery", "<p>Please enter a WiFi name.</p><p><a href=\"/\">Back</a></p>"));
+		return;
+	}
+	saveWifiCredentials(ssid, password);
+	server.send(200, "text/html",
+				page("FlipDot recovery",
+					 "<p>Saved. Connecting to <b>" + htmlEscape(ssid) + "</b>...</p>"
+					 "<p>When it works, the clock starts again (if the WiFi was the problem). "
+					 "If not, connect to <b>" SETUP_AP_SSID "</b> again and check the password.</p>"
+					 "<p><a href=\"/\">Back</a></p>"));
+	delay(500); // let the page reach the browser before the WiFi changes
+	connectStation();
+}
+
+static void handleStart()
+{
+	server.send(200, "text/html", page("FlipDot recovery", "<p>Starting the clock firmware...</p>"));
+	delay(500);
+	recovery.start_main = true;
+}
+
+static void handleStay()
+{
+	recovery.stay = true;
+	server.sendHeader("Location", "/");
+	server.send(303);
+}
+
+// Captive portal: on the setup WiFi every unknown page or host leads to the recovery page
+static void handleNotFound()
+{
+	if (recovery.ap_running && server.client().localIP() == WiFi.softAPIP())
+	{
+		server.sendHeader("Location", "http://" + WiFi.softAPIP().toString() + "/");
+		server.send(302);
+		return;
+	}
+	server.send(404, "text/plain", "Not found");
+}
+
+void portalBegin()
+{
+	server.on("/", HTTP_GET, handleRoot);
+	server.on("/wifi", HTTP_POST, handleWifi);
+	server.on("/start", HTTP_POST, handleStart);
+	server.on("/stay", HTTP_POST, handleStay);
+	server.onNotFound(handleNotFound);
+	server.begin();
+}
+
+void portalLoop()
+{
+	if (dns_running)
+		dns.processNextRequest();
+	server.handleClient();
+}
+
+void portalStartCaptiveDns()
+{
+	dns.start(53, "*", WiFi.softAPIP());
+	dns_running = true;
+}
+
+void portalStopCaptiveDns()
+{
+	dns.stop();
+	dns_running = false;
+}
@@ -0,0 +1,15 @@
+#pragma once
+
+// Web page of the recovery app (port 80), reachable on the normal WiFi and on the
+// FlipDot-Setup access point (http://192.168.4.1):
+//   - status: why recovery started, WiFi, clock firmware in ota_0 / ota_1
+//   - choose another WiFi (with a scan of the networks nearby)
+//   - start the clock firmware now, or stay in recovery
+//   - how to upload a new firmware over OTA
+void portalBegin();
+void portalLoop();
+
+// While the access point is open, all DNS names point to this page, so phones open it
+// automatically after connecting
+void portalStartCaptiveDns();
+void portalStopCaptiveDns();
@@ -7,6 +7,7 @@

 #include <algorithm>
 #include <cstdio>
+#include <cstring>
 #include <string>
 #include <vector>

@@ -247,6 +248,48 @@ void test_crash_count_cleared_after_running_long_enough()
 	TEST_ASSERT_TRUE(shouldClearCrashCount(CRASH_FREE_TIME));
 }

+void test_quick_power_cycles_request_recovery()
+{
+	for (uint8_t count = 0; count < POWER_CYCLES_FOR_RECOVERY; count++)
+		TEST_ASSERT_FALSE(powerCyclesRequestRecovery(count));
+	TEST_ASSERT_TRUE(powerCyclesRequestRecovery(POWER_CYCLES_FOR_RECOVERY));
+}
+
+void test_recovery_for_wifi_starts_clock_when_wifi_works()
+{
+	TEST_ASSERT_FALSE(recoveryShouldStartMain(RecoveryReason::Wifi, 0, false, true));
+	TEST_ASSERT_FALSE(recoveryShouldStartMain(RecoveryReason::Wifi, RECOVERY_TIME * 10, false, true));
+	TEST_ASSERT_FALSE(recoveryShouldStartMain(RecoveryReason::Wifi, RECOVERY_MIN_TIME - 1, true, true));
+	TEST_ASSERT_TRUE(recoveryShouldStartMain(RecoveryReason::Wifi, RECOVERY_MIN_TIME, true, true));
+}
+
+void test_recovery_for_other_reasons_waits_recovery_time()
+{
+	const RecoveryReason reasons[] = {RecoveryReason::Crash, RecoveryReason::Manual,
+									  RecoveryReason::PowerCycle, RecoveryReason::None};
+	for (RecoveryReason reason : reasons)
+	{
+		TEST_ASSERT_FALSE(recoveryShouldStartMain(reason, RECOVERY_TIME - 1, true, true));
+		TEST_ASSERT_TRUE(recoveryShouldStartMain(reason, RECOVERY_TIME, true, true));
+		// The time counts, not the WiFi: an OTA upload over the setup WiFi is possible
+		TEST_ASSERT_TRUE(recoveryShouldStartMain(reason, RECOVERY_TIME, false, true));
+	}
+}
+
+void test_recovery_never_starts_missing_clock_firmware()
+{
+	TEST_ASSERT_FALSE(recoveryShouldStartMain(RecoveryReason::Wifi, RECOVERY_TIME, true, false));
+	TEST_ASSERT_FALSE(recoveryShouldStartMain(RecoveryReason::Manual, RECOVERY_TIME, true, false));
+}
+
+void test_every_recovery_reason_has_a_text()
+{
+	for (uint8_t r = 0; r <= (uint8_t)RecoveryReason::PowerCycle; r++)
+		TEST_ASSERT_TRUE(strlen(recoveryReasonText((RecoveryReason)r)) > 0);
+	TEST_ASSERT_TRUE(strcmp(recoveryReasonText(RecoveryReason::Wifi),
+							recoveryReasonText(RecoveryReason::Crash)) != 0);
+}
+
 // ---------------------------------------------------------------------------
 // Status LED
 // ---------------------------------------------------------------------------
@@ -434,6 +477,12 @@ int main()
 	RUN_TEST(test_new_firmware_needs_time_and_wifi_to_be_accepted);
 	RUN_TEST(test_crash_count_cleared_after_running_long_enough);

+	RUN_TEST(test_quick_power_cycles_request_recovery);
+	RUN_TEST(test_recovery_for_wifi_starts_clock_when_wifi_works);
+	RUN_TEST(test_recovery_for_other_reasons_waits_recovery_time);
+	RUN_TEST(test_recovery_never_starts_missing_clock_firmware);
+	RUN_TEST(test_every_recovery_reason_has_a_text);
+
 	RUN_TEST(test_led_off_while_display_off);
 	RUN_TEST(test_led_blinks_slow_with_wifi_and_fast_without);