Flipdot-display/CLAUDE.md

# Flipdot display: notes for Claude

A PlatformIO project with two firmware targets for an 84×19 Brose flipdot clock.
See README.md for the overview.

## Layout and build

- `platformio.ini` has the environments `nano`, `nano_ota`, `esp32` (OTA to 10.1.1.198),
  `esp32_usb` (COM11), and the PC test environments `test_nano`, `test_esp32` and
  `test_driver`. The
  sources are split with `build_src_filter` (`src/nano/`, `src/esp32/`).
- `lib/BROSE9323/` is the shared driver. Its behaviour depends on build flags set in
  `platformio.ini`, not in the header:
  - `ESPSERIAL` (ESP32): draw calls become serial commands.
  - `FLIPDOT_PLCC_ADAPTER` (Nano, optional): alternative pinout.
- PlatformIO CLI is not on PATH. Run `~/.platformio/penv/Scripts/pio.exe run` (Git Bash)
  or `$env:USERPROFILE\.platformio\penv\Scripts\pio.exe run` (PowerShell).
- If an ESP32 build fails with `No module named 'intelhex'`, run
  `~/.platformio/penv/Scripts/python.exe -m pip install intelhex`.
- `nano_ota` flashes the Nano over WiFi: `tools/nano_net_upload.py` (STK500v1 over TCP)
  talks to `NanoFlashBridge` on the ESP32 (port 2323). The uploader first sends
  `FLASH <baud>\n`. The ESP power cycles the Nano (plus an optional reset pulse on
  `NANO_RESET_PIN`), answers `OK\n`, then passes the bytes through at that baud rate.
  `--baud=auto` tries 57600 (old ATmegaBOOT, which also starts after a power cycle), then
  115200 (Optiboot, which only starts after a reset-pin reset, so it needs `NANO_RESET_PIN`).
  avrdude 6.3's `net:` port is broken on Windows (`ser_drain()` fails), which is why the
  uploader is a Python script.
  The uploader only uses the Python standard library. Test it against a simulated
  bootloader before changing it.
- After any change, build both targets (`pio run`) and run the tests
  (`pio test -e test_nano -e test_esp32 -e test_driver`).

## Constraints

- **Nano RAM is tight** (ATmega328P, 2 KB). The frame buffers are allocated with
  `calloc`. Avoid `String`, large stack buffers and new globals in Nano code. Put constant
  data in `PROGMEM`.
- The ESP32 talks to the Nano on `Serial2` (owned by `NanoLink`, GPIO 17 TX / GPIO 16 RX).
  `Serial` (UART0/USB) is free for debug output. Never write debug text to `Serial2`: the
  Nano runs lines that start with s/S/F/D/P/B/R/T/G/N/E/I/C as commands.
- Open hardware TODOs are listed in the README `TODO` section. Update it when one is done.
- The README is the overview; the documentation is in `docs/` (web UI, pages, Home
  Assistant, how it works, wiring, Nano over WiFi, recovery app). Screenshots:
  `tools/make_webui_screenshots.py` (headless Edge) and `tools/make_page_screenshots.py`
  write `Pictures/web-ui/` and `Pictures/pages/`; run them again after UI changes.
- `src/esp32/secrets.h` holds the real credentials and is gitignored. Never commit it.
  Add new secrets to `secrets.example.h` as placeholders.
- The relay (ESP32 GPIO 13, `LOW` = on) powers both the panels and the Nano. Each time it
  switches on, the Nano cold-boots and then sends `READY`. `NanoLink::on()` waits for
  that message, at most `NANO_BOOT_TIMEOUT`. The timeout also covers the case where the
  Nano's TX isn't wired to the ESP's RX.
- `NanoLink::off()` ends `Serial2` and sets its TX pin to input. Otherwise the idle-high TX
  line powers the Nano through its RX pin. Never send display commands while
  `NanoLink::isOn()` is false (`DisplayController` takes care of this).
- The Nano keeps no state over a power cycle except the flip time in EEPROM. The driver
  forces a full redraw on the first `display()` after boot (`_force_next`).
- Hardware is not available here. Builds can be checked, but runtime behaviour can't.

## Nano serial parser (`src/nano/main.cpp`)

- Files: `main.cpp` (setup/loop, refresh), `config.h` (all constants and debug
  defines), `command_parser.*` (protocol, one method per command), `settings.*` (EEPROM).
- Line based and non-blocking: bytes are collected into a 64-character buffer and a
  command runs when `\n` arrives. Bitmap (`B`) pixel data is the exception. It is drawn
  while it arrives, so it can be longer than the buffer. Keep `loop()` free of busy-waits.
- The driver never blocks while flipping: `display()` only starts a run, `update()` flips
  dots in slices of about 1 ms (frames, effects, and the periodic dot refresh from
  `startRefresh()`, which waits while a frame is received or shown). Anything that blocks
  longer than about 5 ms overflows the 64-byte serial receive buffer; lost bytes shift the
  rest of a bitmap. That was the "picture moved up" bug. `test_driver` checks the time per
  `update()` call with a simulated panel.
- Flip time is stored in EEPROM as a magic word at address 0 and a `uint16_t` at
  address 2. Values outside 50–2000 µs are rejected.
- The Nano replies `READY` (at boot), `ERR: ...` and `C xxxx` (checksum of its image, for
  `C`). There is no per-frame acknowledgement: the ESP paces frames by time.
  `nano_link/nano_monitor` reads the replies and sends `C` after frames;
  `logic/frame_check` decides (wrong/missing answer or `READY` while on: send the whole
  image again via `controller.pageChanged()`). Every wait for a reply needs a timeout.
- Driver tests: `pio test -e test_driver` runs the real `BROSE9323.cpp` (Nano build)
  against `test/driver_mocks/` (Arduino, Adafruit GFX, a panel that decodes the pins).
- Unit tests: `pio test -e test_nano` runs `test/test_nano/` on the PC with the fakes in
  `test/mocks/` (Arduino, EEPROM, and a `BROSE9323` that records calls). It needs gcc on
  PATH (Strawberry Perl provides it at `C:\Strawberry\c\bin`). Add a test for every
  protocol change. The mocks only implement what the Nano code uses, so extend them when
  new Arduino or driver functions are called.
- The board environments have `test_ignore = *`, so test firmware is never uploaded. Plain
  `pio test` runs nothing; always pass `-e test_nano -e test_esp32 -e test_driver`.

## ESP32 structure (`src/esp32/`)

- Folders: `logic/` (pure C++, no Arduino includes, tested in `test/test_esp32`),
  `display/` (frames and bitmaps), `nano_link/` (relay, serial, flash bridge), `net/`
  (WiFi/OTA, Home Assistant). Includes are relative to `src/esp32` (`-I src/esp32`), e.g.
  `#include "logic/schedule.h"`.
- `DisplayController` holds the display logic and only uses the `DisplayPower` and
  `Frames` interfaces. Put new behaviour there, behind those interfaces, so it stays
  testable. `logic/` must not include Arduino headers, or `test_esp32` stops building.
- All settings are in `config.h`. Keep it free of Arduino types for the same reason.
- Frames are only sent when the screen changes. The schedule uses 08:00 as the start of
  the day on purpose (that is what the old code did, and the README documents it).
- Slow mode (`SlowModeSettings`, replaced the late-evening quarter hours) lives in
  `DisplayController::_slowClock()`: one frame per update minute (`slowModeUpdate()`), power
  off in between, nothing else drawn (also no page changes). It updates at once after boot,
  night, the Power switch and live mode; rules with `over_slow_mode` suspend it
  (`setSlowModeOverride()`, fed from `PageChoice` in `appPageLoop()`).
- Home Assistant switches are not optimistic: callbacks must call `sender->setState()`.
  The unique ids (`FlipDotDisplayPower`, `FlipDotDisplayMode`, `FlipDotDisplayOffImage`,
  `FlipDotDisplayStopwatch`, `FlipDotDisplaySlowMode`, `FlipDotDisplaySlowModeInterval`,
  ...) and `HA_DEVICE_ID` must stay
  the same, or Home Assistant creates new entities.

## Keeping ESP32 OTA alive (most important requirement)

- The user never wants to need USB for the ESP32 again. Any change must keep OTA working,
  also when the new code crashes or hangs.
- `system/firmware_guard.cpp` overrides the Arduino core's weak `verifyRollbackLater()`
  (C linkage) to return true. A new OTA image stays `PENDING_VERIFY` until it has run
  `FIRMWARE_VALID_TIME` with WiFi. Any restart before that makes the bootloader roll back.
  Check with `nm` that the strong symbol is linked after toolchain or core updates.
- The task watchdog (`WATCHDOG_TIMEOUT_S`) covers `loop()`. ArduinoOTA's `onStart` stops
  it, because an upload blocks `loop()`. Never block `loop()` for 30 s or longer.
- The crash counter lives in `RTC_NOINIT_ATTR` memory. Decisions are in
  `logic/boot_policy.*` (tested). Safe mode runs only WiFi + OTA.
- Keep `otaLoop()` and `firmwareGuardLoop()` at the top of `loop()`, before anything that
  can return early or block. Keep `firmwareGuardBegin()` first in `setup()`: it holds
  the power-cycle trigger for the recovery app.
- Flash layout `partitions.csv`: factory = recovery app (`src/recovery`, env `recovery`,
  USB only), ota_0/ota_1 = clock firmware. `esp32_usb` uploads to 0x110000
  (`board_upload.offset_address`) so it never overwrites the recovery app. Check with
  `pio run -e <env> -t envdump` (`ESP32_APP_OFFSET`) after changing upload settings.
- The recovery app compiles `esp32/logic/boot_policy.cpp`, `esp32/system/system_store.cpp`,
  `esp32/system/boot_target.cpp`, `esp32/nano_link/nano_link.cpp`,
  `esp32/display/status_screen.cpp` and `esp32/display/text.cpp` (see its
  `build_src_filter`). Keep those free of clock-firmware dependencies.
- Status messages (`StatusScreen`) are at most 14 characters per line, 2 lines. The
  recovery app powers the display off after each message (flipdots keep the image). Both apps share NVS namespace `flipdot`. RTC memory is
  not shared between the apps (different layouts).
- `rebootToRecovery()` selects the factory partition, which erases otadata. The recovery
  app later selects the clock slot again (`startMainFirmware()`), so the clock firmware
  is `PENDING_VERIFY` again and must pass the 60 s check, or the bootloader falls back to
  the recovery app.
- The clock firmware still works on the old flash layout (no factory partition):
  `rebootToRecovery()` returns, and the callers fall back to OTA-only mode or a restart.
- Arduino's `esp_app_desc_t` version and date come from the precompiled core, so the
  clock firmware stores `FIRMWARE_VERSION` per slot in NVS for the recovery page.

## ESP32 web UI

- `app.cpp` owns the objects and the actions (settings, page, live mode, restart);
  `web/web_ui.cpp` is the JSON API (endpoints listed in `web_ui.h`), `web/web_page.h` the
  single HTML/JS page (raw string literal in PROGMEM). Keep image work in the browser; the
  ESP32 only handles 418-character hex bitmaps (`logic/bitmap_format.h`, MSB = left dot).
- Settings: `logic/settings.h` struct with defaults = original behaviour, stored as an NVS
  blob with a version (`system/settings_store.cpp`). When the struct changes, bump
  `VERSION` there (old records then fall back to defaults). Only the current version is
  read: 2.15.0 converted all old settings, schedules and pages on the device, and 2.15.1
  dropped the converters. To keep data over a format change, add a frozen copy of the old
  layout with a converter (git history before 2.15.1 has examples), and store the
  converted data only once the firmware is marked good (a rollback can't read it).
- Bitmaps on LittleFS in the `spiffs` partition (`/bitmaps/<name>.bin`, 209 bytes). Names
  are restricted (`validBitmapName`), which also makes them safe in paths and JSON.
- Icons (`logic/icon_format.h`, `display/icon_store`, `/icons/<name>.bin`: width, height,
  rows of (w+7)/8 bytes) are small images that pages use (Image elements, backgrounds at
  the top left); they are never shown on their own and are not in the page lists or HA
  dropdowns. Bitmaps, icons and pages share one namespace. `PageResources::background()`
  tries bitmaps, then icons (expanded at 0,0), so the renderer doesn't know the difference.
- LittleFS is slow: listing a directory (`listBitmaps()`, `listPages()`, `storedNames()`)
  takes about a second with 40 files. Never list per item in a loop. The first icon
  migration did that, hit the 30 s watchdog and crash-looped into the recovery app; work
  that touches many files runs in time-boxed steps over several `loop()` passes.
- The display mirror comes from the driver's shadow buffer (`BROSE9323::shownPixel`).
- Dynamic pages (`logic/page.h`): background per time slot + elements, stored as a
  versioned binary struct (`display/page_store.cpp`, bump `VERSION` when `DynamicPage`
  changes). They are drawn only on the ESP32 (`logic/page_render`), also for the web
  previews, so there is one renderer. Pages and bitmaps share one namespace (checked in
  `app.cpp`); things a page uses can't be deleted. The page "Clock" replaces the
  old hardcoded clock (also the default and the HA "Clock" option). `pageStoreBegin()`
  stores a copy of each built-in page (`logic/builtin_pages`) at boot when none is stored,
  so it can be edited; the built-in one is the fallback, and deleting the copy resets it.
- Pages hold 16 elements (format 7). A `DynamicPage` is about 4 KB: allocate it with `new`, and reset
  it with `clearPage()`, not `page = DynamicPage()` (a temporary on the 8 KB loop stack).
- Game of Life (`logic/life`, `display/life_player`): a page element without fields of its
  own: it reuses `bar_width/bar_height` (area), `format` (edges), `input_unit` (start),
  `text` (pattern / name start), `factor` (density), `scroll_speed`, `auto_unit`, `target`
  (max generations), `bar_max` (hold s); the list is in `logic/page.h`. JSON adds `start`
  and `generations`. One per page: `LifePlayer` keeps one board (about 7.5 KB) and ticks it
  in `FlipdotFrames::dynamicPage()`; reloading a page with the same start
  (`sameLifeStart()`) keeps it running. Rotating lists the icons only when
  `iconListRevision()` changes. The pattern library (RLE) is in the web page; check new
  patterns with a simulation before adding them.
- HA attributes may be paths into lists (`forecast.1.condition`, `net/ha_api`). The
  forecast comes from the template sensor in `docs/home-assistant.md`.
- `examples/home-pages.json` is written by `tools/export_examples.py` from the display.
  Its PRIVATE map replaces private parts of entity ids; extend it for new pages and check
  the output.
- `ha_api` attribute lookups: cast `char` arrays to `const char *` for ArduinoJson keys.
  ArduinoJson 7.4 takes a `char[N]` as a literal of length N-1, which never matches (that
  broke all attributes from 2.9.0 to 2.10.0).
- Moving pages: `renderPage()` returns when the page changes next (scroll step, second);
  `FlipdotFrames` sends those frames as `D1` (no effect) and paces them by the estimated
  receive and flip time, because the Nano has no acknowledgement. Full screen frames go
  through `BROSE9323::showBitmap()`: only the changed dots, as two `B` commands over the
  changed rectangle (`lib/BROSE9323/frame_commands.h`, tested with the Nano parser in
  `test_nano`). Don't send frames pixel by pixel (`s` commands are 7 characters per dot). Character 127 is the degree sign in
  page texts and fonts (UTF-8 ° in JSON).
- What is shown is decided by `logic/page_selector` (rules > manual pick until the next
  block > rotating schedule > chosen page) in `appPageLoop()`; `settings.page` is the chosen
  page (the HA "Mode" dropdown). Schedule and rules are stored in NVS
  (`system/schedule_store.cpp`, bump `VERSION` when `PageSchedule` changes). Pages used by
  the schedule or rules can't be deleted.
- The ESP32 flash slot is about 90% full (web page and fonts are in flash). Check the size
  after adding features; the web page could be served gzipped if it gets tight.
- Fonts use the "FDF1" format of `logic/font.h`; the web page writes the same format.
  `logic/builtin_fonts.cpp` is generated: change `tools/make_builtin_fonts.py` and rerun it.
- Home Assistant values come from HA's REST API (`net/ha_api`). The HTTP requests run in
  their own FreeRTOS task (core 0, `haApiBegin()`), one at a time; everything else
  (entries, watching, values) stays in `loop()`. The task gets copies of what it needs and
  answers through a queue of plain structs: never share Strings or other state with it,
  and never wait for it in `loop()`. The web UI's test button polls (`{"pending":true}`).
  The web server stays in `loop()` on purpose (its handlers touch everything).
- The stopwatch (`logic/stopwatch.h`) counts in real time (UTC ms from `epochMs()`, not
  `millis()`), stored in NVS on every change (`system/stopwatch_store.cpp`, bump `VERSION`
  when `Stopwatch::State` changes). Start, stop and set need the synced time.
- Entity ids may contain `{plex_user}` (`expandEntity()` in `logic/page`), filled in by
  `net/ha_api` with `settings.plex_user` in entity id form (`plexUserSlug()`).
- Bitmaps are cached in RAM (`display/bitmap_store`, also missing names): a LittleFS read
  takes about 100 ms, and pages are drawn for every frame and every web preview. Writes
  must go through `saveBitmap()`/`deleteBitmap()`, which clear the cache. Long JSON lists
  are sent in pieces of about 2 KB (`sendPiece()` in `web_ui.cpp`).
- Icons are cached too (`display/icon_store`, all of them). A save or delete only forgets
  that name (`forget()`) and updates the cached name list in place (`listAdd()`/
  `listRemove()`): clearing it all made the next icon list read every icon again (7 s).
- Storage usage (`bitmapStorageUsage()`, for `/api/status`) is cached: `usedBytes()` walks
  the file system (about 0.5 s). Every store calls `bitmapStorageChanged()` after a write
  or delete; the cache also expires after 60 s.
- The web UI reloads only the lists that changed (`loadLists(only)`, the ESP32 answers one
  request at a time): saving a bitmap, icon or font also reloads the pages when a page
  uses it (`pagesUsing()`, `pagesWithFont()`).
- Check the page's JavaScript after changes: extract the `<script>` part and run
  `node --check` on it (node is installed).

## Git

- Remote: https://mangavalk.com/mangavalk/Flipdot-display.git, branch `main`.