diff --git a/.gitignore b/.gitignore index 652a243..610489c 100644 --- a/.gitignore +++ b/.gitignore @@ -20,3 +20,6 @@ desktop.ini *.swp *~ *.bak + +# Python +__pycache__/ diff --git a/CLAUDE.md b/CLAUDE.md index 934f15b..2893041 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -16,6 +16,14 @@ See README.md for the overview. or `$env:USERPROFILE\.platformio\penv\Scripts\pio.exe run` (PowerShell). - If an ESP32 build fails with `No module named 'intelhex'`, run `~/.platformio/penv/Scripts/python.exe -m pip install intelhex`. +- `nano_ota` flashes the Nano over WiFi: `tools/nano_net_upload.py` (STK500v1 over TCP) + talks to `NanoFlashBridge` on the ESP32 (port 2323), which power cycles the Nano into + its old bootloader (57600 baud) and passes the bytes through. avrdude 6.3's `net:` port is + broken on Windows (`ser_drain()` fails), which is why the uploader is a Python script. + The uploader only uses the Python standard library. Test it against a simulated + bootloader before changing it. +- `src/esp32/libs.h` defines globals in the header, so it can only be included from + `main.cpp`. New ESP32 modules must be self-contained (see `nano_flash_bridge.*`). - After any change, build both targets (`pio run`). After Nano changes, also run `pio test -e native`. diff --git a/README.md b/README.md index ba4c975..01b0910 100644 --- a/README.md +++ b/README.md @@ -40,6 +40,7 @@ src/esp32/ ESP32 firmware secrets.example.h template for WiFi / MQTT credentials test/test_nano/ unit tests for the Nano firmware (run on the PC) test/mocks/ fake Arduino, EEPROM and display driver for the tests +tools/nano_net_upload.py uploader for flashing the Nano over WiFi archive/ old, unused sketches kept for reference CLAUDE.md project notes for Claude Code ``` @@ -57,12 +58,14 @@ this folder. | Environment | Board | Upload method | |-------------|-------|---------------| | `nano` | Arduino Nano (ATmega328P) | USB (`COM10`) | +| `nano_ota` | Arduino Nano (ATmega328P) | Over WiFi through the ESP32 (`10.1.1.198`). See [Flashing the Nano over WiFi](#flashing-the-nano-over-wifi). | | `esp32` | ESP32 DevKit v1 | OTA over WiFi (`10.1.1.198`) | | `esp32_usb` | ESP32 DevKit v1 | USB (`COM11`). Use it for the first flash or for recovery. | ```sh pio run # build nano + esp32 -pio run -e nano -t upload # flash the Nano +pio run -e nano -t upload # flash the Nano over USB +pio run -e nano_ota -t upload # flash the Nano over WiFi, through the ESP32 pio run -e esp32 -t upload # flash the ESP32 over the air pio run -e esp32_usb -t upload # flash the ESP32 over USB pio device monitor -e nano # serial monitor @@ -76,7 +79,60 @@ In VS Code you can also pick the environment in the PlatformIO status bar. > **Tip:** The ESP32 → Nano wire goes to the Nano's RX pin, which the Nano's USB chip > also uses. Disconnect that wire before you flash the Nano over USB, or the upload may -> fail. +> fail. Flashing over WiFi (`nano_ota`) doesn't have this problem. + +## Flashing the Nano over WiFi + +The ESP32 can flash the Nano for you, so you don't need a USB cable at the display: + +```sh +pio run -e nano_ota -t upload +``` + +What happens: + +1. The uploader ([tools/nano_net_upload.py](tools/nano_net_upload.py)) connects to the + ESP32 on TCP port **2323**. +2. The ESP32 pauses the clock and switches the relay off for 1 second, then on again. This + restarts the Nano, and its bootloader starts. +3. The ESP32 switches its link to the bootloader's 57600 baud and passes bytes between + WiFi and the Nano. +4. The uploader writes the new firmware and reads it back to verify it. +5. When the uploader disconnects, the ESP32 switches the relay off. The clock powers the + Nano on again with the new firmware and continues. + +The display goes dark for the duration of the upload, which takes a few seconds. + +**Requirements** + +- The reply wire must be connected: Nano D1 (TX) → voltage divider → ESP32 GPIO 16. The + uploader needs the bootloader's answers. +- The ESP32 must be running firmware with the flash bridge (this version or newer). +- The Nano must have the **old bootloader** (the `nanoatmega328` board, 57600 baud), which + also runs at power-on. The newer Optiboot bootloader only runs after a reset-pin reset, + so the relay power cycle would not start it. + +**Settings** + +- ESP32 IP address: `upload_port` in the `nano_ota` environment in `platformio.ini`. You + can add a port number: `10.1.1.198:2323`. For a one-time different address, use + `pio run -e nano_ota -t upload --upload-port 10.1.1.50`. +- Port, baud rate and power-off time: `NANO_FLASH_PORT`, `NANO_BOOTLOADER_BAUD` and + `NANO_POWER_OFF_TIME` in `src/esp32/libs.h`. + +**If it fails** + +| Message | Cause | +|---------|-------| +| `can't connect to ...` | Wrong IP address, or the ESP32 is offline or running older firmware | +| `no answer from the Nano bootloader` | Reply wire missing, or the Nano has Optiboot | +| `bad reply to command ...` / `verify failed` | Connection problem. Try again. A failed upload can leave the Nano without working firmware, but the bootloader itself is never overwritten, so flashing again (over WiFi or USB) always fixes it. | + +The ESP32 ends the session after 10 seconds without data, so the clock always resumes, +even if the uploader is interrupted. + +> **Security:** anyone on your network can connect to port 2323 and flash the Nano, just +> like the ESP32's own OTA has no password. Keep the display on a trusted network. ## Wiring between ESP32 and Nano @@ -86,7 +142,7 @@ The ESP32 talks to the Nano on **UART2** (`Serial2`, 115200 baud). The ESP32's o | ESP32 | Nano | Notes | |-------|------|-------| | GPIO 17 (TX2) | D0 (RX) | Display commands | -| GPIO 16 (RX2) | D1 (TX) | `READY` and `ERR` replies. Use a 5 V → 3.3 V voltage divider (e.g. 1 kΩ / 2 kΩ), because the ESP32 is not 5 V tolerant. Optional: without it, the ESP waits a fixed 3 s after power-on. | +| GPIO 16 (RX2) | D1 (TX) | `READY` and `ERR` replies, and bootloader answers when flashing over WiFi. Use a 5 V → 3.3 V voltage divider (e.g. 1 kΩ / 2 kΩ), because the ESP32 is not 5 V tolerant. Without it, the ESP waits a fixed 3 s after power-on, and `nano_ota` doesn't work. | | GND | GND | Common ground | | GPIO 13 | Relay input | Relay powers the panels and the Nano (`LOW` = on) | @@ -118,7 +174,11 @@ The serial protocol is described in [src/nano/README.md](src/nano/README.md). you flash this firmware, or the display gets no data. See [Wiring between ESP32 and Nano](#wiring-between-esp32-and-nano). - [ ] **Add the reply wire:** Nano D1 (TX) → voltage divider → ESP32 GPIO 16 (RX2). With it, - the ESP sends the first frame as soon as the Nano is ready, instead of after 3 s. + the ESP sends the first frame as soon as the Nano is ready, instead of after 3 s, and + the Nano can be flashed over WiFi (`nano_ota`). +- [ ] **Test flashing over WiFi on the real hardware.** The uploader is tested against a + simulated bootloader, not yet against the real Nano. Check especially that the old + bootloader starts after the relay power cycle. - [ ] **Check the boot timing.** Build the Nano with `MEASURE_REFRESH` to see how long a full redraw takes. If the reply wire isn't fitted, check that 3 s (`NANO_BOOT_TIMEOUT`) is enough for the Nano to boot. diff --git a/platformio.ini b/platformio.ini index 91531f9..27296fd 100644 --- a/platformio.ini +++ b/platformio.ini @@ -1,7 +1,8 @@ ; Flipdot display - one PlatformIO project, two firmware targets. ; -; nano Arduino Nano that drives the Brose flipdot panels directly -; esp32 ESP32 "brain" (WiFi, NTP clock, Home Assistant) - OTA upload +; nano Arduino Nano that drives the Brose flipdot panels directly (USB upload) +; nano_ota Same Nano firmware, flashed over WiFi through the ESP32 +; esp32 ESP32 "brain" (WiFi, NTP clock, Home Assistant) - OTA upload ; esp32_usb Same ESP32 firmware, flashed over USB (first flash / recovery) ; native Unit tests for the Nano firmware, run on the PC (needs gcc on PATH) ; @@ -38,6 +39,14 @@ build_src_filter = + upload_port = COM10 monitor_port = COM10 +; Same Nano firmware, flashed over WiFi through the ESP32 (see README.md). +; Needs the Nano TX -> ESP32 GPIO 16 wire. upload_port = ESP32 IP[:port], default port 2323. +[env:nano_ota] +extends = env:nano +upload_protocol = custom +upload_port = 10.1.1.198 +upload_command = "$PYTHONEXE" "$PROJECT_DIR/tools/nano_net_upload.py" "$SOURCE" $UPLOAD_PORT + ; --------------------------------------------------------------------------- ; ESP32 - controller, talks to the Nano over serial ; --------------------------------------------------------------------------- diff --git a/src/esp32/README.md b/src/esp32/README.md index 1b68a46..8ee3166 100644 --- a/src/esp32/README.md +++ b/src/esp32/README.md @@ -14,6 +14,7 @@ debug output. Wiring is described in the [main README](../../README.md#wiring-be | `main.cpp` | setup, OTA, clock / power logic, Home Assistant callbacks | | `libs.h` | includes and global objects (display, MQTT, WiFi, pins, timezone) | | `bitmaps.h` | 1-bit bitmaps (Pikachu, thought bubble, Nintendo logo, ...) | +| `nano_flash_bridge.h/.cpp` | WiFi to serial bridge for flashing the Nano over WiFi (TCP port 2323) | | `secrets.example.h` | template for `secrets.h` (WiFi and MQTT credentials, **gitignored**) | ## Features @@ -34,6 +35,9 @@ debug output. Wiring is described in the [main README](../../README.md#wiring-be - **Off:** the ESP stops sending and releases its TX pin (`Serial.end()` and the pin set to input). An idle UART line is high and would otherwise power the Nano through its RX pin. +- **Nano flash bridge:** `pio run -e nano_ota -t upload` flashes the Nano through the ESP32. + While a flash session runs, the clock is paused. See the + [main README](../../README.md#flashing-the-nano-over-wifi). - **Status LED** (built-in) blinks fast when WiFi is down and slowly when connected. It is off at night or when the display is powered off. diff --git a/src/esp32/libs.h b/src/esp32/libs.h index a2e9765..2b3bddc 100644 --- a/src/esp32/libs.h +++ b/src/esp32/libs.h @@ -1,6 +1,7 @@ #include #include "BROSE9323.h" #include "bitmaps.h" +#include "nano_flash_bridge.h" // WiFi / MQTT credentials (gitignored, copy secrets.example.h -> secrets.h) #if __has_include("secrets.h") @@ -52,6 +53,12 @@ const int nano_tx = 17; // UART2 TX, wired to the Nano RX bool relay_on = false; // Max wait for the Nano's READY after power on (also the wait when Nano TX is not wired to our RX) const uint32_t NANO_BOOT_TIMEOUT = 3000; +const uint32_t NANO_BAUD = 115200; + +// Flashing the Nano over WiFi (see README.md, pio run -e nano_ota -t upload) +const uint16_t NANO_FLASH_PORT = 2323; +const uint32_t NANO_BOOTLOADER_BAUD = 57600; // old Nano bootloader +const uint32_t NANO_POWER_OFF_TIME = 1000; // relay off time to make sure the Nano resets int hour = 0; int minute = 0; diff --git a/src/esp32/main.cpp b/src/esp32/main.cpp index 05d17e4..4253cbc 100644 --- a/src/esp32/main.cpp +++ b/src/esp32/main.cpp @@ -39,7 +39,7 @@ void relayOn() if (relay_on) return; - nanoSerial.begin(115200, SERIAL_8N1, nano_rx, nano_tx); + nanoSerial.begin(NANO_BAUD, SERIAL_8N1, nano_rx, nano_tx); // drop old data (e.g. ERR replies) so only a new READY counts while (nanoSerial.available()) nanoSerial.read(); @@ -59,6 +59,30 @@ void relayOff() nanoSerial.end(); pinMode(nano_tx, INPUT); } + +// Flash session started: power cycle the Nano so its bootloader runs, and talk to the +// bootloader at its baud rate. The PC uploader keeps sending sync bytes until it answers. +void startNanoFlash() +{ + if (relay_on) + relayOff(); + delay(NANO_POWER_OFF_TIME); + + nanoSerial.begin(NANO_BOOTLOADER_BAUD, SERIAL_8N1, nano_rx, nano_tx); + while (nanoSerial.available()) + nanoSerial.read(); + digitalWrite(relay, LOW); +} + +// Flash session ended: switch off, the clock powers the Nano on again with the new firmware +void endNanoFlash() +{ + relayOff(); + // skip the frames that were missed while flashing + next_display = millis(); +} + +NanoFlashBridge nanoFlashBridge(NANO_FLASH_PORT, nanoSerial, startNanoFlash, endNanoFlash); void turnPowerOff(uint32_t delay = 30000, bool _turn_hassio_power_off = false) { if(triggered_power_off == false) @@ -269,6 +293,7 @@ void setup() }); ArduinoOTA.begin(); + nanoFlashBridge.begin(); do { @@ -441,6 +466,11 @@ void loop() { ArduinoOTA.handle(); mqtt.loop(); + + // While the Nano is being flashed, nothing else may talk to it + if (nanoFlashBridge.handle()) + return; + updatePower(); wifiStatus = WiFi.waitForConnectResult() == WL_CONNECTED; diff --git a/src/esp32/nano_flash_bridge.cpp b/src/esp32/nano_flash_bridge.cpp new file mode 100644 index 0000000..bb74389 --- /dev/null +++ b/src/esp32/nano_flash_bridge.cpp @@ -0,0 +1,66 @@ +#include "nano_flash_bridge.h" + +NanoFlashBridge::NanoFlashBridge(uint16_t port, HardwareSerial &serial, void (*onStart)(), void (*onEnd)()) + : _server(port), _serial(serial), _onStart(onStart), _onEnd(onEnd) +{ +} + +void NanoFlashBridge::begin() +{ + _server.begin(); + _server.setNoDelay(true); +} + +bool NanoFlashBridge::handle() +{ + if (!_active) + { + WiFiClient client = _server.available(); + if (!client) + return false; + + _client = client; + _client.setNoDelay(true); + _active = true; + _onStart(); + _last_activity = millis(); + } + + if (!_client.connected() || millis() - _last_activity > IDLE_TIMEOUT) + { + _end(); + return false; + } + + uint8_t buf[256]; + + // PC -> Nano + int n = _client.available(); + if (n > 0) + { + n = _client.read(buf, min(n, (int)sizeof(buf))); + if (n > 0) + { + _serial.write(buf, n); + _last_activity = millis(); + } + } + + // Nano -> PC + n = _serial.available(); + if (n > 0) + { + n = _serial.readBytes(buf, min(n, (int)sizeof(buf))); + _client.write(buf, n); + _last_activity = millis(); + } + + return true; +} + +void NanoFlashBridge::_end() +{ + _client.stop(); + _active = false; + _onEnd(); +} diff --git a/src/esp32/nano_flash_bridge.h b/src/esp32/nano_flash_bridge.h new file mode 100644 index 0000000..c63f4a0 --- /dev/null +++ b/src/esp32/nano_flash_bridge.h @@ -0,0 +1,36 @@ +#pragma once + +#include +#include + +// WiFi to serial bridge for flashing the Nano from the PC (pio run -e nano_ota -t upload). +// +// When a TCP client connects, onStart() is called to restart the Nano into its bootloader. +// After that, bytes are copied between the TCP connection and the Nano's serial port +// until the client disconnects or is idle too long. onEnd() is then called. +// The uploader on the PC side is tools/nano_net_upload.py. +class NanoFlashBridge +{ +public: + NanoFlashBridge(uint16_t port, HardwareSerial &serial, void (*onStart)(), void (*onEnd)()); + + void begin(); + + // Call from loop(). Returns true while a flash session is running; the caller must not + // send anything to the Nano then. + bool handle(); + +private: + // Give up when no data moves in either direction for this long + static const uint32_t IDLE_TIMEOUT = 10000; + + WiFiServer _server; + WiFiClient _client; + HardwareSerial &_serial; + void (*_onStart)(); + void (*_onEnd)(); + bool _active = false; + uint32_t _last_activity = 0; + + void _end(); +}; diff --git a/tools/nano_net_upload.py b/tools/nano_net_upload.py new file mode 100644 index 0000000..c9d204b --- /dev/null +++ b/tools/nano_net_upload.py @@ -0,0 +1,208 @@ +#!/usr/bin/env python3 +"""Flash the Arduino Nano over WiFi, through the ESP32's flash bridge. + +Usage: python nano_net_upload.py firmware.hex HOST[:PORT] + +The ESP32 power cycles the Nano when we connect (so its bootloader starts) and then +passes bytes between this TCP connection and the Nano's serial port. This script talks +the STK500v1 protocol of the Arduino bootloader directly. avrdude can't be used here: +its network ports are unreliable on Windows. + +Normally run by PlatformIO: pio run -e nano_ota -t upload +""" + +import socket +import sys +import time + +DEFAULT_PORT = 2323 +PAGE_SIZE = 128 # ATmega328P flash page in bytes +FLASH_SIZE = 30720 # 32 KB minus the 2 KB bootloader +SIGNATURE = b"\x1e\x95\x0f" # ATmega328P + +# STK500v1 +STK_OK = 0x10 +STK_INSYNC = 0x14 +CRC_EOP = 0x20 +GET_SYNC = 0x30 +ENTER_PROGMODE = 0x50 +LEAVE_PROGMODE = 0x51 +LOAD_ADDRESS = 0x55 +PROG_PAGE = 0x64 +READ_PAGE = 0x74 +READ_SIGN = 0x75 + +SYNC_TIMEOUT = 10.0 # seconds to wait for the bootloader after connecting +REPLY_TIMEOUT = 1.0 + + +class UploadError(Exception): + pass + + +def read_hex(path): + """Parse an Intel HEX file into a bytearray (gaps filled with 0xFF).""" + image = bytearray() + base = 0 + with open(path) as f: + for line_no, line in enumerate(f, 1): + line = line.strip() + if not line: + continue + if not line.startswith(":"): + raise UploadError(f"{path}:{line_no}: not an Intel HEX line") + raw = bytes.fromhex(line[1:]) + if sum(raw) & 0xFF: + raise UploadError(f"{path}:{line_no}: checksum error") + count, addr, rtype, data = raw[0], (raw[1] << 8) | raw[2], raw[3], raw[4:4 + raw[0]] + if len(data) != count: + raise UploadError(f"{path}:{line_no}: length mismatch") + if rtype == 0x00: # data + start = base + addr + if start + count > len(image): + image.extend(b"\xff" * (start + count - len(image))) + image[start:start + count] = data + elif rtype == 0x01: # end of file + break + elif rtype == 0x02: # extended segment address + base = ((data[0] << 8) | data[1]) << 4 + elif rtype == 0x04: # extended linear address + base = ((data[0] << 8) | data[1]) << 16 + if len(image) > FLASH_SIZE: + raise UploadError(f"firmware is {len(image)} bytes, the Nano has room for {FLASH_SIZE}") + # pad to whole pages + if len(image) % PAGE_SIZE: + image.extend(b"\xff" * (PAGE_SIZE - len(image) % PAGE_SIZE)) + return image + + +class Bootloader: + def __init__(self, sock): + self.sock = sock + + def _recv(self, n, timeout=REPLY_TIMEOUT): + self.sock.settimeout(timeout) + data = b"" + deadline = time.monotonic() + timeout + while len(data) < n: + remaining = deadline - time.monotonic() + if remaining <= 0: + break + self.sock.settimeout(remaining) + try: + chunk = self.sock.recv(n - len(data)) + except socket.timeout: + break + if not chunk: + raise UploadError("connection closed by the ESP32") + data += chunk + return data + + def _drain(self): + """Throw away everything that is still arriving (e.g. answers to earlier syncs).""" + while self._recv(256, timeout=0.1): + pass + + def command(self, payload, reply_len=0): + """Send a command and return its reply data (between INSYNC and OK).""" + self.sock.sendall(bytes(payload) + bytes([CRC_EOP])) + reply = self._recv(reply_len + 2) + if len(reply) != reply_len + 2 or reply[0] != STK_INSYNC or reply[-1] != STK_OK: + raise UploadError(f"bad reply to command 0x{payload[0]:02x}: {reply.hex() or 'nothing'}") + return reply[1:-1] + + def sync(self): + """Keep sending GET_SYNC until the bootloader answers (it starts after the power cycle).""" + deadline = time.monotonic() + SYNC_TIMEOUT + while time.monotonic() < deadline: + self.sock.sendall(bytes([GET_SYNC, CRC_EOP])) + if self._recv(2, timeout=0.2) == bytes([STK_INSYNC, STK_OK]): + # Answers to earlier syncs may still be on their way: clear them, then + # sync once more so the replies line up with our commands. + self._drain() + self.command([GET_SYNC]) + return + raise UploadError( + f"no answer from the Nano bootloader within {SYNC_TIMEOUT:.0f} s. " + "Is the Nano TX -> ESP32 GPIO 16 wire connected?") + + def load_address(self, byte_addr): + word = byte_addr // 2 + self.command([LOAD_ADDRESS, word & 0xFF, word >> 8]) + + def write_page(self, byte_addr, data): + self.load_address(byte_addr) + self.command([PROG_PAGE, len(data) >> 8, len(data) & 0xFF, ord("F")] + list(data)) + + def read_page(self, byte_addr, length): + self.load_address(byte_addr) + return self.command([READ_PAGE, length >> 8, length & 0xFF, ord("F")], reply_len=length) + + +def progress(label, done, total, step=PAGE_SIZE): + """Print a line every 10% (PlatformIO shows each update on its own line).""" + percent = 100 * done // total + previous = 100 * (done - step) // total + if done == step or done == total or percent // 10 != previous // 10: + print(f"{label} {percent:3d}% ({done}/{total} bytes)", flush=True) + + +def upload(hex_path, host, port): + image = read_hex(hex_path) + print(f"Firmware: {hex_path} ({len(image)} bytes)") + + print(f"Connecting to the ESP32 flash bridge at {host}:{port} ...") + try: + sock = socket.create_connection((host, port), timeout=10) + except OSError as e: + raise UploadError(f"can't connect to {host}:{port}: {e}") + sock.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1) + + with sock: + boot = Bootloader(sock) + print("Waiting for the Nano bootloader (the ESP32 power cycles the display) ...") + boot.sync() + + signature = boot.command([READ_SIGN], reply_len=3) + if signature != SIGNATURE: + raise UploadError(f"unexpected device signature {signature.hex()} (expected ATmega328P)") + print("Found ATmega328P") + + boot.command([ENTER_PROGMODE]) + + for addr in range(0, len(image), PAGE_SIZE): + boot.write_page(addr, image[addr:addr + PAGE_SIZE]) + progress("Writing ", addr + PAGE_SIZE, len(image)) + + for addr in range(0, len(image), PAGE_SIZE): + expected = image[addr:addr + PAGE_SIZE] + actual = boot.read_page(addr, PAGE_SIZE) + if actual != expected: + for i, (a, b) in enumerate(zip(actual, expected)): + if a != b: + raise UploadError( + f"verify failed at 0x{addr + i:04x}: read 0x{a:02x}, expected 0x{b:02x}") + progress("Verifying", addr + PAGE_SIZE, len(image)) + + boot.command([LEAVE_PROGMODE]) + + print("Done. The ESP32 restarts the Nano with the new firmware.") + + +def main(): + if len(sys.argv) != 3: + print(__doc__) + return 2 + hex_path, target = sys.argv[1], sys.argv[2] + host, _, port = target.partition(":") + try: + upload(hex_path, host, int(port) if port else DEFAULT_PORT) + except UploadError as e: + sys.stdout.flush() + print(f"ERROR: {e}", file=sys.stderr) + return 1 + return 0 + + +if __name__ == "__main__": + sys.exit(main())