Multi-byte fonts: GB2312, Big5, Shift-JIS, KS C 5601, UTF-8

Font Creator output decoded (lead/trail ranges and gaps in the header, map codes = bytes as
little-endian u16 or Unicode code points) and round-tripped byte for byte; from_ttf(chars=)
builds them; the renderer splits text like the simulator (any byte >= 0x81 starts a pair,
missing characters advance one space) -- pixel-exact for all five encodings.
Intelligent glyph data is 8-byte aligned (was 4). CLI add-font --chars/--chars-file, MCP chars.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
mangavalkandClaude Opus 5.5 committed 2026-11-16 13:42:50 +01:00
1 parent c102157ac2
commit 0138a0ba56
11 files changed
+137 -26

No files matched your search

+4
View File
@@ -33,6 +33,10 @@ Back up the user's file first (copy it) unless they said not to; edits write in
`direction` and `encoding`.
- Text needs a font: `font` is a font id. A new project has **no fonts** -- call `add_font`
first (height in px; no TrueType path = bundled CC0 font). `txt_maxl` must be >= the text length.
- Chinese/Japanese/Korean or UTF-8 text: create the project with that `encoding`, add a font with
`encoding` + `chars` (only the characters you need) and a TrueType file that has them; decode the
project with the same encoding to see the text readably. Pictures: `quality` 1-99 stores JPEG
(Intelligent only, smaller files).
- Pictures: `add_picture` returns an id for `pic` / `picc` (crop) / `pic2` etc. Works for Basic
and Intelligent displays (Intelligent keeps PNG transparency).
- Event code is Nextion script, one line per list item, e.g. `["page settings"]`,
+1 -1
View File
@@ -28,7 +28,7 @@ Every percentage is a count with a stated basis, so it can be checked (details i
| Component types (read, write, create) | **100 %** (31/31) | all types in the Editor's toolbox, incl. 11 Intelligent-only |
| Display series | **40 %** (2/5) | Basic (T) and Intelligent (P) confirmed; Discovery, Enhanced, Edge not sampled |
| Picture formats | **100 %** (2/2) | raw RGB565 (Basic) and run-length + alpha or JPEG for quality < 100 (Intelligent), byte-identical to the Editor (transparent JPEG pictures: near-identical), every display direction |
| Font modes | **100 %** (2/2) | black/white and "anti-aliased" (a header flag; glyphs stay 1-bit, verified in the simulator); Editor fonts with overhanging glyphs round-trip byte-identical |
| Font modes | **100 %** (2/2) | black/white and "anti-aliased" (a header flag; glyphs stay 1-bit, verified in the simulator); single- and multi-byte encodings (GB2312, Big5, Shift-JIS, KS C 5601, UTF-8), Editor fonts round-trip byte-identical |
| Readable attribute names in JSON | **100 %** (31/31 types) | every value list the Editor documents, incl. the Intelligent-only types and load effects |
| Rendering, Basic components | **100 %** (20/20 pixel-exact) | incl. gauges and QR codes (own encoder, no dependencies) |
| Rendering, all components | **94 %** (29/31 pixel-exact) | Intelligent ComboBox and Switch are drawn, but their anti-aliased edges are approximate (~99 % / ~92 % of pixels exact) |
+1 -2
View File
@@ -14,7 +14,7 @@ byte for byte, or a file written by nxhmi was opened in the Editor and showed th
| Components | **31 / 31 types** (20 Basic + 11 Intelligent-only) read/write/create |
| Readable JSON (enum names, colours) | done for every value list the Editor documents |
| Pictures | **complete**: format 0 (Basic) and format 4 (Intelligent, with alpha; run-length or JPEG for quality < 100), all display directions |
| Fonts | **complete** for single-byte encodings: black/white and the anti-aliasing flag, overhanging glyphs, all display directions |
| Fonts | **complete**: single- and multi-byte encodings (GB2312, Big5, Shift-JIS, KS C 5601, UTF-8), black/white and the anti-aliasing flag, overhanging glyphs, all display directions; generation from TrueType for all of them |
| Rendering pages like the Editor | **pixel-identical** for every Basic component and 9 of 11 Intelligent-only ones; ComboBox and Switch approximate at anti-aliased edges |
| New projects from scratch | `project.new_project(model)` for 12 Basic/Intelligent models; the generated demo is byte-identical to the Editor's re-save |
| Compiling `.tft` firmware | **research only** -- not possible yet |
@@ -93,7 +93,6 @@ byte for byte, or a file written by nxhmi was opened in the Editor and showed th
### `.HMI` side
| Item | Notes |
|---|---|
| Multi-byte encodings (GB2312, Big5, ...) | only single-byte fonts studied |
| Other display series | Discovery / Enhanced / Edge not sampled; only Basic + Intelligent templates |
| Display direction | byte 12 decoded and written (`new_project(portrait=...)`); picture order verified for all four (format 4); font glyph order for 90° only |
| Some header fields | main.HMI bytes 8, 13, 36, 0x41; page header 21/23; entry bytes 25-27 |
+26 -5
View File
@@ -266,12 +266,12 @@ Character map: 10 bytes per character: u16 code, u8 glyph advance width, u8 extr
left, u8 extra columns on the right (glyphs that overhang, e.g. `j`, `f`, `W`; the stored glyph is
advance + left + right wide and drawn starting `left` px before the pen position),
u24 glyph offset (relative to the map start), u16 glyph length. In Intelligent projects the glyph
data starts 4-byte aligned (zero padding after the map); Basic fonts have no padding.
data starts at a multiple of 8 from the map start (zero padding after the map); Basic fonts have no padding.
Glyph: mode byte (0x01 = black/white), then run-length codes over the pixels in the same order
as pictures for the project's display direction (runs continue across rows/columns): at 90° column
by column, bottom to top; at 0° row by row. When an Intelligent project is compiled, the Editor
converts a 90° font to the project direction; our encoder reproduces that conversion byte for byte.
as pictures for the project's display direction (runs continue across rows/columns): at 90° column
by column, bottom to top; at 0° row by row. When an Intelligent project is compiled, the Editor
converts a 90° font to the project direction; our encoder reproduces that conversion byte for byte.
| Code | Meaning |
|---|---|
@@ -286,7 +286,28 @@ converts a 90
The Editor encodes greedily (transparent run + 1..4 ink if possible, else the 3+3-bit form, else a
plain run); our encoder reproduces a 224-glyph Editor font byte for byte. The v3/v5 write-ups in
github.com/hagronnestad/nextion-font-editor were a useful starting point; v6 differs (scan order,
B/W codes `10`/`11`). **OPEN**: multi-byte encodings, the remaining unknown header bytes.
B/W codes `10`/`11`). **OPEN**: the remaining unknown header bytes (0x1F, 0x20).
Multi-byte fonts (Font Creator "Specified character" range; verified for GB2312, Big5, Shift-JIS,
KS C 5601 and UTF-8, round-trip byte-identical, rendered pixel-exact in the simulator):
| Header | Meaning | GB2312 | Big5 | Shift-JIS | KS C 5601 | UTF-8 |
|---|---|---|---|---|---|---|
| 1, 2 | end of the first trail-byte range, trail codes skipped after it | FF 00 | 7E 22 | 7E 01 | FF 00 | FF 00 |
| 4 | encoding (main.HMI numbering) | 2 | 16 | 23 | 15 | 24 |
| 5 | 2 = multi-byte (0 for single-byte fonts) | 2 | 2 | 2 | 2 | 2 |
| 6 | width (0 for multi-byte) | 0 | 0 | 0 | 0 | 0 |
| 8, 9 | lead-byte range | A1 F7 | A0 F9 | 81 EF | A1 C8 | FF FF |
| 10, 11 | trail-byte range | A1 FE | 40 FE | 40 FC | A1 FE | 00 FF |
| 0x1C, 0x1D | end of the first lead range, lead codes skipped | FF 00 | FF 00 | 9F 40 | FF 00 | FF 00 |
| 0x20 | ? | 4 | 5 | 5 | 4 | 5 |
Map codes: double-byte encodings store the character's two bytes as a little-endian u16 (GB2312
中 = bytes D6 D0 -> 0xD0D6); UTF-8 fonts store the Unicode code point. The map is sorted by
code. The display splits text into characters by taking every byte >= 0x81 together with the
next byte (not limited to header 8-9: KS C hanja leads above C8 still pair; Shift-JIS half-width
katakana pair up as well, so they cannot be shown); a character without a glyph advances by the
width of the space glyph.
## 6. Not yet covered
+2
View File
@@ -74,6 +74,8 @@ TextSelects, several of each other type):
blended as `bg + round((src - bg) * alpha / 255)` per channel. Exact against the Intelligent
simulator (342 samples). The Editor *canvas* blends slightly differently (up to 3 levels off at
high alpha); the renderer follows the simulator, i.e. the display.
- Text in multi-byte encodings: every byte >= 0x81 starts a two-byte character; characters the
font lacks advance by the space glyph's width (measured in the Intelligent simulator).
- Text never starts above/left of its box: centring clamps at 0 (a 32-px font in a 30-px box
starts at the top).
- Text is laid out inside the border: 1 px for 3D styles, `borderw` for style "border".
+9 -2
View File
@@ -116,7 +116,11 @@ def cmd_new_page(a):
def cmd_add_font(a):
from . import fonts, project
h = HMIFile.load(a.hmi)
f = fonts.from_ttf(a.ttf, a.height, a.encoding, a.fixed_width)
chars = a.chars
if a.chars_file:
with open(a.chars_file, encoding="utf-8") as cf:
chars = (chars or "") + cf.read()
f = fonts.from_ttf(a.ttf, a.height, a.encoding, a.fixed_width, chars=chars)
fid = project.add_font(h, f)
h.save(a.out or a.hmi)
print("added font id %d (%s, %d chars)" % (fid, f.name, len(f.glyphs)))
@@ -192,7 +196,10 @@ def main(argv=None):
s = sub.add_parser("kinds", help="list component kinds that can be added"); s.set_defaults(f=cmd_kinds)
s = sub.add_parser("add-font", help="rasterize a .ttf into a new font (needs Pillow)")
s.add_argument("hmi"); s.add_argument("ttf"); s.add_argument("height", type=int)
s.add_argument("--encoding", default="ascii", choices=["ascii", "iso-8859-1"])
s.add_argument("--encoding", default="ascii",
choices=["ascii", "iso-8859-1", "gb2312", "big5", "shift-jis", "ks_c_5601-1987", "utf-8"])
s.add_argument("--chars", help="characters to include (needed for multi-byte encodings)")
s.add_argument("--chars-file", help="UTF-8 text file with the characters to include")
s.add_argument("--fixed-width", type=int); s.add_argument("-o", "--out"); s.set_defaults(f=cmd_add_font)
s = sub.add_parser("add-picture", help="add a picture (png/jpg/bmp; needs Pillow)")
s.add_argument("hmi"); s.add_argument("image"); s.add_argument("-o", "--out")
+63 -8
View File
@@ -73,7 +73,27 @@ class Font:
glyphs: list = field(default_factory=list)
header: bytes = b"" # original 44-byte header, for fields not modelled
direction: int = 1 # display direction the glyphs are stored for (header byte 3 = 0x0A + d)
align: bool = False # Intelligent: glyph data starts 4-byte aligned (zero padding after the map)
align: bool = False # Intelligent: glyph data starts 8-byte aligned (zero padding after the map)
def codes(self, text: bytes) -> list:
"""Character codes of text bytes as this font's map stores them: UTF-8 fonts (encoding 24)
use code points; double-byte fonts (header byte 5 = 2: GB2312, Big5, Shift-JIS, KS C 5601)
take any byte >= 0x81 together with the next one, as lead | trail << 8 (measured in the
simulator: KS C hanja leads above the header's lead range still pair, and Shift-JIS
half-width katakana bytes pair up too, so they cannot be shown)."""
h = self.header
if h and h[4] == 24:
return [ord(ch) for ch in text.decode("utf-8", "replace")]
if h and h[5] == 2:
out, i = [], 0
while i < len(text):
b = text[i]
if b >= 0x81 and i + 1 < len(text):
out.append(b | text[i + 1] << 8); i += 2
else:
out.append(b); i += 1
return out
return list(text)
@property
def antialias(self) -> bool:
@@ -85,7 +105,7 @@ class Font:
# ------------------------------------------------------------------ decode
@classmethod
def parse(cls, blob: bytes) -> "Font":
if blob[:3] != b"\x04\xff\x00":
if blob[0] != 4: # bytes 1-2 are 0xFF 0x00 except for Big5 / Shift-JIS
raise ValueError("not a .zi font")
if blob[16] != 6:
raise ValueError("unsupported .zi version %d (only 6 is implemented)" % blob[16])
@@ -116,7 +136,7 @@ class Font:
for g in self.glyphs]
mp = bytearray()
off = MAP_ENTRY * len(self.glyphs)
pad = -off % 4 if self.align else 0
pad = -off % 8 if self.align else 0
off += pad
for g, b in zip(self.glyphs, bodies):
mp += struct.pack("<HB", g.code, g.width) + g.map_extra[:2].ljust(2, b"\0")
@@ -124,7 +144,7 @@ class Font:
off += len(b)
payload = name + bytes(mp) + bytes(pad) + b"".join(bodies)
hdr = bytearray(self.header or bytes(HEADER_SIZE))
hdr[0:4] = bytes([4, 0xFF, 0, 0x0A + self.direction % 4])
hdr[0:4] = bytes([4, hdr[1], hdr[2], 0x0A + self.direction % 4]) if self.header else bytes([4, 0xFF, 0, 0x0A + self.direction % 4])
struct.pack_into("<H", hdr, 4, self.encoding)
hdr[6], hdr[7], hdr[10], hdr[11] = self.width, self.height, self.first, self.last
struct.pack_into("<I", hdr, 12, len(self.glyphs))
@@ -195,18 +215,50 @@ def encode_glyph(rows: list, w: int, h: int, direction: int = 1) -> bytes:
ENCODINGS = {"ascii": (1, 0x20, 0x7E), "iso-8859-1": (3, 0x20, 0xFF)}
# Multi-byte fonts (header values from the Editor's Font Creator, fixed per encoding):
# encoding id, Python codec, header bytes 1-2 (first trail range end, trail gap), 8-9 (lead range),
# 10-11 (trail range), 0x1C-0x1D (first lead range end, lead gap), 0x20
MULTIBYTE = {
"gb2312": (2, "gb2312", (0xFF, 0x00), (0xA1, 0xF7), (0xA1, 0xFE), (0xFF, 0x00), 4),
"ks_c_5601-1987": (15, "euc_kr", (0xFF, 0x00), (0xA1, 0xC8), (0xA1, 0xFE), (0xFF, 0x00), 4),
"big5": (16, "big5", (0x7E, 0x22), (0xA0, 0xF9), (0x40, 0xFE), (0xFF, 0x00), 5),
"shift-jis": (23, "shift_jis", (0x7E, 0x01), (0x81, 0xEF), (0x40, 0xFC), (0x9F, 0x40), 5),
"utf-8": (24, "utf-8", (0xFF, 0x00), (0xFF, 0xFF), (0x00, 0xFF), (0xFF, 0x00), 5),
}
def char_code(ch: str, encoding: str) -> int:
"""Map code of one character in a font of this encoding (see Font.codes)."""
if encoding == "utf-8":
return ord(ch)
if encoding in MULTIBYTE:
b = ch.encode(MULTIBYTE[encoding][1])
return b[0] | b[1] << 8 if len(b) == 2 else b[0]
return ch.encode("latin-1")[0]
def from_ttf(path: str | None, height: int, encoding: str = "ascii", fixed_width: int | None = None,
name: str | None = None, threshold: int = 128) -> "Font":
name: str | None = None, threshold: int = 128, chars: str | None = None) -> "Font":
"""Rasterize a TrueType/OpenType font into a black/white Nextion font (needs Pillow).
``path=None`` uses Pillow's bundled default font (Aileron Regular, CC0) -- handy for tests and
demos. The font size is chosen so ascent + descent fills ``height`` pixels. Glyph widths follow
the font's advance widths unless ``fixed_width`` is given.
Multi-byte encodings (gb2312, big5, shift-jis, ks_c_5601-1987, utf-8) need ``chars``: the
characters to include (like the Font Creator's "Specified character" range), and a TrueType
file that has them.
"""
import os
from PIL import Image, ImageDraw, ImageFont
enc, first, last = ENCODINGS[encoding]
if encoding in MULTIBYTE:
if not chars:
raise ValueError("encoding %s needs chars= (the characters to include)" % encoding)
mb = MULTIBYTE[encoding]
enc, first, last = mb[0] | 2 << 8, mb[4][0], mb[4][1]
todo = sorted({(char_code(ch, encoding), ch) for ch in chars if ch not in "\r\n"})
else:
enc, first, last = ENCODINGS[encoding]
todo = [(code, bytes([code]).decode("latin-1")) for code in range(first, last + 1)]
def load(size):
return ImageFont.load_default(size=size) if path is None else ImageFont.truetype(path, size)
@@ -220,8 +272,7 @@ def from_ttf(path: str | None, height: int, encoding: str = "ascii", fixed_width
ascent, descent = f.getmetrics()
top = (height - ascent - descent) // 2
glyphs = []
for code in range(first, last + 1):
ch = bytes([code]).decode("latin-1")
for code, ch in todo:
adv = fixed_width or max(1, int(round(f.getlength(ch))))
img = Image.new("L", (adv, height), 0)
ImageDraw.Draw(img).text((0, top), ch, font=f, fill=255)
@@ -235,6 +286,10 @@ def from_ttf(path: str | None, height: int, encoding: str = "ascii", fixed_width
hdr[0:4] = b"\x04\xff\x00\x0b"
hdr[0x1C] = last
hdr[0x20] = 0x0B
if encoding in MULTIBYTE:
hdr[1:3], hdr[8:10], hdr[0x1C:0x1E] = bytes(mb[2]), bytes(mb[3]), bytes(mb[5])
hdr[0x20] = mb[6]
width = 0 # the Editor writes 0 for multi-byte fonts
hdr[18] = len(base[:255].encode("latin-1")) # name length twice, like the Editor re-saves it
struct.pack_into("<I", hdr, 0x24, len(glyphs))
return Font(base[:255], width, height, enc, first, last, glyphs, bytes(hdr))
+7 -4
View File
@@ -158,10 +158,10 @@ def t_add_picture(path, image_path, quality=100):
return {"picture_id": pid}
def t_add_font(path, height, ttf_path=None, encoding="ascii"):
def t_add_font(path, height, ttf_path=None, encoding="ascii", chars=None):
from . import fonts
h = _load(path)
f = fonts.from_ttf(ttf_path, int(height), encoding)
f = fonts.from_ttf(ttf_path, int(height), encoding, chars=chars)
fid = project.add_font(h, f)
h.save(path)
return {"font_id": fid, "name": f.name, "height": f.height}
@@ -243,8 +243,11 @@ TOOLS = {
"add_picture": (t_add_picture, "Add a picture from an image file (png/jpg/bmp); returns its picture id. "
"quality 1-99 stores a JPEG (Intelligent displays only, smaller), default 100.",
{"path": S, "image_path": S, "quality": {"type": "integer"}}, ["path", "image_path"]),
"add_font": (t_add_font, "Add a font rasterized from a TrueType file (omit ttf_path for the bundled CC0 font).",
{"path": S, "height": {"type": "integer"}, "ttf_path": S, "encoding": S}, ["path", "height"]),
"add_font": (t_add_font, "Add a font rasterized from a TrueType file (omit ttf_path for the bundled CC0 font). "
"encoding: ascii, iso-8859-1, or multi-byte gb2312/big5/shift-jis/ks_c_5601-1987/utf-8 "
"(then give chars, and a ttf_path that has those characters; the project's encoding "
"should match).",
{"path": S, "height": {"type": "integer"}, "ttf_path": S, "encoding": S, "chars": S}, ["path", "height"]),
"verify": (t_verify, "Check every checksum in the project.", {"path": S}, ["path"]),
"list_kinds": (t_kinds, "Component kinds that can be added.", {}, []),
"schema": (t_schema, "Attributes of a component kind: names, types, defaults, allowed values.", {"kind": S}, ["kind"]),
+1 -1
View File
@@ -260,7 +260,7 @@ def add_font(h: HMIFile, font) -> int:
n += 1
main = h.read("main.HMI")
font.direction = main[12] % 4 # glyphs are stored rotated like pictures
font.align = main[11] == 3 # Intelligent: glyph data 4-byte aligned
font.align = main[11] == 3 # Intelligent: glyph data 8-byte aligned
h.write("%d.zi" % n, font.pack())
h.write("main.HMI", main_insert_resource(h.read("main.HMI"), "zi", "%d.zi" % n))
return n
+3 -1
View File
@@ -256,13 +256,15 @@ class Renderer:
ycen = c.get("ycen") if ycen is None else ycen
wrap = c.get("isbr") if wrap is None else wrap
glyphs = {g.code: g for g in font.glyphs}
missing = glyphs[0x20].width if 0x20 in glyphs else font.width # unknown chars advance a space
def gw(ch):
g = glyphs.get(ord(ch))
return (g.width if g else font.width) + spax
return (g.width if g else missing) + spax
lines = []
for para in s.replace("\r\n", "\n").split("\n"):
para = "".join(map(chr, font.codes(para.encode("latin-1", "replace")))) # multi-byte text
cur = ""
for ch in para:
if wrap and cur and sum(gw(k) for k in cur + ch) - spax > w:
+20 -2
View File
@@ -126,6 +126,24 @@ class Format4(unittest.TestCase):
self.assertEqual(blob[24 + aoff:], bytes([63, 17, 0, 15]) * 4 + bytes([63, 17, 0, 255, 0, 112]))
self.assertEqual(pictures.to_image(blob).getpixel((3, 3))[3], 126)
@unittest.skipUnless(HAVE_PIL, "needs Pillow")
def test_multibyte_fonts(self):
# header layout of the Editor's Font Creator output (GB2312 / Shift-JIS / UTF-8)
gb = fonts.from_ttf(None, 16, "gb2312", chars="AB中文")
b = gb.pack()
self.assertEqual((b[4], b[5], b[8], b[9], b[10], b[11]), (2, 2, 0xA1, 0xF7, 0xA1, 0xFE))
self.assertEqual([g.code for g in gb.glyphs], [0x41, 0x42, 0xC4CE, 0xD0D6]) # bytes, little-endian
back = fonts.Font.parse(b)
self.assertEqual(back.codes("A中".encode("gb2312")), [0x41, 0xD0D6])
self.assertEqual(back.codes(bytes([0xE9, 0xFE])), [0xFEE9]) # any byte >= 0x81 starts a pair
sj = fonts.from_ttf(None, 16, "shift-jis", chars="A日").pack()
self.assertEqual((sj[1], sj[2], sj[0x1C], sj[0x1D]), (0x7E, 0x01, 0x9F, 0x40))
u8 = fonts.Font.parse(fonts.from_ttf(None, 16, "utf-8", chars="Aé中").pack())
self.assertEqual([g.code for g in u8.glyphs], [0x41, 0xE9, 0x4E2D]) # code points
self.assertEqual(u8.codes("é中".encode("utf-8")), [0xE9, 0x4E2D])
with self.assertRaises(ValueError):
fonts.from_ttf(None, 16, "big5") # needs chars=
def test_overhang_columns(self):
# Font Creator fonts: map bytes 3/4 = extra columns left/right of the advance width
rows = [[1, 0, 0, 1, 1]] * 2
@@ -139,7 +157,7 @@ class Format4(unittest.TestCase):
@unittest.skipUnless(HAVE_PIL, "needs Pillow")
def test_font_follows_project_direction(self):
# an Intelligent landscape project (0 degrees): glyphs row by row, header byte 3 = 0x0A,
# glyph data 4-byte aligned -- what the Editor converts a 90-degree font to
# glyph data 8-byte aligned -- what the Editor converts a 90-degree font to
from nxhmi import project
h = project.new_project("NX8048P050_011")
f = fonts.from_ttf(None, 16)
@@ -151,7 +169,7 @@ class Format4(unittest.TestCase):
self.assertEqual(back.glyph("0").pixels, fonts.from_ttf(None, 16).glyph("0").pixels)
self.assertEqual(back.pack(), blob)
m = 0x2C + blob[17]
self.assertEqual((blob[m + 5] | blob[m + 6] << 8) % 4, 0)
self.assertEqual((blob[m + 5] | blob[m + 6] << 8) % 8, 0)
@unittest.skipUnless(HAVE_PIL, "needs Pillow")
def test_image_roundtrip(self):