Better Text Rendering

Tl;dr: Starlight 1.0 was a TUI app, running entirely in the system terminal and leveraging it for text rendering. This had many benefits in terms of out of the box support for copy/pasting, text highlighting, customization, and screen reader support. However, differences in platform font support required me to switch to handling text rendering in the app itself.

Some people testing the game out highlighted an issue where the braille-rendered header text in files was displayed using question marks or other missing symbols. This turned out to be the result of Windows 10 using the older conhost/cmd rather than the new Windows 11 terminal app, which has a different set of fonts available. I quickly patched the game to request the Segoe UI font on Windows 10 since it has support for Braille fonts, but noticed that the overall rendering of text significantly degraded with overlapping letters since it is not intended for use with the command line.

This led to an adventure in re-writing the backend to handle text rendering. The initial migration went fairly quickly thanks to two factors:

• The TUI library I use - ratatui - has support for custom backends, which allowed me to preserve almost all of the game’s current rendering and just write a new text rendering backend.
• I was already familiar with macroquad, which is quick to setup and easy to use for rendering.

I soon had a working text rendering backend leveraging macroquad, but ran into the issue of finding an appropriate font that met all of the following requirements:

1. Compatibly licensed (i.e. allows commercial use, does not require source code to be made available).
2. Monospaced.
3. Supports both Braille (used for headers) and utf8 full/half blocks (used for image rendering).
4. Renders crisply (a surprising number of fonts that looked good otherwise ended up blurry when rendered in the actual game).
5. (Nice-to-have) Free.
6. (Nice-to-have) 1:2 aspect ratio.

I think I tried over a hundred different fonts before I found one that matched all of my requirements - Fairfax.

With all of that out of the way, I was able to release version 1.1 of the game with the new font rendering.

There are a couple of downsides to this approach:

1. No support yet for copy/pasting text.
2. No screen reader support.

But, since the alternative was really poor Windows 10 support, I decided I could live with them.

Now that I had working rendering, I went to work fixing bugs and improving performance. While doing so, I ran into some frustrations with macroquad’s support for fullscreen apps. The are a few ways to open file dialogs in the game, and with macroquad in fullscreen mode, the dialogs opened up underneath the application. This broke the ability to e.g. export and import save files.

Another frustration was that macroquad required rendering to a separate texture before rendering to the screen as it clears the screen between each frame. Since the game is based on a tui, only changed characters are rendered during a frame, resulting in a mostly empty screen each frame. Fixing this required either a full relayout + repaint of the app each frame or rendering to a separate texture that would preserve the image data between frames and keeping it manually in sync with the screen size.

Due to these factors, I switched backends to sdl2 for input and rendering with fontdue for text rasterization. This worked really well, with file dialogs appearing above the game and remaining interactive.

I then focused on reducing the game’s memory usage. It was using roughly 600MB of ram, which was far too much for how simple the internals of the engine are (text processing and rendering). Profiling heap usage using dhat revealed most of the memory consumption was related to loaded sound files. Reducing this usage was as simple as switching kira to use streaming rather than static sounds, bringing the total memory usage down to roughly 200MB.

The remaining usage was in allocations for font data, and a quick switch ab_glyph for rasterization dropped memory usage further down to roughly 120MB.

Given that the binary itself is 70MB, with 42MB of sound files, I’m pretty happy with the end result.

Throughout this, I had a subgoal of supporting more fonts (ideally Cascadia Code since I like it better than Fairfax) without dealing with blurry or artifacted text rendering in addition to using ttf fonts for header rendering rather than ascii bitmap fonts. Fixing this required switching away from ab_glyph to my own rasterization using ttf-parser and tiny-skia. Getting the scaling and position of rendered glyphs correct took quite a bit more work than using ab_glyph, but the end result was pixel perfect rendering of Cascadia Code using pixel scaling rather than pt scaling, allowing me to better control the text grid in the game. Not to mention shrinking the binary by ~10MB. There’s still improvements to be made using hinting, but it’s good enough for now.

After perfecting rendering, I switched to working on screen reader support. I was pretty sad to lose it with the switch to custom text rendering and wanted it back. Integrating it ended up taking quite a bit of effort, hampered by a few factors:

• I don’t use screenreaders day-to-day, so I’m unfamiliar with their usage.
• My only experience with accessibility support is from web and the knowledge didn’t transfer over to a Rust application.
• The major Rust library for a11y (accesskit) has fairly sparse examples.
• There are some quirks with how screen readers process text.

The first challenge I ran into was getting the screen reader to read anything at all. I eventually figured out this was due to two things. The first was that Windows Narrator will often only read the text contents of the name field for some objects (this took a long time to discover). The second was that Windows itself would only detect the top level element of the tree if it was ready at the time the application gained focus, and if it didn’t find a top level element, it wouldn’t rebuild the tree later when it was invalidated. Fixing the first issue was simple. Fixing the second issue required spawning the application hidden and only showing it once loading completes.

The second challenge I ran into was the screen reader pausing after reading only a portion of the screen. This turned out to be due to the special characters and extra whitespace embedded in the screen as part of the graphical representation of menus. Fixing this required stripping all non-alphabetic/punctuation/whitespace characters from the text set on the object and collapsing all whitespace down to a single space.

After all of that, version 1.1.12 of Starlight supports screen readers again! It probably will have issues with non-English languages, but it’s better than nothing.