blackhole://nilFM

adventures in uxn and crystal

I've been taking a break from felt the past few weeks as far as my computing adventures go. I've been doing two things.

  1. rebuilding my desktop environment — more on that in an upcoming post
  2. creating a more intuitive email experience — this will be the subject of my current ramblings

I've had a lot of fun and learned a fair amount in this adventure, and I wanted to share, so here goes.

background

About this time last year, I made a post about my email setup. I used this setup pretty much unchanged for an entire year, and enjoyed it a lot. But I wanted something a little more polished - where I didn't have to type commands and craft pipelines just to view and rifle through my mail. So taro was born.

Taro mail client - the left window is the list view, showing mailboxes and a search query for the uxn mailing list; the right window is a terminal window displaying d6

I wanted something multi-window, lean, and built on top of mblaze. Inasmuch as it provided a dedicated interface to my email that "just worked", I did like thunderbird, so my initial experiences with that also guided me in crafting the UX.

why you do like this

uxn + crystal may seem like a weird stack to build a desktop application on at first glance, but it really was quite the natural choice.

For the frontend, the uxn VM varvara was the obvious choice. It was built for the express purpose of crafting low-friction, lightweight graphical applications. Having built xrxs and made a few other forays into the uxn ecosystem, I felt confident I could leverage the system effectively for my purposes, and the 8-bit aesthetic is to die for.

Still, the heavy lifting was best left to something else. I had recently run across the crystal language (I had seen it in passing a couple years ago, but it didn't grab me then), and reading its history and seeing some of its use cases, I decided I wanted to try it out. I had tried using ruby, which has a syntax largely identical to crystal's, in the past but it never made an impression on me; still, the compiled and type-safe nature of crystal and especially its concurrency model — something ruby entirely lacks — being similar to that of go (which has become one of my favorite languages recently) had me very interested.

So I had the basis of a three-tier architecture:

(low-level) backend (high-level) | frontend
[mblaze] <======> [crystal] <=======> [uxn]

the dirty details

uxn frontend

Up till now, the uxntal I'd crafted was pretty rudimentary. xrxs had a nice piece of code that I reused and refined though - the listbox. The thing about taro though, is that it has not one but two listboxes. So I exercised my stack machine brain and generalized the listbox drawing and interaction logic so that I could use the same code for both of them. It basically amounted to storing each listbox datastructure in the zero page, and adding macros for readability so that I could provide the listbox's address as an argument to the "methods", STH (stash) the listbox at the start of the method, and then when using the listbox's "properties" STHkr (stash-keep-return) the listbox back onto the stack and apply the macro to offset from the base address to the property of interest. This allowed me to bring a bit of object-oriented style to the uxntal code.

Another big change from xrxs was the scale of the data. In xrxs the listbox can only contain up to 255 items, and I only tested it with a couple dozen. taro handles a lot more data in the UI. Not only does it use a short (16 bits) to store the list length, selection index, etc., but each list item contains a lot more text than the ones in xrxs typically do. Since the listbox rendering routine has to process the entire list and determine what to draw and what not to on every draw, it is a significatly expensive computation. It can easily use a whole CPU core on an older machine like ksatrya with a mailbox with a couple hundred emails in it to process. So I implemented lazy drawing to lighten the load. Frames are only redrawn when the underlying data changes, which drops the CPU utilization to just 1 or 2%.

And how does this frontend communicate with the backend? To any experienced Uxner (or Unix sysadmin), the answer should be rather obvious. While the xrxs client used file I/O to talk 9p, taro uses stdio for IPC. Each message is unceremoniously a stream of bytes. The first byte denotes the message type. The next two bytes are the size of the message payload in big-endian unsigned-16-bit format, and any remaining bytes are said payload, if it exists. Even message types are consumed by the frontend, and odd message types are dispatched by it.

crystal backend

In the backend, we leverage crystal's concurrency model as well as the Process and IO modules to reduce the cognitive load in handling messaging. We get a nice division of labor amongst the fibers:

      [ChildWindow]                   [Ctl]
+--(Process.run uxnemu)<-+       (@socket read)--------+
|                        |                             |
+->(@@msg.send read_msg)-+=>(ChildWindow.msg.recieve)<-+

The main fiber runs the TaroCtl. This contains a ChildWindow which it instantiates in the initializer/constructor, and this spawns the uxnemu in a background fiber, while attaching FileDescriptors to the standard input and output of it. This fiber waits for the uxnemu to exit and sends a byte on a channel when it does so that the whole program can terminate.

A second background fiber is spawned by the ChildWindow at the same time as the uxnemu Process, which waits for output on the other end of its stdout. It decodes messages as detailed above and when it completes one sends it on the @@msg channel to be handled by the main thread.

A third fiber is spawned by the TaroCtl after initialization, which listens for bytes on a UNIXSocket. When it receives one, it sends a message on the ChildWindow.msg channel to signal an update. This is used to push UI updates from outside the program, eg, when new mail arrives in your inbox.

Back to the main fiber, it listens for messages on the ChildWindow.msg channel and performs the appropriate action, sending the output back to the uxnemu Process via the write_msg method which encodes the message and writes it to the uxnemu's stdin. It also listens for a byte to come in on the ChildWindow.lifetime channel mentioned before, so the backend can exit when the user closes the uxnemu window.

the loose ends

The uxnemu window is only used for managing the mailboxes — reading and writing mail is relegated to terminals running shell scripts. This is a case of "the right tool for the job", for a couple reasons: First, uxn doesn't support unicode (I had it render non-ASCII codepoints as question marks). Second, terminals along with tools like less, mcom, and so on do a great job of providing an interface for reading and composing emails, and leveraging this existing infrastructure gave me a working setup much faster than if I implemented these from scratch in varvara

These terminal windows are spawned in background fibers as well which wait for their execution to finish before sending update-UI messages on the ChildWindow.msg channel so that the UI can, eg, show them marked as read/replied, etc. The reader script is adapted and streamlined from the one I've been using for the past year.

musings

As I mentioned, I learend a lot on this project. And I ended up with a fully customized email client that has everything I need and nothing I don't.

In this adventure, I really sharpened my teeth on uxntal and got my feet wet with crystal. This being my first experience with the latter language, I had some positive and negative things to note:

The good things about crystal:

The things I didn't like so much:

Ultimately, though, I really like crystal and will likely continue to use it, and the uxn + crystal stack as a whole was really pleasant to work with.