Reverse Engineering Terminal Color Schemes

This post steps through the OS X Terminal theme file format (.terminal), showing how to extract encoded values, then introduces a program to convert themes to json, plist or set values from the command line.

While TextMate themes, .Xresources colors and the likes can be generated with banal scripts, Terminal color schemes are a reoccurring nuisance. Color after color must be individually sampled and set, but exporting a terminal theme to manually edit (by clicking on the little systems gear icon under the theme list) leads to this obfuscated mess:

$ cat theme.terminal

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
  <key>ANSIBlackColor</key>
  <data>

    YnBsaXN0MDDUAQIDBAUGFRZYJHZlcnNpb25YJG9iamVjdHNZJGFyY2hpdmVy
    VCR0b3ASAAGGoKMHCA9VJG51bGzTCQoLDA0OVU5TUkdCXE5TQ29sb3JTcGFj
    ZVYkY2xhc3NPECcwLjA2ODU2Nzc3NzQgMC4xMDA4MzQ5NzA4IDAuMTE4OTg1
    MjYyOAAQAoAC0hAREhNaJGNsYXNzbmFtZVgkY2xhc3Nlc1dOU0NvbG9yohIU
    WE5TT2JqZWN0XxAPTlNLZXllZEFyY2hpdmVy0RcYVHJvb3SAAQgRGiMtMjc7
    QUhOW2KMjpCVoKmxtL3P0tcAAAAAAAABAQAAAAAAAAAZAAAAAAAAAAAAAAAA
    AAAA2Q==

  </data>
  <key>ANSIBlueColor</key>
  <data>
    ...

And here the adventure begins. Given the header and structure of the file, this is a property list (plist), the OS X data object file. These are commonly used for frameworks and system configs, but restrained in data types. CFData, for values like ANSIBlueColor, are encoded in base64 when serialized. So decode the data.

$ base64 -D <<< "YnBsaXN0MDDUAQIDBAUGFRZYJHZlcnNpb25YJG9i..."

bplist00?X$versionX$objectsY$archiverT$top??U$null?     


UNSRGB\NSColorSpaceV$classO'0.0685677774 0.1008349708 0.1189852628??Z$classnameX$classesWNSColor?XNSObject_NSKeyedArchiver?Troot#-27;AHN[b?????????????]

Somehow, we're in a worse position than before. But much time has led me to enlightenment: this is another plist in itself. It's in the binary plist format though and we'll need plutil to convert it to something a bit more comprehend.

$ base64 -D <<< ... | plutil -convert xml1 - -o -

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
  <key>$archiver</key>
  <string>NSKeyedArchiver</string>
  <key>$objects</key>
  <array>

...

    <key>NSColorSpace</key>
    <integer>2</integer>
    <key>NSRGB</key>
    <data>
      MC4wNjg1Njc3Nzc0IDAuMTAwODM0OTcwOCAwLjExODk4NTI2MjgA
    </data>

...

Now things are looking up. ANSIBlueColor and other data items are stored as binary plists because of the complexity of each value. Colors include colorspaces and opacity; fonts include size and spacing. Most of these values are accessible now, but as a final tease, the RGB values themselves are stored as data items. Decode them once more and the RGB floats are free.

$ base64 -D <<< "MC4wNjg1Njc3Nzc0IDAuMTAwODM0OTcwOC..."

0.0685677774 0.1008349708 0.1189852628\0

Now if you want to deconstruct a terminal theme and extract the RGB values, there's one behemoth of a bash "one"-liner on StackOverflow. However, reconstructing a modified theme is another nightmare. That's where my work comes in.

OSX Terminal Themer

Convert between plist and json or set values from the command line

I've put together a simple python program to handle the manipulation of these terminal themes. It's on Github.

If extracting encoded values or manually setting them is your goal, OSX Terminal Themer will let you convert an exported theme to json. Edit it as needed and the program will convert it back to a valid theme to be imported. This can be scripted as needed.

For setting values on the fly, I've thrown in a few shortcuts. Colors and fonts can be set from the command line, making it even easier to modify a theme.

Have at it.

UPDATE 06/03/14: I've expanded my analysis as I spent more time working with Terminal themes to create OSX Terminal Themer.

SINSH

Sinsh Screenshot

Right as I said I wanted to explore Lisp and C, I got overloaded with school work. Any personal projects had to sit on the back burner for awhile. Pursuing a statistics minor didn't make the load much better, but as luck would have it, my classes ended up throwing Lisp and C right at me.

So now I've written a capable, interactive shell in C, when just a three weeks ago I was stepping through K&R chapter by chapter. It was rough, there was a lot of learning needed, but man is it cool to have built something complex in C.

Sinsh (like cinch) comes with some sweet ASCII art and aggressively parallelizes groups of commands that don't have dependency conflicts. It only works on a small subset of POSIX shells - mainly redirects, pipes and subshells – but with relevant bottlenecked shell scripts, it runs faster than sh. I wish I could post the code because I'm damn proud of the amount of work I put into the parser, but it's a school project that gets reused.

--

So anyway, first lesson from C:

The real coding happens on paper.

I'm sure I've said it before, but I've wasted enough hours confused by my own code to know that architecture matters more than anything in programming. I started to realize it while working in Python, but it's a whole other monster in C. In Python, you can hack your way around bad architecture. Sure it's a pain but you just plow through everything with your super flexible dynamic scripting steamroller or whatever. But in C, you can code yourself right into a dead end because you didn't think your data structures through correctly. And that's exactly what happened to me.

It was a school project, I wanted to get it done so I worked fast and paid the price. I ended up having to throw away 1000 lines of (abysmal) code. The second time around though, I did it right. I took the time to read what a parser looks like in C – tokenizer, lexer, parser. I mapped out the data structures, as well as all of the function specs. All of this can happen on paper, separating the coding and the thinking, rather than conflating the two.

Leaving Python to Learn Python

A few months ago, I decided to declare myself as a Python developer. When I was younger, I would hop from language to language, never staying all that long with one. I was much more interested in building things than learning the intricacies of software engineering. But I decided to slow down and pick a language to stick with until I've learned everything from the low level internal representation to controlling the highly abstracted packages.

But reading Armin Ronacher's post makes me step back. Here is something I've never seen: Python from a C perspective. And it's really, incredibly insightful. So I realized I don't know Python like Armin because I don't know C like him.

And the same can be said about concurrency, functional programming, and so on. I've been keeping a list, actually, and I think it's time to step away from Python for a bit. Not because I want to leave Python, but because I want to understand Python in ways that writing Python code doesn't expose me to.

At the very least, I would like to write C until I consciously think about memory storage, internal implementations and the tradeoffs. Although daunting, C will also be a great place to explore concurrency.

Alongside of C, I need work on functional programming to not only become comfortable with the paradigm but also the motivations. While C is better for understanding Python internals, functional programming is about being aware of side effects, modular design and problem decomposition, all of which would lead to more reliable code.

For learning, I've been looking for any non-trivial exercise I can. University courses seem to be a great source, and I'll start solving assignments from other universities for practice. I'll also be hopping about with functional programming languages until I find one I enjoy.

Brevity

Sometimes I get frustrated going through the amount of fluff on the internet. I wish people would just say their point and be done with it.

But then I look to my own writing and remember the most challenging part of communicating is being clear and effective, while still being concise and readable.

Hence my blog, where I too struggle to become more articulate.

Abusing __getattr__ for the sake of API design

The other day, I found myself up against a little API disaster. I've been working on a color manipulation Python library (Chroma), and a lot of the module's power comes from an intuitive API. Which is fair, as the library is designed to alleviate the pains of color coordinate systems.

To simplify color handling, Chroma includes a group of properties that allow a user to directly modify color coordinates, and, the API is pretty straightforward: to modify blue the property is obviously Color.blue, hue is Color.hue and the user can jump right in without reading through a documentation full of caveats and special cases.

But when it comes to saturation, this intuitive API model breaks:

Under HSV model, adding white to a pure color reduces its saturation, while adding black to a pure color reduces it's value. Under the [HLS] model, adding white or black to a pure color simply moves you up and down the brightness axis, and only by adding combinations of white AND black can you alter the color's saturation.

I've seen other libraries default the saturation property to HLS, but in some senses, this is only an intuitive API design when working in HLS. For those in HSV, this is deliberately misleading and creates painful, painful, hidden bugs since changing HLS saturation will shift both HSV saturation and value in unexpected ways.

In trying to solve this, I've thought of a potential solution: abuse a class's __getattr__ to inform users of the correct API calls.

When a user accesses any class attribute, Python invokes the class __getattribute__ method. If this attribute does not exist, __getattr__ is called and returns an AttributeError. By overriding __getattr__, it is possible to issue a warning in the special case of saturation, pointing the user to Color.hlssaturation and Color.hsvsaturation, before raising the default AttributeError. Without changing the library's functionality, the user gets a slight benefit of not having to leave coding or search documentation for the correct attributes. Likewise, in other API designs __getattribute__ can be use to issue a warning, detailing that Color.saturation is defaulting to HLS, and HSV users should be wary.

Productivity

I'm a sucker for productivity apps. And it's not just that I keep finding ones that are incredibly beautiful and make me all giddy when I use them because the UI is so pleasing. I actually, honestly find myself thinking: Wow, this app is going to change how I do things.

It's every other week or so I find something new to try. A couple days ago, I found this task manager called Plain Tasks; it's sort of like a simple Org Mode, but in Sublime Text. Yesterday, I found tree.app, which lets you organize all your subtasks as a horizontal tree. And today, it was TaskPaper. I don't even know much about it but it sounds awesome. It's not like I know where I find all these apps, they just keep popping up.

But, behind their UI interpretations, these apps are all the same: just a way of organizing a list. The main "feature" is the external gratification they offer for getting everyday work done. Any app you pick is equally decent, equally functional, and equally attractive. There are slight differences in implementation and interface, but the nuances in design are there to create a distinct presentation for a distinct personality; some people organize their mind one way and others another.

Now, of course, the trick is to find the tools that fit your personality. Unfortunately, when it comes to task managers, it seems any incompatibility between app and personality leads to an incredibly jarring user experience. So, unless you have a truly horizontal task manager--i.e. pen and paper--none of these apps will solve the grand, elusive productivity problem for everyone.

Somewhat ironically, trying to find which apps best suit your personality is the perfect way to waste many hours being entirely unproductive. And so, after having experimented with so much, I realize the best thing is to design the task manager you need, not search endlessly.

Nowadays, I've just been using plain text file, split up by days of the week, with an ordered list for each day. Around that, I added a Python script for sanity checks, GeekTools for a relaxed display on my desktop, and Dropbox, for mobile and cloud support. It works for me. I can't say it'll work for you or anyone else, but it seems to work for me.

Open URLs in Chrome Tabs

Looking through John Gruber's (Daring Fireball) projects, I found the great feature services, and Gruber's own "Open URLs in Safari Tabs." This service (found in the control-click contextual menu) will parse your selected text and open any URLs as individual Safari tabs.

However, after many years of use, I too have switched from Safari to Chrome, making the service no longer helpful to me.

Luckily, Gruber made the URL regex public, and using that, I patched together another service using Python. Using Python's web browser module, I can access the system's default browser, making the service browser agnostic. So I ported over the Safari service and made an Open URLs in Chrome Tabs (although, technically, for any browser).

Download it here.

Source here.

Desaturate

Desaturate is a lightweight (sitting at only 195kb) menu bar application for Mac OS X. It lets you toggle grayscale on and off as you so desire without ever having to open the Universal Access pane in System Preferences. And it's free for download too!

--

I keep my Mac display in grayscale. I actually like it. It's refreshing. Without color, I'm not bothered by things like the blaring red notifications every needy applications employs nowadays. But, syntax highlighting and Netflix still bring me back to Universal Access, where I can deactivate grayscale, more often than I'd like.

Really often, actually. I'm probably Universal Access's only returning visitor. Not to say I enjoy these visits, waiting for System Preferences to load gets pretty tedious when you're opening it forty times a day.

Fortunately, I've come up with a solution, albeit after searching for quite a long time. Most people who've asked about the same thing have accepted proposed AppleScript techniques, which use AppleScript to open System Preferences and select options using KeyEvents. Some others like me tried to use Terminal to change plists and restart WindowServer. I tried to take a new approach. After coming across display tinting software like F.lux that operates by placing a transparent window above all your applications and applying a filter underneath, I came up with Desaturate.

As it turns out, the solution to my grayscale problems is pretty simply: CGDisplayForceToGray(). More sophisticated software can be made to tint your display or even dynamically blur it using similar private API's.

I hope I've helped those few others in the Mac help forums with the same questions as me.

Download it here.