chris blogs: October 2007

31oct2007 · On Mail

I have just aborted a week-long oddisey of finding a good mail client that can deal with pretty big IMAP mail boxes. (Of course, this is because Gmail now does IMAP, which is a really nice idea.)

They all suck, are too slow, or both.

Before everyone now comes shouting, “Bah, just use Mail.app”, these are my requirements for this use case (I’ll try Mail.app again as soon as I get Leopard, and will tell you how it breaks down then…):

  • Must work over SSH without X, which means console/line based
  • Must support IMAP well (bye mutt)
  • Must support big mailboxes (100k messages and more)
  • Must be open-source
  • Should be unixish in some way

One more word about the big mailboxes: I do not need to see all headers at all times, and if the mailer can operate fast on a subset (say, the last 2000 mails), this is fine enough. I’d rather use a quick program with a smaller working set than a slow program which shows all messages of this century. In fact, this is recommended by the IMAP client coding HOWTO. (I think only Pine does it, and that didn’t work too well either.)

I tried all of these: Mutt (the only program with nicer source than interface), Pine (works well with smallish IMAP boxes, but breaks down with my 172k-ruby-talk box), Nail (same, and the UI sucks), and Cone (I gave up compiling).

As you will notice, most of these clients are written in C. It’s probably because of their age, but really it is a waste of time. Mailers written in C are the worst you can imagine, except for mailers written in C++, which share the issues but take ten times as long to compile.

Mail clients written in C usually means that they are noncustomizable/untweakable (Pine) or they use hacky configuration language (Mutt). Scripting languages exist, and given proper algorithms and datastructures (which you need anyway, if you want to make a scalable mail reader), they are fast enough to do anything mail related, while still being proper, portable languages for extensions and plugins. (There are a few mail programs written in Perl, but they don’t seem successful.)

I’m back to Gnus/fetchmail/Maildir now. It can support IMAP, and works relatively fast (see below for reason), but searching for new mail is slow, and blocks my complete Emacs (I may end up just starting two instances…).

And Gnus, I now realize, is a fine mail reader. This probably is because it was made as a news reader, and the virtues of a news reader are what counts in my case: Gnus asks you how many messages to load (usually just the unread ones), and works zippy with them then, but even with 10k of messages, it still *is* usable.

Furthermore, it has a seriously cool feature: expiring. You don’t delete mail (I never do that, anyway, which is why my mailboxes are getting so big), but you expire it, and if the mail is expired and older than a week or so (configurable), it either deletes it, or moves it into a different mail box, or does anything you tell it with elisp. I now use this to make monthly mbox-archives of ruby-core and ruby-talk, since HFS+ doesn’t really like 150k+ files in a directory.

Which gets me to a side note… one of my first posts on this blog was about mail storage formats, and Maildir essentially was the winner. Maildir is rock-solid, but tools like rsync or rsnapshot really have to work hard to back them up, if they reach a decent size. And many file systems (still no ZFS in OS X) slow down a lot. I’d like to propose a Multimaildir format that stores mails like Git in Maildir/000/999 and the next one in Maildir/001/000 instead of stuffing them all into one directory. Should be pretty easy to do, and makes everything faster. (You also could move your old mail easily, just move Maildir/000 somewhere else (O(1)), instead of globbing like hell (O(n)).) End of sidenote.

I also had a closer look at MH, which always fascinated me. There is a good O’Reilly book on it available online, and I like how it was designed. Very unixish. I can’t really imagine using it, though. (I read/skim lots of mailing lists, and apparently MH doesn’t thread, and I guess it’s just too slow to vgrep a summary and read the few interesting posts. Maybe with a really well-tuned Zsh setup, with keybindings and everything.)

For half a day I glimpsed the idea of writing an IMAP-based MH. In fact, such a thing, written in Python, exists as MHI. But I’m not sure I really want to use it, and it would suck to spend a lot of time to reinvent the wheel and not even drive with it.

So, I thought about my needs, did a bit of research with antique mail clients. (Did you know jwz used Netscape 3.02 for a looong time to read his mail? Not console based, unfortunately.)

Now, I’m sucking it down and will write my own client, and I’ll use something one rarely sees on Unix: lets call them “interactive non-screen based interfaces”. Actually, that’s wrong, because you probably use the shell every day. Mutt and Pine take all your screen and are nontrivial to implement (you need to do a pager, and all the curses stuff, yeech), while mail/mailx/nail are totally-line based and you need to end each command by pressing return (which is one key too many for lots of mails).

I think we should do it like this: make a small library that provides a few widgets, like “line picker”, “item picker”, “line reader”, add an Emacs-style (or, almost easier, vi-style) keyboard map system to dispatch between these and make all input interactive (cbreak). Many apps on ITS worked like that, and it’s pretty comfortable to use, while still fast, flexible and text-based.

I’m pondering making a Gnus feelalike in Ruby based on this scheme, with the following very limited function set: just IMAP (but that well), and all configuration by editing/adding Ruby, threading like jwz does it, and just the stuff I need. One should have usable results within a week, who knows.

I could end up in the history of men by writing the first mail reader that doesn’t suck. :-P

NP: Bob Dylan—No More Auction Block

Copyright © 2004–2016