chris blogs

02jan2017 · zz: a smart and efficient directory changer

A nice feature I’ve become used to in the last year is a so-called “smart directory changer” that keeps track of the directories you change into, and then lets you jump to popular ones quickly, using fragments of the path to find the right location.

There is quite some prior art in this, such as autojump, fasd or z, but I could not resist building my own implementation of it, optimized for zsh.

As far as I can see, my zz directory changer is the only one with a “pay-as-you-go” performance impact, i.e., not every directory change is slowed down, but only every use of the smart matching functonality.

The idea is pretty easy: we add a chpwd hook to zsh to keep track of directory changes, and log for each change a line looking like “0 $epochtime 1 $path” into a file ~/.zz. This is an operation with effectively constant cost on a Unix system.

chpwd_zz() {
  print -P '0\t%D{%s}\t1\t%~' >>~/.zz
chpwd_functions=( ${(kM)functions:#chpwd?*} )

The actual jumping function is called zz:

zz() {

How does the matching work? It’s an adaption of the z algorithm: The lines of ~/.zz are tallied by directory and last-used time stamp, so for example the lines

0 1483225200 1 ~/src
0 1483225201 1 ~/tmp
0 1483225202 1 ~/src
0 1483225203 1 ~/tmp
0 1483225204 1 ~/src

would turn into

6 1483225204 3 ~/src
4 1483225203 2 ~/tmp

Also, the initial number, the effective score of the directory, is computed: We take the relative age of the directory (that is, seconds since we went there), and boost or dampen the results: the frequency is multiplied by 4 for directories not older than 1 hour, doubled for directories we went into today, halved for directories we went into this week, and divided by 4 else.

  awk -v ${(%):-now=%D{%s}} <~/.zz '
    function r(t,f) {
      age = now - t
      return (age<3600) ? f*4 : (age<86400) ? f*2 : (age<604800) ? f/2 : f/4
    { f[$4]+=$3; if ($2>l[$4]) l[$4]=$2 }
    END { for(i in f) printf("%d\t%d\t%d\t%s\n",r(l[i],f[i]),l[i],f[i],i) }' |

By design, this tallied file can be appended again with new lines originating from chpwd, and recomputed whenever needed.

The output of this tally is then sorted by age, truncated to 9000 lines, then sorted by score. (My ~/.zz is only 350 lines, however.)

      sort -k2 -n -r | sed 9000q | sort -n -r -o ~/.zz

With this precomputed tally (which is generated in linear time), finding the best match is easy. It is the first string that matches all arguments:

  if (( $# )); then
    local p=$(awk 'NR != FNR { exit }  # exit after first file argument
                   { for (i = 3; i < ARGC; i++) if ($4 !~ ARGV[i]) next
                     print $4; exit }' ~/.zz ~/.zz "$@")

If nothing was found, we bail with exit code 1. If zz is used interactively, it changes into the best match, else the best match is just printed. This allows using things like cp foo.mkv $(zz mov).

    [[ $p ]] || return 1
    local op=print
    [[ -t 1 ]] && op=cd
    if [[ -d ${~p} ]]; then
      $op ${~p}

If we found a directory that doesn’t exist anymore, we clean up the ~/.zz file, and try it all over.

      # clean nonexisting paths and retry
      while read -r line; do
        [[ -d ${~${line#*$'\t'*$'\t'*$'\t'}} ]] && print -r $line
      done <~/.zz | sort -n -r -o ~/.zz
      zz "$@"

With no arguments, zz simply prints the top ten directories.

    sed 10q ~/.zz

I actually shortcut zz to z and add a leading space to not store z calls into history:

alias z=' zz'

The full code (possibly updated) can be found as usual in my .zshrc.

I use lots of shell hacks, but zz definitely is among my most successful ones.

NP: Leonard Cohen—Leaving The Table

24dec2016 · Merry Christmas!

Comic © Liz Climo

Frohe Weihnachten, ein schönes Fest, und einen guten Rutsch ins neue Jahr wünscht euch
Christian Neukirchen

Merry Christmas and a Happy New Year!

NP: Against Me!—Haunting, Haunted, Haunts

15jan2016 · Dear Github

These kind of posts seem popular these days.

My top six of features GitHub is missing:

  1. Searching for text in commit messages. Fixed 2017-01-04. About 2/3 of the repos I clone, I solely clone to run git log --grep.
  2. Searching in the wiki. Fixed 2016-08-08.
  3. Archive tarballs with submodule checkouts included; else submodule usage is totally pointless.
  4. Marking issues private to committers. Useful both for embargoed security issues and to keep out an angry mob.
  5. Being able to disable pull requests. For projects that use Github mainly as a mirror.
  6. IPv6 support. It’s 2016, damnit.


NP: Revolte Springen—Hinter den Barrikaden

24dec2015 · Merry Christmas!

Consumers' crèche

Frohe Weihnachten, ein schönes Fest, und einen guten Rutsch ins neue Jahr wünscht euch Christian Neukirchen

Merry Christmas and a Happy New Year!

NP: Elende Bande—Uns das Leben

19feb2015 · Six hacks for less(1)

Recently I got around to configuring less, and I collected these few tricks:

  1. Sometimes I look at lists with less, and then do things step-by-step, keeping the current action at the top of the page. This works nicely until you end up at the last page of the file, and then can’t scroll down. You lose track of where you are at and get confused.

    It would be much nicer scrolling down, and filling up the buffer with ~ after the end of file, just as if you had searched in the pager.

    Actually, with ESC-SPC, you can move a full page down, filling up the buffer with ~. Toying around a bit, you’ll find out that you can override the “page length” with a prefix, i.e. 1 ESC-SPC will move down one line only!

    However, this is still inconvenient to type all the time, thus let’s define a keybinding. For this, create a file ~/.lesskey where we will put the key definitions. This file then will be compiled using lesskey(1) and generate a binary configuration file ~/.less. (I guess you can be lucky that m4 is not involved in this mess…)

    One problem is actually binding the key. You can easily bind the cursor down key (\kd) to forw-screen-force, but how do you pass 1? The canonical hack is to use the noaction action, which will behave just like you’ve typed the keys after it. Thus, we write:

    \kd noaction 1\e\40
    j noaction 1\e\40

    (By the way, that #command comment is important to tell lesskey you are defining key commands.)

    Finally, scrolling bliss!

    Actually, scrap that.

    The badly underdocumented key J (and K) will scroll how I want, but you only read about that in the example inside lesskey(1). Therefore, we can just do:

    \kd forw-line-force
    j forw-line-force

    These keybindings are there since at least 1997 and I’ve never found them before…

  2. While we are redefining keys, I’ve always found it a bit clumsy to read multiple files, having to type :n and :p. Using [ and ] is much more convenient (at least on a US keyboard), and by default these keys do things of questionable utility.

    [ prev-file
    ] next-file
  3. Did you ever wish to give feedback from less? Like have a script output some info, and you decide how to go on? Since less always exits with status 0 usually, this I thought this was tricky to do, but the quit action actually can return an arbitrary exit code, encoded as a character.

    I bound Q and :cq (like in vim) to exit with status 1:

    Q quit \1
    :cq quit \1

    Now you can do stuff like look at all files and have them deleted when you press Q instead of q to exit:

    for f in *; do less $f || rm $f; done
  4. I use less a lot to look at patches, git log output, and ocassionally mailboxes. The D command as defined below will move to the next line starting with diff or commit or From␣.

    D noaction j/\^diff|commit|From \n\eu

    It will also “type” ESC-u to hide the highlighting. Now I can simply press D to jump to the next chunk of interest.

  5. To return to where you started from after a search or going to the end of file, type ''. Typing '' again will go back, so this is also nice to toggle between two search results.

  6. Back in the old days of X11R2(?) there was a tool called xless, which was exactly that: a pager like less that ran in its own X11 window. It’s quite useful. We can recreate this by combining a X11 terminal emulator and plain less with a small zsh snippet:

    xless() {
        exec {stdin}<&0 {stderr}>&2
        exec urxvt -e sh -c "less ${(j: :)${(qq)@}} </dev/fd/$stdin 2>/dev/fd/$stderr"
      } &!

    Watch the trick how we pass the stdin/stderr file descriptors and the file arguments!

    Now you can just run command-spitting-out-loads | xless and the output will be shown in a new terminal and not lock your shell.

    NP: Feine Sahne Fischfilet—Dreieinhalb Meter Lichtgestalt

17feb2015 · 10 fancy zsh tricks you may not know...

Wow, almost two years have passed since the latest installment of our favorite clickbait zsh tricks series.

  1. When editing long lines in the zle line editor, sometimes you want to move “by physical line”, that is, to the character in the terminal line below (like gj and gk in vim).

    We can fake that feature by finding out the terminal width and moving charwise:

    _physical_up_line()   { zle backward-char -n $COLUMNS }
    _physical_down_line() { zle forward-char  -n $COLUMNS }
    zle -N physical-up-line _physical_up_line
    zle -N physical-down-line _physical_down_line
    bindkey "\e\e[A" physical-up-line
    bindkey "\e\e[B" physical-down-line

    Now, ESC-up and ESC-down will move by physical line.

  2. Sometimes it’s nice to do things in random order. Many tools such as image viewers, music or media players have a “shuffle” mode, but when they don’t, you can help yourself with this small trick:


    Just append ($SHUF) to any glob, and get the matches shuffled:

    % touch a b c d
    % echo *($SHUF)
    d c a b
    % echo *($SHUF)
    c a d b

    Note that this shuffle is slightly biased, but it should not matter in practice. In doubt, use shuf or sort -R or something else…

  3. Are you getting sick of typing cd ../../.. all the time? Why not type up 3?

    up() {
      local op=print
      [[ -t 1 ]] && op=cd
      case "$1" in
        '') up 1;;
        -*|+*) $op ~$1;;
        <->) $op $(printf '../%.0s' {1..$1});;
        *) local -a seg; seg=(${(s:/:)PWD%/*})
           local n=${(j:/:)seg[1,(I)$1*]}
           if [[ -n $n ]]; then
             $op /$n
             print -u2 up: could not find prefix $1 in $PWD
             return 1

    With this helper function, you can do a lot more actually: Say you are in ~/src/zsh/Src/Builtins and want to go to ~/src/zsh. Just say up zsh. Or even just up z.

    And as a bonus, if you capture the output of up, it will print the directory you want, and not change to it. So you can do:

    mv foo.c $(up zsh)
  4. Previous tricks (#6/#7) introduced the dirstack and how to navigate it. But why type cd -<TAB> and figure out the directory you want to go to when you simply can type cd ~[zsh] and go to the first directory in the dirstack matching zsh? For this, we define the zsh dynamic directory function:

    _mydirstack() {
      local -a lines list
      for d in $dirstack; do
        lines+="$(($#lines+1)) -- $d"
      _wanted -V directory-stack expl 'directory stack' \
        compadd "$@" -ld lines -S']/' -Q -a list
    zsh_directory_name() {
      case $1 in
        c) _mydirstack;;
        n) case $2 in
             <0-9>) reply=($dirstack[$2]);;
             *) reply=($dirstack[(r)*$2*]);;
        d) false;;

    The first function is just the completion, so cd ~[<TAB> will work as well.

  5. Did you ever want to move a file with spaces in the name, and mixed up argument order?

    % mv last-will.tex My\ Last\ Will.rtf

    Pressing ESC-t (transpose-words) between the file names will do the wrong thing by default:

    % mv My last-will.tex\ Last\ Will.rtf

    Luckily, we can teach transpose-words to understand shell syntax:

    autoload -Uz transpose-words-match
    zstyle ':zle:transpose-words' word-style shell
    zle -N transpose-words transpose-words-match


    % mv My\ Last\ Will.rtf last-will.tex
  6. If you are an avid Emacs user like me, you’ll find this function useful. It enters the directory the currently active Emacs file resides in:

    cde() {
      cd ${(Q)~$(emacsclient -e '(with-current-buffer
                                   (window-buffer (selected-window))
                                   default-directory) ')}

    You need the emacs-server functionality enabled for this to work.

  7. I’m working on many different systems and try to keep a portable .zshrc between those. One problem used to be setting $PATH portably, because there is quite some difference among systems. I now let zsh figure out what belongs to $PATH:

    export PATH
    path=( ${(u)^path:A}(N-/) )

    The last line will normalize all paths, and remove duplicates and nonexisting directories. Also, notice how I pick up the latest Ruby version to find the Gem bin dir by sorting them numerically.

  8. One of the hardest things is to set the xterm title “correctly”, because most people do it wrong in some way, and then it will break when you have literal tabs or percent signs or tildes in your command line. Here is what I currently use:

    case "$TERM" in
        precmd() {  print -Pn "\e]0;%m: %~\a" }
        preexec() { print -n "\e]0;$HOST: ${(q)1//(#m)[$'\000-\037\177-']/${(q)MATCH}}\a" }
  9. For a cheap, but secure password generator, you can use this:

    zpass() {
      LC_ALL=C tr -dc '0-9A-Za-z_@#%*,.:?!~' < /dev/urandom | head -c${1:-10}
  10. Sometimes it’s interesting to find a file residing in some directory “above” (e.g. Makefile, .git and similar). We can glob these by repeating ../ using the #-operator (You have EXTENDED_GLOB enabled, right?). This will result in all matches, so let’s first sort them by directory depth:

    % pwd
    % print -l (../)

    Now we can pick the first one, and also make the file name absolute:

    % print (../)[1]:A) 

    I knew the #-operator, but it never occurred to me to use it this way before.

    Until next time!

    NP: Pierced Arrows—On Our Way

24dec2014 · Merry Christmas!

Consumers' crèche

Frohe Weihnachten, ein schönes Fest, und einen guten Rutsch ins neue Jahr wünscht euch Christian Neukirchen

Merry Christmas and a Happy New Year!

Bitte lesen: Liebeserklärung an die Vielfalt - eine Weihnachtsbotschaft.

NP: Against Me!—Holy Shit

03dec2014 · Recovering a Git repository from filesystem corruption

Recently I had to fix a Git repository where something unfortunate happened: probably due to accessing a NTFS partition that was still mounted in a hibernated alternative operating system, several files became corrupted (and actually had their contents exchanged with different files on the disk!).

git fsck discovered corrupted blobs, which we tried to recover first, when we detected their content does not make sense at all. These blobs were irrevocably lost, but we still wanted to get out the rest of the Git history alive.

Usually, applying the following technique is not necessary, because you can either just clone again from your Git upstream or recover the repository from the last backup—but both did not exist in this case.

The corrupted blobs actually all belonged to a single commit that happened a few months ago. The solution was thus to remove this commit from the history, keeping all other trees intact (of course, commit ids would change, but the content won’t).

I first tried to do this with git rebase, but it is of course the wrong tool, since it will try to remove the change of the defect in all following history.

Finally, I had a use-case for git filter-branch. To make it short, we can filter out the defect commit using:

git filter-branch --commit-filter \
  '[ $GIT_COMMIT = badbadbadbad ] && skip_commit "$@" ||
                                     git commit-tree "$@"'

This will rewrite all commits after badbadbadbad, but not touch their actual content.

git fsck still was not happy, we thus made a clean copy using

git clone --no-local --no-hardlinks mybrokengit myfixedgit

Now git fsck reported no errors and all other revisions were still ok. (Also, the blobs have been packed, so the next data corruption will be more fatal… ;))

I cannot think of any other version control system where a repair like this would have been possible. Thanks, Git!

NP: Against Me!—Exhaustion & Disgust

16mar2014 · Review: Learning Shell Scripting with Zsh

Learning Shell Scripting with Zsh
by Gastón Festari.
Packt Publishing, Birmingham 2014.
132 pages.

[Full disclosure: I have received a digital copy of the book in exchange for this review.]

The book is titled Learning Shell Scripting with Zsh: Your one-stop guide to reading, writing, and debugging simple and complex Z shell scripts which could not be more wrong. A far more appropriate title would have been: Getting started with Zsh: Your guide to interactive use and first steps of customization.

A first glance through the table of contents proves me right: There are chapters on “Getting started”, “Alias and History”, “Advanced Editing”, “Globbing”, “Completion”, and “Tips and Tricks”, but no explicit mention of writing shell scripts for purposes other than customization of an interactive shell.

I’m not completely sure of the target audience: The book assumes basic familiarity with (Bourne) shell already, half-heartedly explains pipes or redirections as well as how to define shell functions, but e.g. there is no mention of how to pass or parse arguments. After a single example of a very simple completion function in Chapter 5 (with explanation), the reader is assumed know enough for writing his own completions. I highly doubt that.

I found the book riddled with small mistakes, imprecise to wrong explanations and plain terribly sloppy typesetting of the code examples, which hurts particularly in a book about shell where syntax is lax but proper newlines matter. Examples are often not very well chosen and occasionally confusing even to me, who is very familiar with the topic.

The over-emphatic style with its needless rambling and cheeky language does not save the book but probably annoys the reader even more. (I hope not to ever read “You know, because widescreen.” in a book again.) Without, a lot more actual content and perhaps a real introduction to shell usage would have fit into these compact 118 pages.

The book finishes with a recommendation to read From Bash To Z Shell next. All I can recommend is: better skip this book completely and start with that one if you are interested in a good book about zsh.

Rating: 2 of 5 points.

NP: Against Me!—White Crosses

02feb2014 · The road to OpenSSH bliss: ED25519 and the IdentityPersist patch

The recent release of OpenSSH 6.5 had many convincing new features to make me update to it early, quoting from the release notes:

  • support for key exchange using elliptic-curve Diffie–Hellman in Daniel Bernstein’s Curve25519
  • support for Ed25519 as a public key type
  • a new private key format that uses a bcrypt KDF to better protect keys at rest
  • a new transport cipher that combines Daniel Bernstein’s ChaCha20 stream cipher and Poly1305 MAC to build an authenticated encryption mode

Since OpenSSH 5.7, there is support for ECDSA keys according to RFC5656. The problem with this schema is that it uses NIST curves generally. Due to recent events, everyone is (rightfully) more paranoid now, and there are reasons to consider these curves to be problematic. Thus, I decided to disable support for ECDSA host keys and use the superior ED25519 scheme for new keys.

However, I also need to access many machines which don’t run the latest version of OpenSSH, but for these we can at least make use of the new, safer public key format.

First, I will show how to update your OpenSSH installation to make use of the new features, and then I’ll explain what else I had to do to make everything work correctly.

Upgrading OpenSSH

Arch is, at the time of writing, providing binaries in “testing”, but plucking a single package is easy:

# pacman -U

After merging configuration files (if required), we edit /etc/ssh/sshd_config and spell out the HostKeys to disable the built-in defaults, which include ECDSA.

HostKey /etc/ssh/ssh_host_rsa_key
HostKey /etc/ssh/ssh_host_dsa_key
HostKey /etc/ssh/ssh_host_ed25519_key

Upon restarting SSH, a new ED25519 hostkey will be generated. Using ignite:

# sv restart sshd

Essentially, that’s it. Let’s check that the ECDSA hostkey is disabled:

% ssh-keyscan -t ecdsa,ed25519 localhost
# localhost SSH-2.0-OpenSSH_6.5
no hostkey alg
# localhost SSH-2.0-OpenSSH_6.5
localhost ssh-ed25519 AAAA...

(AFAIU, ECDSA user keys in authorized_keys will still work. It’s your task to replace them with ED22519 ones, I have found no way to blacklist them.)

We continue by creating a new ED25519 key for the user:

% ssh-keygen -t ed25519
Generating public/private ed25519 key pair.
Enter file in which to save the key (/home/chris/.ssh/id_ed25519): 
Enter passphrase (empty for no passphrase): 
Enter same passphrase again:

For testing, we can add it to the locally accepted keys:

% cat .ssh/ >>.ssh/authorized_keys

Finally, we can test it:

% ssh chris@localhost
Enter passphrase for key '/home/chris/.ssh/id_ed25519':

It seems to work! (Perhaps you’ll see an update of the fingerprint if you had the ECDSA one saved in .ssh/known_hosts.)

I was happy about that until I realized that ssh asked for the passphrase, and not my gnome-keyring-daemon… which brings us to part two of this post.

Migrating to ssh-agent and the IdentityPersist patch

Since new ED25519 keys are always stored in the new bcrypt format, they won’t work (as of right now) with SSH agents that don’t support it (I know of gnome-keyring and mate-keyring; gpg-agent stores the key itself, anyway). Essentially, only plain ssh-agent supports it, but I used to hate ssh-agent since it doesn’t support adding keys upon use: gnome-keyring will ask for the password and keep the key unlocked. Since I try to keep my keys locked when I’m not using them, I don’t want to keep them unlocked in every session, and neither unlock them manually, because that is inconvenient.

Luckily, I found this patch which adds the key automatically. I hope it gets accepted, because it is very useful. I couldn’t wait and patched OpenSSH myself. With this tiny patch, I could finally drop my use of gnome-keyring, which is one more step towards a GTK3-free desktop.

I run ssh-agent from my .xinitrc:

eval $(ssh-agent -s)
xscreensaver-ssh-helper &

Now, we can convert our old keys to the new storage format:

% ssh-keygen -p -o -f ~/.ssh/id_dsa
Enter old passphrase: 
Key has comment 'id_dsa'
Enter new passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved with the new passphrase.

The key file should now start with -----BEGIN OPENSSH PRIVATE KEY-----. The public key format is unchanged.

For the IdentityPersist patch, we need to add a line to .ssh/config, stating the lifetime of the key (or true for infinity):

IdentityPersist 300

Now, we can try everything together:

% ssh myhost
Enter passphrase for key '/home/chris/.ssh/id_dsa': 
Identity added: /home/chris/.ssh/id_dsa (id_dsa)

Another SSH connect within the next 5 minutes will connect without asking for a password.

I noticed one more difference between gnome-keyring and ssh-agent: gnome-keyring would unlock any key that has a *.pub in ~/.ssh, while ssh-agent requires an explicit list in .ssh/config:

IdentityFile ~/.ssh/id_ed25519
IdentityFile ~/.ssh/id_dsa
IdentityFile ~/.ssh/id_work

Fixing Gnus

I had one final issue, which is rather niche: My newsreader gnus connects to my NNTP feed via a SSH hop. The configuration looked like this:

(setq gnus-select-method
      '(nntp "localhost"
             (nntp-address "myshellhost")
             (nntp-rlogin-program "ssh")
             (nntp-open-connection-function nntp-open-rlogin)
             (nntp-end-of-line "\n")
             (nntp-rlogin-parameters ("nc" "mynntpserver" "nntp"))))

A configuration like this will fail, because Emacs runs this ssh process with a pty, and there ssh will stupidly ask for the password to unlock (if required). But I learned this configuration is outdated anyway, and the recommended version using nntp-open-via-rlogin-and-netcat works correctly, and asks for the password (if required) using x11-ask-sshpass. This is mentioned in a comment in nntp.el, so I’m not the first on to stumble on this.

A fixed version of above would be:

(setq gnus-select-method
      '(nntp "localhost"
             (nntp-address "mynntpserver")
             (nntp-via-address "myshellserver")
             (nntp-via-rlogin-command "ssh")
             (nntp-via-rlogin-command-switches ("-C"))
             (nntp-open-connection-function nntp-open-via-rlogin-and-netcat)))

Enjoy your safer OpenSSH setup!

NP: Slime—A.C.A.B.

Copyright © 2004–2016