chris blogs

19feb2015 · Six hacks for less(1)

Recently I got around to configuring less, and I collected these few tricks:

  1. Sometimes I look at lists with less, and then do things step-by-step, keeping the current action at the top of the page. This works nicely until you end up at the last page of the file, and then can’t scroll down. You lose track of where you are at and get confused.

    It would be much nicer scrolling down, and filling up the buffer with ~ after the end of file, just as if you had searched in the pager.

    Actually, with ESC-SPC, you can move a full page down, filling up the buffer with ~. Toying around a bit, you’ll find out that you can override the “page length” with a prefix, i.e. 1 ESC-SPC will move down one line only!

    However, this is still inconvenient to type all the time, thus let’s define a keybinding. For this, create a file ~/.lesskey where we will put the key definitions. This file then will be compiled using lesskey(1) and generate a binary configuration file ~/.less. (I guess you can be lucky that m4 is not involved in this mess…)

    One problem is actually binding the key. You can easily bind the cursor down key (\kd) to forw-screen-force, but how do you pass 1? The canonical hack is to use the noaction action, which will behave just like you’ve typed the keys after it. Thus, we write:

    \kd noaction 1\e\40
    j noaction 1\e\40

    (By the way, that #command comment is important to tell lesskey you are defining key commands.)

    Finally, scrolling bliss!

    Actually, scrap that.

    The badly underdocumented key J (and K) will scroll how I want, but you only read about that in the example inside lesskey(1). Therefore, we can just do:

    \kd forw-line-force
    j forw-line-force

    These keybindings are there since at least 1997 and I’ve never found them before…

  2. While we are redefining keys, I’ve always found it a bit clumsy to read multiple files, having to type :n and :p. Using [ and ] is much more convenient (at least on a US keyboard), and by default these keys do things of questionable utility.

    [ prev-file
    ] next-file
  3. Did you ever wish to give feedback from less? Like have a script output some info, and you decide how to go on? Since less always exits with status 0 usually, this I thought this was tricky to do, but the quit action actually can return an arbitrary exit code, encoded as a character.

    I bound Q and :cq (like in vim) to exit with status 1:

    Q quit \1
    :cq quit \1

    Now you can do stuff like look at all files and have them deleted when you press Q instead of q to exit:

    for f in *; do less $f || rm $f; done
  4. I use less a lot to look at patches, git log output, and ocassionally mailboxes. The D command as defined below will move to the next line starting with diff or commit or From␣.

    D noaction j/\^diff|commit|From \n\eu

    It will also “type” ESC-u to hide the highlighting. Now I can simply press D to jump to the next chunk.

  5. To return to where you started from after a search or going to the end of file, type ''. Typing '' again will go back, so this is also nice to toggle between two search results.

  6. Back in the old days of X11R2(?) there was a tool called xless, which was exactly that: a pager like less that ran in its own X11 window. It’s quite useful. We can recreate this by combining a X11 terminal emulator and plain less with a small zsh snippet:

    xless() {
        exec {stdin}<&0 {stderr}>&2
        exec urxvt -e sh -c "less ${(j: :)${(qq)@}} </dev/fd/$stdin 2>/dev/fd/$stderr"
      } &!

    Watch the trick how we pass the stdin/stderr file descriptors and the file arguments!

    Now you can just run command-spitting-out-loads | xless and the output will be shown in a new terminal and not lock your shell.

    NP: Feine Sahne Fischfilet—Dreieinhalb Meter Lichtgestalt

17feb2015 · 10 fancy zsh tricks you may not know...

Wow, almost two years have passed since the latest installment of our favorite clickbait zsh tricks series.

  1. When editing long lines in the zle line editor, sometimes you want to move “by physical line”, that is, to the character in the terminal line below (like gj and gk in vim).

    We can fake that feature by finding out the terminal width and moving charwise:

    _physical_up_line()   { zle backward-char -n $COLUMNS }
    _physical_down_line() { zle forward-char  -n $COLUMNS }
    zle -N physical-up-line _physical_up_line
    zle -N physical-down-line _physical_down_line
    bindkey "\e\e[A" physical-up-line
    bindkey "\e\e[B" physical-down-line

    Now, ESC-up and ESC-down will move by physical line.

  2. Sometimes it’s nice to do things in random order. Many tools such as image viewers, music or media players have a “shuffle” mode, but when they don’t, you can help yourself with this small trick:


    Just append ($SHUF) to any glob, and get the matches shuffled:

    % touch a b c d
    % echo *($SHUF)
    d c a b
    % echo *($SHUF)
    c a d b

    Note that this shuffle is slightly biased, but it should not matter in practice. In doubt, use shuf or sort -R or something else…

  3. Are you getting sick of typing cd ../../.. all the time? Why not type up 3?

    up() {
      local op=print
      [[ -t 1 ]] && op=cd
      case "$1" in
        '') up 1;;
        -*|+*) $op ~$1;;
        <->) $op $(printf '../%.0s' {1..$1});;
        *) local -a seg; seg=(${(s:/:)PWD%/*})
           local n=${(j:/:)seg[1,(I)$1*]}
           if [[ -n $n ]]; then
             $op /$n
             print -u2 up: could not find prefix $1 in $PWD
             return 1

    With this helper function, you can do a lot more actually: Say you are in ~/src/zsh/Src/Builtins and want to go to ~/src/zsh. Just say up zsh. Or even just up z.

    And as a bonus, if you capture the output of up, it will print the directory you want, and not change to it. So you can do:

    mv foo.c $(up zsh)
  4. Previous tricks (#6/#7) introduced the dirstack and how to navigate it. But why type cd -<TAB> and figure out the directory you want to go to when you simply can type cd ~[zsh] and go to the first directory in the dirstack matching zsh? For this, we define the zsh dynamic directory function:

    _mydirstack() {
      local -a lines list
      for d in $dirstack; do
        lines+="$(($#lines+1)) -- $d"
      _wanted -V directory-stack expl 'directory stack' \
        compadd "$@" -ld lines -S']/' -Q -a list
    zsh_directory_name() {
      case $1 in
        c) _mydirstack;;
        n) case $2 in
             <0-9>) reply=($dirstack[$2]);;
             *) reply=($dirstack[(r)*$2*]);;
        d) false;;

    The first function is just the completion, so cd ~[<TAB> will work as well.

  5. Did you ever want to move a file with spaces in the name, and mixed up argument order?

    % mv last-will.tex My\ Last\ Will.rtf

    Pressing ESC-t (transpose-words) between the file names will do the wrong thing by default:

    % mv My last-will.tex\ Last\ Will.rtf

    Luckily, we can teach transpose-words to understand shell syntax:

    autoload -Uz transpose-words-match
    zstyle ':zle:transpose-words' word-style shell
    zle -N transpose-words transpose-words-match


    % mv My\ Last\ Will.rtf last-will.tex
  6. If you are an avid Emacs user like me, you’ll find this function useful. It enters the directory the currently active Emacs file resides in:

    cde() {
      cd ${(Q)~$(emacsclient -e '(with-current-buffer
                                   (window-buffer (selected-window))
                                   default-directory) ')}

    You need the emacs-server functionality enabled for this to work.

  7. I’m working on many different systems and try to keep a portable .zshrc between those. One problem used to be setting $PATH portably, because there is quite some difference among systems. I now let zsh figure out what belongs to $PATH:

    export PATH
    path=( ${(u)^path:A}(N-/) )

    The last line will normalize all paths, and remove duplicates and nonexisting directories. Also, notice how I pick up the latest Ruby version to find the Gem bin dir by sorting them numerically.

  8. One of the hardest things is to set the xterm title “correctly”, because most people do it wrong in some way, and then it will break when you have literal tabs or percent signs or tildes in your command line. Here is what I currently use:

    case "$TERM" in
        precmd() {  print -Pn "\e]0;%m: %~\a" }
        preexec() { print -n "\e]0;$HOST: ${(q)1//(#m)[$'\000-\037\177-']/${(q)MATCH}}\a" }
  9. For a cheap, but secure password generator, you can use this:

    zpass() {
      LC_ALL=C tr -dc '0-9A-Za-z_@#%*,.:?!~' < /dev/urandom | head -c${1:-10}
  10. Sometimes it’s interesting to find a file residing in some directory “above” (e.g. Makefile, .git and similar). We can glob these by repeating ../ using the #-operator (You have EXTENDED_GLOB enabled, right?). This will result in all matches, so let’s first sort them by directory depth:

    % pwd
    % print -l (../)

    Now we can pick the first one, and also make the file name absolute:

    % print (../)[1]:A) 

    I knew the #-operator, but it never occurred to me to use it this way before.

    Until next time!

    NP: Pierced Arrows—On Our Way

24dec2014 · Merry Christmas!

Consumers' crèche

Frohe Weihnachten, ein schönes Fest, und einen guten Rutsch ins neue Jahr wünscht euch Christian Neukirchen

Merry Christmas and a Happy New Year!

Bitte lesen: Liebeserklärung an die Vielfalt - eine Weihnachtsbotschaft.

NP: Against Me!—Holy Shit

03dec2014 · Recovering a Git repository from filesystem corruption

Recently I had to fix a Git repository where something unfortunate happened: probably due to accessing a NTFS partition that was still mounted in a hibernated alternative operating system, several files became corrupted (and actually had their contents exchanged with different files on the disk!).

git fsck discovered corrupted blobs, which we tried to recover first, when we detected their content does not make sense at all. These blobs were irrevocably lost, but we still wanted to get out the rest of the Git history alive.

Usually, applying the following technique is not necessary, because you can either just clone again from your Git upstream or recover the repository from the last backup—but both did not exist in this case.

The corrupted blobs actually all belonged to a single commit that happened a few months ago. The solution was thus to remove this commit from the history, keeping all other trees intact (of course, commit ids would change, but the content won’t).

I first tried to do this with git rebase, but it is of course the wrong tool, since it will try to remove the change of the defect in all following history.

Finally, I had a use-case for git filter-branch. To make it short, we can filter out the defect commit using:

git filter-branch --commit-filter \
  '[ $GIT_COMMIT = badbadbadbad ] && skip_commit "$@" ||
                                     git commit-tree "$@"'

This will rewrite all commits after badbadbadbad, but not touch their actual content.

git fsck still was not happy, we thus made a clean copy using

git clone --no-local --no-hardlinks mybrokengit myfixedgit

Now git fsck reported no errors and all other revisions were still ok. (Also, the blobs have been packed, so the next data corruption will be more fatal… ;))

I cannot think of any other version control system where a repair like this would have been possible. Thanks, Git!

NP: Against Me!—Exhaustion & Disgust

16mar2014 · Review: Learning Shell Scripting with Zsh

Learning Shell Scripting with Zsh
by Gastón Festari.
Packt Publishing, Birmingham 2014.
132 pages.

[Full disclosure: I have received a digital copy of the book in exchange for this review.]

The book is titled Learning Shell Scripting with Zsh: Your one-stop guide to reading, writing, and debugging simple and complex Z shell scripts which could not be more wrong. A far more appropriate title would have been: Getting started with Zsh: Your guide to interactive use and first steps of customization.

A first glance through the table of contents proves me right: There are chapters on “Getting started”, “Alias and History”, “Advanced Editing”, “Globbing”, “Completion”, and “Tips and Tricks”, but no explicit mention of writing shell scripts for purposes other than customization of an interactive shell.

I’m not completely sure of the target audience: The book assumes basic familiarity with (Bourne) shell already, half-heartedly explains pipes or redirections as well as how to define shell functions, but e.g. there is no mention of how to pass or parse arguments. After a single example of a very simple completion function in Chapter 5 (with explanation), the reader is assumed know enough for writing his own completions. I highly doubt that.

I found the book riddled with small mistakes, imprecise to wrong explanations and plain terribly sloppy typesetting of the code examples, which hurts particularly in a book about shell where syntax is lax but proper newlines matter. Examples are often not very well chosen and occasionally confusing even to me, who is very familiar with the topic.

The over-emphatic style with its needless rambling and cheeky language does not save the book but probably annoys the reader even more. (I hope not to ever read “You know, because widescreen.” in a book again.) Without, a lot more actual content and perhaps a real introduction to shell usage would have fit into these compact 118 pages.

The book finishes with a recommendation to read From Bash To Z Shell next. All I can recommend is: better skip this book completely and start with that one if you are interested in a good book about zsh.

Rating: 2 of 5 points.

NP: Against Me!—White Crosses

02feb2014 · The road to OpenSSH bliss: ED25519 and the IdentityPersist patch

The recent release of OpenSSH 6.5 had many convincing new features to make me update to it early, quoting from the release notes:

  • support for key exchange using elliptic-curve Diffie–Hellman in Daniel Bernstein’s Curve25519
  • support for Ed25519 as a public key type
  • a new private key format that uses a bcrypt KDF to better protect keys at rest
  • a new transport cipher that combines Daniel Bernstein’s ChaCha20 stream cipher and Poly1305 MAC to build an authenticated encryption mode

Since OpenSSH 5.7, there is support for ECDSA keys according to RFC5656. The problem with this schema is that it uses NIST curves generally. Due to recent events, everyone is (rightfully) more paranoid now, and there are reasons to consider these curves to be problematic. Thus, I decided to disable support for ECDSA host keys and use the superior ED25519 scheme for new keys.

However, I also need to access many machines which don’t run the latest version of OpenSSH, but for these we can at least make use of the new, safer public key format.

First, I will show how to update your OpenSSH installation to make use of the new features, and then I’ll explain what else I had to do to make everything work correctly.

Upgrading OpenSSH

Arch is, at the time of writing, providing binaries in “testing”, but plucking a single package is easy:

# pacman -U

After merging configuration files (if required), we edit /etc/ssh/sshd_config and spell out the HostKeys to disable the built-in defaults, which include ECDSA.

HostKey /etc/ssh/ssh_host_rsa_key
HostKey /etc/ssh/ssh_host_dsa_key
HostKey /etc/ssh/ssh_host_ed25519_key

Upon restarting SSH, a new ED25519 hostkey will be generated. Using ignite:

# sv restart sshd

Essentially, that’s it. Let’s check that the ECDSA hostkey is disabled:

% ssh-keyscan -t ecdsa,ed25519 localhost
# localhost SSH-2.0-OpenSSH_6.5
no hostkey alg
# localhost SSH-2.0-OpenSSH_6.5
localhost ssh-ed25519 AAAA...

(AFAIU, ECDSA user keys in authorized_keys will still work. It’s your task to replace them with ED22519 ones, I have found no way to blacklist them.)

We continue by creating a new ED25519 key for the user:

% ssh-keygen -t ed25519
Generating public/private ed25519 key pair.
Enter file in which to save the key (/home/chris/.ssh/id_ed25519): 
Enter passphrase (empty for no passphrase): 
Enter same passphrase again:

For testing, we can add it to the locally accepted keys:

% cat .ssh/ >>.ssh/authorized_keys

Finally, we can test it:

% ssh chris@localhost
Enter passphrase for key '/home/chris/.ssh/id_ed25519':

It seems to work! (Perhaps you’ll see an update of the fingerprint if you had the ECDSA one saved in .ssh/known_hosts.)

I was happy about that until I realized that ssh asked for the passphrase, and not my gnome-keyring-daemon… which brings us to part two of this post.

Migrating to ssh-agent and the IdentityPersist patch

Since new ED25519 keys are always stored in the new bcrypt format, they won’t work (as of right now) with SSH agents that don’t support it (I know of gnome-keyring and mate-keyring; gpg-agent stores the key itself, anyway). Essentially, only plain ssh-agent supports it, but I used to hate ssh-agent since it doesn’t support adding keys upon use: gnome-keyring will ask for the password and keep the key unlocked. Since I try to keep my keys locked when I’m not using them, I don’t want to keep them unlocked in every session, and neither unlock them manually, because that is inconvenient.

Luckily, I found this patch which adds the key automatically. I hope it gets accepted, because it is very useful. I couldn’t wait and patched OpenSSH myself. With this tiny patch, I could finally drop my use of gnome-keyring, which is one more step towards a GTK3-free desktop.

I run ssh-agent from my .xinitrc:

eval $(ssh-agent -s)
xscreensaver-ssh-helper &

Now, we can convert our old keys to the new storage format:

% ssh-keygen -p -o -f ~/.ssh/id_dsa
Enter old passphrase: 
Key has comment 'id_dsa'
Enter new passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved with the new passphrase.

The key file should now start with -----BEGIN OPENSSH PRIVATE KEY-----. The public key format is unchanged.

For the IdentityPersist patch, we need to add a line to .ssh/config, stating the lifetime of the key (or true for infinity):

IdentityPersist 300

Now, we can try everything together:

% ssh myhost
Enter passphrase for key '/home/chris/.ssh/id_dsa': 
Identity added: /home/chris/.ssh/id_dsa (id_dsa)

Another SSH connect within the next 5 minutes will connect without asking for a password.

I noticed one more difference between gnome-keyring and ssh-agent: gnome-keyring would unlock any key that has a *.pub in ~/.ssh, while ssh-agent requires an explicit list in .ssh/config:

IdentityFile ~/.ssh/id_ed25519
IdentityFile ~/.ssh/id_dsa
IdentityFile ~/.ssh/id_work

Fixing Gnus

I had one final issue, which is rather niche: My newsreader gnus connects to my NNTP feed via a SSH hop. The configuration looked like this:

(setq gnus-select-method
      '(nntp "localhost"
             (nntp-address "myshellhost")
             (nntp-rlogin-program "ssh")
             (nntp-open-connection-function nntp-open-rlogin)
             (nntp-end-of-line "\n")
             (nntp-rlogin-parameters ("nc" "mynntpserver" "nntp"))))

A configuration like this will fail, because Emacs runs this ssh process with a pty, and there ssh will stupidly ask for the password to unlock (if required). But I learned this configuration is outdated anyway, and the recommended version using nntp-open-via-rlogin-and-netcat works correctly, and asks for the password (if required) using x11-ask-sshpass. This is mentioned in a comment in nntp.el, so I’m not the first on to stumble on this.

A fixed version of above would be:

(setq gnus-select-method
      '(nntp "localhost"
             (nntp-address "mynntpserver")
             (nntp-via-address "myshellserver")
             (nntp-via-rlogin-command "ssh")
             (nntp-via-rlogin-command-switches ("-C"))
             (nntp-open-connection-function nntp-open-via-rlogin-and-netcat)))

Enjoy your safer OpenSSH setup!

NP: Slime—A.C.A.B.

24dec2013 · Merry Christmas!

Wow Such Christmas

Frohe Weihnachten, ein schönes Fest, und einen guten Rutsch ins neue Jahr wünscht euch Christian Neukirchen

Merry Christmas and a Happy New Year!

NP: ADULT.—We Will Rest

31jul2013 · Summer of Scripts: tarhash and pacverify

For the final installment of my “Summer of Scripts”, I’m showing a generic tool I wrote and a specialized version of it useful to Arch Linux users.

tarhash computes checksums for files inside tarballs without unpacking them.

% tarhash ~/src/mutt-1.5.21.tar.gz | head -3
9cc2ec57dc43e6768516898ebb90f3d76cb24d72  ./mutt-1.5.21/ABOUT-NLS
a87360b6b5b8d6d2cdeb83d54b3aa4a0a35bf090  ./mutt-1.5.21/BEWARE
5d1b9cfe259891e3408938afa6bdd3821953973f  ./mutt-1.5.21/COPYRIGHT

It defaults to SHA1, but you can specify other hashes easily:

% tarhash --sha512 ~/src/mutt-1.5.21.tar.gz | head -3
808297837049d5b84b54ba780f87f08c22fb83ebbc62edaf3085966428593e76d28a7bf08cc4f029ee24a3a455fa292aac064b01ab8700240cb9ab0cc0284fae  ./mutt-1.5.21/ABOUT-NLS
b0ac0f3c9297c0bf26c20ce58bf7bb234bd2ab84e5ee545345f39142e83f4d93ca1eaf406d77fb8ffab8ac748bb25ea8891412f6dc3d0058db73de73442b38eb  ./mutt-1.5.21/BEWARE
68c306e6fc7a0b9a1dc47bbc700f034bc40c6c4e2125c35ce24deba44a95eb8113ce8dbd81a9fd9ad7208d28108e36a8f6bb078de416e42d7ad46271b13cca77  ./mutt-1.5.21/COPYRIGHT

Also, since it uses the powerful bsdtar of libarchive, it supports other archive formats as well:

% tarhash /usr/lib/python3.3/test/
da39a3ee5e6b4b0d3255bfef95601890afd80709  ./a/b/c

Since the hashes of tarballs themselves easily can change (due to changed metadata, different order of files, etc…), this tool is nice to compare tarballs contentwise.

However, tarhash actually is a by-product of pacverify, which tries to find files that have changed in your Arch Linux installation, compared to the original packages. Simply run it and after some time you’ll see output like:

cpupower 3.10-1: /./etc/default/cpupower: FAILED
cpupower 3.10-1: sha256sum: WARNING: 1 computed checksum did NOT match

Of course, it’s ok that some config files have been changed, but that is your job to decide.

pacverify is also good if you think some (possibly undetected?) filesystem corruption took place (or someone fiddled in your files, but be sure to compare against verified package files them).

That’s it for the summer. I hope you had fun and learned something. :)

NP: Toxoplasma—Alte Zeichen

30jul2013 · Summer of Scripts: ssh-chain

ssh-chain wasn’t written by me, but it’s so incredibly useful and I use it a lot, which makes it a perfect fit to be listed here.

Often (admittedly, with the advent of IPv6 less often) you want to ssh to a machine you can’t connect to directly, and have to “hop” over a proxy host. Many people configure such hops statically in their .ssh/config, for example (the latter variant works with modern versions of OpenSSH only):

Host office
  ProxyCommand ssh -qAx nc -q5 22

Host office2
  ProxyCommand ssh -W

However, these configurations get nasty quickly if you need multiple hops or want to use different hops depending on other things.

ssh-chain is a simple Perl script you put into the path of your remote machines, and add the following line to .ssh/config:

Host *^*
  ProxyCommand ssh-chain %h %p

Now, you can use that host to hop to other machines over SSH, by just giving a “path” (think UUCP ;)) of immediate hosts (which all need ssh-chain installed):

ssh faraway^hop3^hop2^hop1

Since ssh is so central to many tools, this syntax also works for scp, rsync, git, and many others. Really useful if you work with non-trivial network topologies.

NP: Toxoplasma—Weltverbesserer

29jul2013 · Summer of Scripts: now-playing

now-playing is the script generating these “now playing”-lines you can find at the bottom of this post, but I also use it to post the currently playing song to IRC, for example.

Thanks to mpd this is really easy. Back when I used OS X, it was a bit more complicated, having to interface with AppleScript (yikes) and so on. I restored it from an old backup:


osascript -e '
tell application "iTunes"
        set theTitle to name of current track
        set theArtist to artist of current track
        return {theArtist,"---", theTitle} as string
end tell'

NP: Toxoplasma—Alles oder nichts

Copyright © 2004–2013