chris blogs

03dec2014 · Recovering a Git repository from filesystem corruption

Recently I had to fix a Git repository where something unfortunate happened: probably due to accessing a NTFS partition that was still mounted in a hibernated alternative operating system, several files became corrupted (and actually had their contents exchanged with different files on the disk!).

git fsck discovered corrupted blobs, which we tried to recover first, when we detected their content does not make sense at all. These blobs were irrevocably lost, but we still wanted to get out the rest of the Git history alive.

Usually, applying the following technique is not necessary, because you can either just clone again from your Git upstream or recover the repository from the last backup—but both did not exist in this case.

The corrupted blobs actually all belonged to a single commit that happened a few months ago. The solution was thus to remove this commit from the history, keeping all other trees intact (of course, commit ids would change, but the content won’t).

I first tried to do this with git rebase, but it is of course the wrong tool, since it will try to remove the change of the defect in all following history.

Finally, I had a use-case for git filter-branch. To make it short, we can filter out the defect commit using:

git filter-branch --commit-filter \
  '[ $GIT_COMMIT = badbadbadbad ] && skip_commit "$@" ||
                                     git commit-tree "$@"'

This will rewrite all commits after badbadbadbad, but not touch their actual content.

git fsck still was not happy, we thus made a clean copy using

git clone --no-local --no-hardlinks mybrokengit myfixedgit

Now git fsck reported no errors and all other revisions were still ok. (Also, the blobs have been packed, so the next data corruption will be more fatal… ;))

I cannot think of any other version control system where a repair like this would have been possible. Thanks, Git!

NP: Against Me!—Exhaustion & Disgust

16mar2014 · Review: Learning Shell Scripting with Zsh

Learning Shell Scripting with Zsh
by Gastón Festari.
Packt Publishing, Birmingham 2014.
132 pages.

[Full disclosure: I have received a digital copy of the book in exchange for this review.]

The book is titled Learning Shell Scripting with Zsh: Your one-stop guide to reading, writing, and debugging simple and complex Z shell scripts which could not be more wrong. A far more appropriate title would have been: Getting started with Zsh: Your guide to interactive use and first steps of customization.

A first glance through the table of contents proves me right: There are chapters on “Getting started”, “Alias and History”, “Advanced Editing”, “Globbing”, “Completion”, and “Tips and Tricks”, but no explicit mention of writing shell scripts for purposes other than customization of an interactive shell.

I’m not completely sure of the target audience: The book assumes basic familiarity with (Bourne) shell already, half-heartedly explains pipes or redirections as well as how to define shell functions, but e.g. there is no mention of how to pass or parse arguments. After a single example of a very simple completion function in Chapter 5 (with explanation), the reader is assumed know enough for writing his own completions. I highly doubt that.

I found the book riddled with small mistakes, imprecise to wrong explanations and plain terribly sloppy typesetting of the code examples, which hurts particularly in a book about shell where syntax is lax but proper newlines matter. Examples are often not very well chosen and occasionally confusing even to me, who is very familiar with the topic.

The over-emphatic style with its needless rambling and cheeky language does not save the book but probably annoys the reader even more. (I hope not to ever read “You know, because widescreen.” in a book again.) Without, a lot more actual content and perhaps a real introduction to shell usage would have fit into these compact 118 pages.

The book finishes with a recommendation to read From Bash To Z Shell next. All I can recommend is: better skip this book completely and start with that one if you are interested in a good book about zsh.

Rating: 2 of 5 points.

NP: Against Me!—White Crosses

02feb2014 · The road to OpenSSH bliss: ED25519 and the IdentityPersist patch

The recent release of OpenSSH 6.5 had many convincing new features to make me update to it early, quoting from the release notes:

  • support for key exchange using elliptic-curve Diffie–Hellman in Daniel Bernstein’s Curve25519
  • support for Ed25519 as a public key type
  • a new private key format that uses a bcrypt KDF to better protect keys at rest
  • a new transport cipher that combines Daniel Bernstein’s ChaCha20 stream cipher and Poly1305 MAC to build an authenticated encryption mode

Since OpenSSH 5.7, there is support for ECDSA keys according to RFC5656. The problem with this schema is that it uses NIST curves generally. Due to recent events, everyone is (rightfully) more paranoid now, and there are reasons to consider these curves to be problematic. Thus, I decided to disable support for ECDSA host keys and use the superior ED25519 scheme for new keys.

However, I also need to access many machines which don’t run the latest version of OpenSSH, but for these we can at least make use of the new, safer public key format.

First, I will show how to update your OpenSSH installation to make use of the new features, and then I’ll explain what else I had to do to make everything work correctly.

Upgrading OpenSSH

Arch is, at the time of writing, providing binaries in “testing”, but plucking a single package is easy:

# pacman -U

After merging configuration files (if required), we edit /etc/ssh/sshd_config and spell out the HostKeys to disable the built-in defaults, which include ECDSA.

HostKey /etc/ssh/ssh_host_rsa_key
HostKey /etc/ssh/ssh_host_dsa_key
HostKey /etc/ssh/ssh_host_ed25519_key

Upon restarting SSH, a new ED25519 hostkey will be generated. Using ignite:

# sv restart sshd

Essentially, that’s it. Let’s check that the ECDSA hostkey is disabled:

% ssh-keyscan -t ecdsa,ed25519 localhost
# localhost SSH-2.0-OpenSSH_6.5
no hostkey alg
# localhost SSH-2.0-OpenSSH_6.5
localhost ssh-ed25519 AAAA...

(AFAIU, ECDSA user keys in authorized_keys will still work. It’s your task to replace them with ED22519 ones, I have found no way to blacklist them.)

We continue by creating a new ED25519 key for the user:

% ssh-keygen -t ed25519
Generating public/private ed25519 key pair.
Enter file in which to save the key (/home/chris/.ssh/id_ed25519): 
Enter passphrase (empty for no passphrase): 
Enter same passphrase again:

For testing, we can add it to the locally accepted keys:

% cat .ssh/ >>.ssh/authorized_keys

Finally, we can test it:

% ssh chris@localhost
Enter passphrase for key '/home/chris/.ssh/id_ed25519':

It seems to work! (Perhaps you’ll see an update of the fingerprint if you had the ECDSA one saved in .ssh/known_hosts.)

I was happy about that until I realized that ssh asked for the passphrase, and not my gnome-keyring-daemon… which brings us to part two of this post.

Migrating to ssh-agent and the IdentityPersist patch

Since new ED25519 keys are always stored in the new bcrypt format, they won’t work (as of right now) with SSH agents that don’t support it (I know of gnome-keyring and mate-keyring; gpg-agent stores the key itself, anyway). Essentially, only plain ssh-agent supports it, but I used to hate ssh-agent since it doesn’t support adding keys upon use: gnome-keyring will ask for the password and keep the key unlocked. Since I try to keep my keys locked when I’m not using them, I don’t want to keep them unlocked in every session, and neither unlock them manually, because that is inconvenient.

Luckily, I found this patch which adds the key automatically. I hope it gets accepted, because it is very useful. I couldn’t wait and patched OpenSSH myself. With this tiny patch, I could finally drop my use of gnome-keyring, which is one more step towards a GTK3-free desktop.

I run ssh-agent from my .xinitrc:

eval $(ssh-agent -s)
xscreensaver-ssh-helper &

Now, we can convert our old keys to the new storage format:

% ssh-keygen -p -o -f ~/.ssh/id_dsa
Enter old passphrase: 
Key has comment 'id_dsa'
Enter new passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved with the new passphrase.

The key file should now start with -----BEGIN OPENSSH PRIVATE KEY-----. The public key format is unchanged.

For the IdentityPersist patch, we need to add a line to .ssh/config, stating the lifetime of the key (or true for infinity):

IdentityPersist 300

Now, we can try everything together:

% ssh myhost
Enter passphrase for key '/home/chris/.ssh/id_dsa': 
Identity added: /home/chris/.ssh/id_dsa (id_dsa)

Another SSH connect within the next 5 minutes will connect without asking for a password.

I noticed one more difference between gnome-keyring and ssh-agent: gnome-keyring would unlock any key that has a *.pub in ~/.ssh, while ssh-agent requires an explicit list in .ssh/config:

IdentityFile ~/.ssh/id_ed25519
IdentityFile ~/.ssh/id_dsa
IdentityFile ~/.ssh/id_work

Fixing Gnus

I had one final issue, which is rather niche: My newsreader gnus connects to my NNTP feed via a SSH hop. The configuration looked like this:

(setq gnus-select-method
      '(nntp "localhost"
             (nntp-address "myshellhost")
             (nntp-rlogin-program "ssh")
             (nntp-open-connection-function nntp-open-rlogin)
             (nntp-end-of-line "\n")
             (nntp-rlogin-parameters ("nc" "mynntpserver" "nntp"))))

A configuration like this will fail, because Emacs runs this ssh process with a pty, and there ssh will stupidly ask for the password to unlock (if required). But I learned this configuration is outdated anyway, and the recommended version using nntp-open-via-rlogin-and-netcat works correctly, and asks for the password (if required) using x11-ask-sshpass. This is mentioned in a comment in nntp.el, so I’m not the first on to stumble on this.

A fixed version of above would be:

(setq gnus-select-method
      '(nntp "localhost"
             (nntp-address "mynntpserver")
             (nntp-via-address "myshellserver")
             (nntp-via-rlogin-command "ssh")
             (nntp-via-rlogin-command-switches ("-C"))
             (nntp-open-connection-function nntp-open-via-rlogin-and-netcat)))

Enjoy your safer OpenSSH setup!

NP: Slime—A.C.A.B.

24dec2013 · Merry Christmas!

Wow Such Christmas

Frohe Weihnachten, ein schönes Fest, und einen guten Rutsch ins neue Jahr wünscht euch Christian Neukirchen

Merry Christmas and a Happy New Year!

NP: ADULT.—We Will Rest

31jul2013 · Summer of Scripts: tarhash and pacverify

For the final installment of my “Summer of Scripts”, I’m showing a generic tool I wrote and a specialized version of it useful to Arch Linux users.

tarhash computes checksums for files inside tarballs without unpacking them.

% tarhash ~/src/mutt-1.5.21.tar.gz | head -3
9cc2ec57dc43e6768516898ebb90f3d76cb24d72  ./mutt-1.5.21/ABOUT-NLS
a87360b6b5b8d6d2cdeb83d54b3aa4a0a35bf090  ./mutt-1.5.21/BEWARE
5d1b9cfe259891e3408938afa6bdd3821953973f  ./mutt-1.5.21/COPYRIGHT

It defaults to SHA1, but you can specify other hashes easily:

% tarhash --sha512 ~/src/mutt-1.5.21.tar.gz | head -3
808297837049d5b84b54ba780f87f08c22fb83ebbc62edaf3085966428593e76d28a7bf08cc4f029ee24a3a455fa292aac064b01ab8700240cb9ab0cc0284fae  ./mutt-1.5.21/ABOUT-NLS
b0ac0f3c9297c0bf26c20ce58bf7bb234bd2ab84e5ee545345f39142e83f4d93ca1eaf406d77fb8ffab8ac748bb25ea8891412f6dc3d0058db73de73442b38eb  ./mutt-1.5.21/BEWARE
68c306e6fc7a0b9a1dc47bbc700f034bc40c6c4e2125c35ce24deba44a95eb8113ce8dbd81a9fd9ad7208d28108e36a8f6bb078de416e42d7ad46271b13cca77  ./mutt-1.5.21/COPYRIGHT

Also, since it uses the powerful bsdtar of libarchive, it supports other archive formats as well:

% tarhash /usr/lib/python3.3/test/
da39a3ee5e6b4b0d3255bfef95601890afd80709  ./a/b/c

Since the hashes of tarballs themselves easily can change (due to changed metadata, different order of files, etc…), this tool is nice to compare tarballs contentwise.

However, tarhash actually is a by-product of pacverify, which tries to find files that have changed in your Arch Linux installation, compared to the original packages. Simply run it and after some time you’ll see output like:

cpupower 3.10-1: /./etc/default/cpupower: FAILED
cpupower 3.10-1: sha256sum: WARNING: 1 computed checksum did NOT match

Of course, it’s ok that some config files have been changed, but that is your job to decide.

pacverify is also good if you think some (possibly undetected?) filesystem corruption took place (or someone fiddled in your files, but be sure to compare against verified package files them).

That’s it for the summer. I hope you had fun and learned something. :)

NP: Toxoplasma—Alte Zeichen

30jul2013 · Summer of Scripts: ssh-chain

ssh-chain wasn’t written by me, but it’s so incredibly useful and I use it a lot, which makes it a perfect fit to be listed here.

Often (admittedly, with the advent of IPv6 less often) you want to ssh to a machine you can’t connect to directly, and have to “hop” over a proxy host. Many people configure such hops statically in their .ssh/config, for example (the latter variant works with modern versions of OpenSSH only):

Host office
  ProxyCommand ssh -qAx nc -q5 22

Host office2
  ProxyCommand ssh -W

However, these configurations get nasty quickly if you need multiple hops or want to use different hops depending on other things.

ssh-chain is a simple Perl script you put into the path of your remote machines, and add the following line to .ssh/config:

Host *^*
  ProxyCommand ssh-chain %h %p

Now, you can use that host to hop to other machines over SSH, by just giving a “path” (think UUCP ;)) of immediate hosts (which all need ssh-chain installed):

ssh faraway^hop3^hop2^hop1

Since ssh is so central to many tools, this syntax also works for scp, rsync, git, and many others. Really useful if you work with non-trivial network topologies.

NP: Toxoplasma—Weltverbesserer

29jul2013 · Summer of Scripts: now-playing

now-playing is the script generating these “now playing”-lines you can find at the bottom of this post, but I also use it to post the currently playing song to IRC, for example.

Thanks to mpd this is really easy. Back when I used OS X, it was a bit more complicated, having to interface with AppleScript (yikes) and so on. I restored it from an old backup:


osascript -e '
tell application "iTunes"
        set theTitle to name of current track
        set theArtist to artist of current track
        return {theArtist,"---", theTitle} as string
end tell'

NP: Toxoplasma—Alles oder nichts

28jul2013 · Summer of Scripts: 1t

1t is an old but useful tool I wrote. I like to have small windows showing logfiles with tail -F on my desktop, and it always annoyed me that this wastes one line, showing nothing but the cursor.

1t to the rescue, which disables the cursor and only outputs the final newline after the next line has arrived, thus making use of the last display line as well:

screenshot of 1t


% tail -F /var/log/messages.log | 1t

NP: Fliehende Stürme—Alles falsch

27jul2013 · Summer of Scripts: spongegrep

spongegrep is like grep, but for whole files: if any line matches the regular expression given, all lines are output:

% utter foo bar baz | spongegrep foo
% utter foo bar baz | spongegrep quux || echo not found
not found

It also supports negation:

% utter foo bar baz | spongegrep -v quux
% utter foo bar baz | spongegrep -v foo

The benefit over using grep -q foo file && cat file is that spongegrep works on streams and doesn’t need rewinding.

A good use is checking logs or command outputs for error messages, and eating them up if nothing worrisome happened (also nice in crontabs):

% sensors | spongegrep ALARM

NP: Fliehende Stürme—Es gibt mich nicht!

26jul2013 · Summer of Scripts: stee

The stee command made an appearance at day 1 already, but it’s so useful to be it deserves its own mention.

While tee(1) copies standard input to standard output as well as the file argument, stee is silent tee and doesn’t copy to standard output (well, it throws the output away).

What’s that good for, you may wonder? Well, it allows you to write into a file without having to setup a shell direction (nor fiddle with argument strings as with dd of=...). This saves you an additional layer of quoting:

% date | ssh localhost 'cat >/tmp/out'
% date | ssh localhost stee /tmp/out

Also, an often overlooked feature of tee is that it supports output to multiple files. stee of course does as well:

% fortune | stee a b c
% sum a b c
51569     1 a
51569     1 b
51569     1 c

# echo min_power | stee /sys/class/scsi_host/*/link_power_management_policy

(Of course, a good shell can do that already internally.)

NP: Die Schnitter—Klöne

Copyright © 2004–2013