chris blogs: January 2011

16jan2011 · Got root?

I lost my “administrative hymen” last month when a server of mine was rooted due to the recent Exim exploit (CVE-2010-4345). This is a post-mortem of the incident.

For shame, it took me almost a month to even detect the break-in, when I wondered why I didn’t get any mail from that address.

The same intruder cracked many machines and installed rootkits on them. He was not very professional: he left traces in the “panic log” of exim when he tried to download a configuration file with wget like this:

wget ... >exim.conf

instead of

wget -O exim.conf ...

… which of course writes the wget status messages into the file, too, which results in fatal parse errors:

# cat /var/log/exim4/paniclog
2010-12-16 01:09:54 string too large in smtp_notquit_exit()
2010-12-16 02:27:17 string too large in smtp_notquit_exit()
2010-12-16 05:12:11 string too large in smtp_notquit_exit()
2010-12-16 14:49:27 string too large in smtp_notquit_exit()
2010-12-16 18:25:20 Exim configuration error in line 1 of /etc/exim4/exim4.conf:
  option setting expected: --2010-12-16 18:25:20--  http://62.141.42.28/exim4.conf
2010-12-16 18:25:20 Exim configuration error in line 1 of /etc/exim4/exim4.conf:
  option setting expected: --2010-12-16 18:25:20--  http://62.141.42.28/exim4.conf
2010-12-16 18:45:03 Exim configuration error in line 1 of /etc/exim4/exim4.conf:
  option setting expected: --2010-12-16 18:25:20--  http://62.141.42.28/exim4.conf
2010-12-16 18:54:13 Exim configuration error in line 1 of /etc/exim4/exim4.conf:
  option setting expected: --2010-12-16 18:54:12--  http://62.141.42.28/exim4.conf
2010-12-16 18:55:02 Exim configuration error in line 1 of /etc/exim4/exim4.conf:
  option setting expected: --2010-12-16 18:54:12--  http://62.141.42.28/exim4.conf
2010-12-16 19:55:02 Exim configuration error in line 1 of /etc/exim4/exim4.conf:
  option setting expected: --2010-12-16 18:54:12--  http://62.141.42.28/exim4.conf

The installation of the rootkit also changed timestamps:

He added a key to root’s .ssh/authorized_keys and started another sshd on a non-standard port (59997). He did not detect that this port was blocked out-bound with iptables.

The exploit circulated around December 7, the patch was released December 10, and I got hacked December 16. And I actually read the news but thought I was not affected. Sigh.

Clearly, this should not happen. I admit the following mistakes:

  • The system ran a old (2007), un-updated version of GRML that I installed in a hurry with grml2hd without spending too much time thinking about the details.

  • It was not updated often because things used to break, being based in Debian unstable, and since it had many packages installed, updating often was a hassle.

  • It came with loads of stuff that I didn’t need, but they cluttered up logwatch and other intrusion detection tools.

  • As a result of this, I didn’t forward Cron messages to a mailbox I actually read.

  • It came with Exim preinstalled, so I used that for the single mailbox I had on that machine.

    (Not that this would have saved much; the bug was also in the last version before the security update, and since the intruder (accidentally) shut down the mail system, I would not have gotten any messages about the break-in.)

    Lessons learned for the future:

  • Set up minimal, but meaningful logging, so it will be read.

  • Send administrative messages to external hosts.

  • Update the system regularily. I now installed Debian squeeze, which is going to be stable “really soon now”, and then will serve its purpose for the next few years with comparatively few updates.

  • Do a minimal install. This time I used grml-debootstrap, which only installed a core Debian system, on which I added exactly the packages I need.

  • Use Postfix instead of Exim. Postfix not only is vastly easier to set up than exim (at least on Debian, Postfix default configuration fits on a single screen, while Exim has a configuration directory filled with almost 2000 lines of text), but also has a better security record. And I use Postfix on other servers too, so I will be more alert if I read about it being unsecure.

    Considerations for the future that I probably won’t follow:

  • Make lots of partitions and sprinkle noexec, nosuid mount options. (Complicates setup and disk space planning.)

  • Setup remote syslog. (I don’t have the resources for this, not worth it for a single machine.)

  • Switch to a distro that doesn’t package antique stuff. (Which?)

  • Use SELinux or such stuff.

    NP: Leonard Cohen—Avalanche

Copyright © 2004–2016