step 2: anti-spam, anti-virus

ow. my head.

Not only is the spam problem on the internet horrible, but so is the how-to-implement-spam-prevention problem. There's sooo many walkthroughs, guides, howtos and different packages for different UNIX flavors that to attempt to accomplish the task. Here's the list of tools I'm starting off with:

amavis-new
spamassassin
clamav
razor-agents
pyzor

I started with the adminspotting walkthrough but that's debian based and my CentOS box needs additional configuration. I read over the SA wiki, but still didn't fit right. I think the closest is the HowToForge howto, but my virtual setup is different (file based vs. mysql based). I also added OpenProtect's sa-update channel and I built my own pyzor rpm using the fedora spec file. Below are some key config steps. I might have missed one or two, but I think I got "the big ones". Of course, there are more components that I could add (dcc, DomainKeys, spf, etc. etc.) but my VM is already wheezing on memory and thats with only 2 amavisd children and zero mail traffic.

Man, what a pain in the ass.

---8<---
# install rpmforge pkgs
yum install spamassassin
yum install clamav-db clamav clamd
yum install amavisd-new yum install razor-agents
rpm -ihv /www/src/rpms/pyzor-0.4.0-11.noarch.rpm

#
# for /etc/postfix/main.cf:
#

#
# amavis
#
content_filter=smtp-amavis:[127.0.0.1]:10024
receive_override_options = no_address_mappings

#
# for /etc/postfix/master.cf:
#

#
# amavis setup
#
smtp-amavis unix - - n - 2 smtp
-o smtp_data_done_timeout=1200
-o smtp_send_xforward_command=yes

127.0.0.1:10025 inet n - n - - smtpd
-o content_filter=
-o local_recipient_maps=
-o relay_recipient_maps=
-o smtpd_restriction_classes=
-o smtpd_client_restrictions=
-o smtpd_helo_restrictions=
-o smtpd_sender_restrictions=
-o smtpd_recipient_restrictions=permit_mynetworks,reject
-o mynetworks=127.0.0.0/8
-o strict_rfc821_envelopes=yes
-o smtpd_error_sleep_time=0
-o smtpd_soft_error_limit=1001
-o smtpd_hard_error_limit=1000


[root@vps1 ~]# cd /etc
[root@vps1 etc]# rcsdiff -u clamd.conf
===================================================================
RCS file: RCS/clamd.conf,v
retrieving revision 1.1
diff -r1.1 clamd.conf
72c72,73
< LocalSocket /tmp/clamd.socket
---
> #LocalSocket /tmp/clamd.socket
> LocalSocket /var/run/clamav/clamd

[root@vps1 etc]# rcsdiff -u amavisd.conf
===================================================================
RCS file: RCS/amavisd.conf,v
retrieving revision 1.1
diff -u -r1.1 amavisd.conf
--- amavisd.conf 2008/07/13 17:56:22 1.1
+++ amavisd.conf 2008/07/14 02:35:48
@@ -18,7 +18,7 @@
$daemon_user = "amavis"; # (no default; customary: vscan or amavis), -u
$daemon_group = "amavis"; # (no default; customary: vscan or amavis), -g

-$mydomain = 'example.com'; # a convenient default for other settings
+$mydomain = 'localhost'; # a convenient default for other settings

# $MYHOME = '/var/amavis'; # a convenient default for other settings, -H
$TEMPBASE = "$MYHOME/tmp"; # working directory, needs to exist, -T
@@ -46,7 +46,8 @@
$enable_global_cache = 1; # enable use of libdb-based cache if $enable_db=1
$nanny_details_level = 2; # nanny verbosity: 1: traditional, 2: detailed

-@local_domains_maps = ( [".$mydomain"] ); # list of all local domains
+#@local_domains_maps = ( [".$mydomain"] ); # list of all local domains
+read_hash(\%local_domains, '/virtual/etc/vdomains');

@mynetworks = qw( 127.0.0.0/8 [::1] [FE80::]/10 [FEC0::]/10
10.0.0.0/8 172.16.0.0/12 192.168.0.0/16 );
@@ -90,8 +91,8 @@
auth_required_release => 0, # do not require secret_id for amavisd-release
};

-$sa_tag_level_deflt = 2.0; # add spam info headers if at, or above that level
-$sa_tag2_level_deflt = 6.2; # add 'spam detected' headers at that level
+$sa_tag_level_deflt = 0.0; # add spam info headers if at, or above that level
+$sa_tag2_level_deflt = 4.0; # add 'spam detected' headers at that level
$sa_kill_level_deflt = 6.9; # triggers spam evasive actions (e.g. blocks mail)
$sa_dsn_cutoff_level = 10; # spam level beyond which a DSN is not sent
# $sa_quarantine_cutoff_level = 25; # spam level beyond which quarantine is off
@@ -132,7 +133,8 @@
$MIN_EXPANSION_QUOTA = 100*1024; # bytes (default undef, not enforced)
$MAX_EXPANSION_QUOTA = 300*1024*1024; # bytes (default undef, not enforced)

-$sa_spam_subject_tag = '***SPAM*** ';
+#$sa_spam_subject_tag = '***SPAM*** ';
+$sa_spam_subject_tag = '[SPAM] ';
$defang_virus = 1; # MIME-wrap passed infected mail
$defang_banned = 1; # MIME-wrap passed mail containing banned name
# for defanging bad headers only turn on certain minor contents categories:
@@ -143,11 +145,16 @@

# OTHER MORE COMMON SETTINGS (defaults may suffice):

-# $myhostname = 'host.example.com'; # must be a fully-qualified domain name!
+$myhostname = 'vps1.tonns.org'; # must be a fully-qualified domain name!

# $notify_method = 'smtp:[127.0.0.1]:10025';
# $forward_method = 'smtp:[127.0.0.1]:10025'; # set to undef with milter!

+$final_virus_destiny = D_REJECT;
+$final_banned_destiny = D_REJECT;
+$final_spam_destiny = D_PASS;
+$final_bad_header_destiny = D_PASS;
+
# $final_virus_destiny = D_DISCARD;
# $final_banned_destiny = D_BOUNCE;
# $final_spam_destiny = D_BOUNCE;


#
# NOTE: I also uncommented the clamav checks and commented out all the other
# AV checks, but that diff is too large to bother with here
#

# after following the OpenProtect update docs:
[root@vps1 ~]# cd /usr/share/spamassassin/
[root@vps1 spamassassin]# diff sa-update.cron.orig sa-update.cron
5c5
< /usr/bin/sa-update && /etc/init.d/spamassassin condrestart > /dev/null
---
> /usr/bin/sa-update --gpgkey D1C035168C1EBC08464946DA258CDB3ABDE9DC10 --channel saupdates.openprotect.com --channel updates.spamassassin.org && /etc/init.d/amavisd condrestart > /dev/null

#
# setup razor & pyzor
#
su -s/bin/bash amavis
razor-admin -create
razor-admin -register
pyzor discover

[root@vps1 ~]# cd /etc/mail/spamassassin/
[root@vps1 spamassassin]# diff local.cf.orig local.cf
9a10,22
>
> #pyzor
> use_pyzor 1
> pyzor_path /usr/bin/pyzor
>
> #razor
> use_razor2 1
> razor_config /var/amavis/.razor/razor-agent.conf
>
> #bayes
> use_bayes 1
> use_bayes_rules 1
> bayes_auto_learn 1

step 1: greylisting

Short story: I fiddled with gps for a while since it seems like it would perform better than postgrey. I've thrown in the towel for now. gps has the nice feature of whitelisting on sender, but it just seems like it has too much "other" baggage.

postgrey install:

yum install postgrey
chkconfig postgrey on

# add to /etc/sysconfig/postgrey
# OPTIONS="--unix=$SOCKET --delay=120 --auto-whitelist-clients=8 --greylist-text='Service temporarily unavailable. Please rety in %s seconds.' "

# add to /etc/postfix/main.cf:
# smtpd_recipient_restrictions =
# permit_mynetworks
# reject_unauth_destination
# check_policy_service unix:postgrey/socket

service postgrey start
service postfix restart

Long story: OMGWTFBBQ@$%^@#$!!!! You'd think using a nice database abstraction layer like libdbi would make gps a snap. But nooooo RedHat has to be a total pain in my ass. The include libdbi-dbd RPMs for MySQL and PostgreSQL but not for SQLite. And the one thing I don't want to run on my slicehost is an memory-hogging database server, so SQLite is really what I want. So after contemplating it, I just rolled my own spec file and that did it... mostly. gps and it's accompanying perl script gps-maintain.pl have different opinions on what 'dbtype' should be and what the accompanying db_dbtype_dbdir should be, but a post on the forums allowed me to hack it up so it was working. In the end, I spent a lot of time on it and if postgrey sucks the life out of my VM, I might reconsider gps. But for now, I'm tired of installing complex software.

dive! dive! dive!

Looks like my server located at Dorsai is experiencing extended downtime, reason unknown. I've cutover the key websites, but not all of them and not mail yet. I need to really get the spam filtering, etc. going first. Time to hustle that setup, on the quick. More to come shortly...

Getting back on track

After a long, LONG time, I'm finally back on the road to updating my mail server setup. I eluded to setting up a new system a while back, and I'm starting to make progress on it. A tip of my hat to "Joe" at adminspotting.net for his messaging server setup. Some bugs arose with CentOS 5 on x86_64 and dovecot, but other than that, so far so good. The nuts-n-bolts of SMTP/POP/IMAP are setup and next is the anti-virus, anti-spam, greylisting, etc. etc. gauntlet.

An Ode to memcached

When your servers are hurting
And your database needs some relief
Something small, simple and efficient
Can make your response times brief

Sometimes, you don't need anything fancy
You just need something good
My friend is lightweight and speedy
And he does exactly what he should

Keys and values
Hashes and slabs
K.I.S.S.
And viola! it all just works

So go forth, apply caching
And let your servers rest.
Because any sysadmin will tell you,
memcached is the best.

bad timing

Previously, I had my fileserver sync time with my primary wireless router and some pool.ntp.org servers. Then I noticed this:

Jul  7 08:41:10 remote  /usr/sbin/ntpd[521]: skew change 255.181 exceeds limit
Jul  7 08:56:48 remote  /usr/sbin/ntpd[521]: skew change -34.707 exceeds limit
Jul  7 09:24:28 remote  /usr/sbin/ntpd[521]: adjusting local clock by -0.159868s
Jul  7 09:24:28 remote  /usr/sbin/ntpd[521]: skew change -32.121 exceeds limit
Jul  7 09:39:06 remote  /usr/sbin/ntpd[521]: adjusting local clock by 0.465410s
Jul  7 09:39:06 remote  /usr/sbin/ntpd[521]: skew change 132.634 exceeds limit
Jul  7 09:44:23 remote  /usr/sbin/ntpd[521]: adjusting local clock by 0.727679s

According to the documentation, there's no real-time clock on most OpenWrt compatible hardware. I guess when you consider the minimalistic setup of these devices, you can't expect everything.

I've now switched it around. My routers now sync with my fileserver and us.pool.ntp.org. My fileserver ignores the access points and syncs with:

server timex.cs.columbia.edu
server ntp-2.cso.uiuc.edu
server ntppub.tamu.edu
server ntp-1.vt.edu
server ntp3.cs.wisc.edu
server 0.pool.ntp.org
server 1.north-america.pool.ntp.org
server 2.us.pool.ntp.org

Those are the same NTP servers I use at work, with 3 pool.ntp.org servers thrown in. It just annoys me that OpenNTPD doesn't have the equivalent of "ntpq -p" to check the status of the NTP sync.

Remote backups with rdiff-backup

I set this up a while back, but I needed to make some notes on my setup of rdiff-backup that backs up my colo'd PC to my home fileserver. I've been using it for more than 3 months and it has done a great job - space efficient, time efficient and bandwidth light. I followed the recommended unattended HOWTO and it was easy to setup - even with different OSes (Solaris x86 vs. CentOS), versions of ssh and versions of rdiff-backup itself. Only modifications were:

• In the authorized_keys2 on the source system, I didn't include the from="kitty" bit as the destination PC that I'm pulling from has a dynamic IP
  
• I included --print-statistics in the cron job's command line, redirected the output to a logfile
  
• I setup the three config files/scripts setup to have LogWatch tell me the daily summary of the rdiff-backup results. i.e.:
  • /etc/log.d/conf/logfiles/nyc-backup.conf
    
  • /etc/log.d/conf/services/nyc-backup.conf
    
  • /etc/log.d/scripts/services/nyc-backup

My next step is to get rdiff-backup setup to backup by local fileserver. Right now I have a simple rsync from one drive to the other. The reason I haven't done it yet is that the filesystems were setup without LVM (what was I thinking!). So, I need to clean up, reformat with LVM and then get rdiff-backup setup - maybe even with the snapshot recommendations on the unattended HOWTO page.

xubuntu issues, round 1

the notebook was not powering off after shutdown. in some message board threads, "acpi=off" or "acpi=force" as grub args were recommended. what worked was adding the line "apm power_off=1" in /etc/modules.

what's next is to find out what's wrong with the pcmcia xircom network card. i still have to eject & insert the card in order to make it pick it up.

giving xubuntu a try

I'm nuking my crappy WindowsXP install on ye olde notebook - a Toshiba Satellite 4090XDVD. Since I fixed the VPN at work with OpenVPN, I don't need it to connect to the stupid Firebox PPTP (pronounced "PoPToP") VPN anymore. So I'm giving Xubuntu a try because it's supposed to be a lighter desktop.

One gotcha that I encountered on the install is that it didn't initially like my Xircom 10/100+56k modem network card. But I found a tip buried deep in a message board - eject the card, re-insert in and then tell the installer to redetect the network, and it works fine.

Another gotcha was the software dependencies. I am not exactly sure what went wrong, but it failed to install all the dependencies correctly on the first pass. I just told it to re-install again and it was fine. (shrug)

Minor Correction

I said earlier that freshclam isn't an init script and I'd have to craft my own. Not required, as there's a script in cron.daily that updates it once a day.

Tweaking the anti-spam setup

1. Had to tweak the SA ruleset for LogWatch, specifically:
   

   header TONNS_LOGWATCH  Subject =~ /^LogWatch for /
   score  TONNS_LOGWATCH  -20.0

   Eventually, it should be a real whitelist entry, but since this is a dry-run, I'll just leave it at this.
   
   
2. The LogWatch amavisd script is out-of-sync with the logs that amavisd generates. This fixed things:
   

   # pwd
   /etc/log.d/scripts
   # diff -u services.orig/amavis services/amavis 
   --- services.orig/amavis        2005-08-22 21:16:47.000000000 -0400
   +++ services/amavis     2006-11-10 11:01:50.000000000 -0500
   @@ -60,7 +60,7 @@
            or ($ThisLine =~ /^cached [a-zA-Z0-9]+ /)
            or ($ThisLine =~ /^starting.  amavisd at/) ) {
           # We don't care about these
   -   } elsif ($ThisLine =~ /^Passed, /) {
   +   } elsif ($ThisLine =~ /^Passed CLEAN, /) {
          $CleanMsgs++;
       } elsif (($FileName, $From) = ( $ThisLine =~ /^BANNED name\/type \(([^\)]+)\)\, \<([^\>]*)\>/ )) {
    
   @@ -82,7 +82,18 @@
            $Viruses{$Virus}{$From}++;
          }; # if
    
   -   } elsif (($Fromspam, $Towards) = ( $ThisLine =~ /^SPAM, [\(\<]([^\>\)]+)[\)\>] -\> [\(\<]([^\>\)]+)[\)\>]/ )) {
   +   } elsif (($Fromspam, $Towards) = ( $ThisLine =~ /^Passed SPAM, \[[\d\.]+\] \[[\d\.]+\] [\(\<]([^\>\)]+)[\)\>] -\> [\(\<]([^\>\)]+)[\)\>]/ )) {
   +      $SpamMsgs++;
   +
   +      if ($Detail >= 5) {
   +        $Spamtypes{$Towards}++;
   +      }; # if
   +
   +      if ($Detail >= 10) {
   +        $Spams{$Towards}{$Fromspam}++;
   +      }; # if
   +
   +   } elsif (($Fromspam, $Towards) = ( $ThisLine =~ /^Passed SPAM, [\(\<]([^\>\)]+)[\)\>] -\> [\(\<]([^\>\)]+)[\)\>]/ )) {
          $SpamMsgs++;
    
          if ($Detail >= 5) {

   
   

Anti-spam setup

Man, setting up an anti-spam toolsuite is a PITA. Spammers suck.

Anyway, I started off with the this HOWOTO from howtoforge.com, skipping all the SQL stuff because I'm not using a virtual setup (for now - maybe the final setup will, I'm not sure yet). Some key
differences in my setup:

In the install:

• The HOWOTO is for Fedora, not CentOS. You need to add "clamd" and "razor-agents" to the install.
  
• There's no "pyzor" package for CentOS. Need to install from source.
  
• I didn't setup razor/pyzor exactly like the HOWTO. I can't recall all the details, but I just su'd to amavis and did it as a "standalone user". But since amavisd does all the SA processing as amavis, it's effectively global. I think I just followed the default documentation.
  
• There's no "freshclam" init script. I still have to write something that crons (or borrow the script from Fedora). I just ran it by hand for now.
  
• Installed "fetchmail". I'm majorly impressed by Horde on this one. IMP was working great and as soon as I installed fetchmail, all the options for remote mail retrival showed up
  

In amavisd.conf:

• $sa_tag_level_deflt = undef; - always print the SA headers
  
• @local_domains_maps = ( [".$mydomain", 'example.com', 'foobar.tld'] ); - if all your domains aren't here, the SA headers won't be added to the messages
  
• $sa_spam_subject_tag = '[SPAM] '; - damn, i hate the asterisks in the subject
  
• Uncomment the 'ClamAV-clamd' section
  
• Make sure you add amavis to the clamd group
  

In clamd.conf

• Add "LocalSocket /var/run/clamav/clamd"
  
• Comment out the TCPSocket and TCPAddr sections
  
• Make sure you add clamd to the amavis group, just to be sure.
  

In Postfix

• Got rid of the home_mailbox directive for Maildirs from yesterday's setup
  
• added "mailbox_command = /usr/bin/procmail"
  
• created /etc/procmailrc with:

  DEFAULT=$HOME/Maildir/
  LOGFILE=/var/log/procmail.log
  LOBABSTRACT=all

  The logging stuff is just temporary.
  
• The reason for the switch to procmail was the Ingo module for horde. It's pretty solid. It will update a user's .procmailrc via the FTP VFS backend, all using the single-signon password. My backend looks like this:

  $backends['procmail'] = array(
      'driver' => 'vfs',
      'preferred' => 'home.tonns.net',
      'hordeauth' => true,
      'params' => array(
          'hostspec' => 'localhost',
          'filename' => '.procmailrc',
          'port' => 21,
          'vfstype' => 'ftp'
      ),
      'script' => 'procmail',
      'scriptparams' => array(
          'path_style' => 'maildir',
          'variables' => array(
              'DEFAULT' => '$HOME/Maildir/',
          )
      )
  );

  That last variable is redundant with the /etc/procmailrc settings, but I don't want the users to try anything crazy.
  
• added to my .procmailrc using Ingo:

  ##### SPAM #####
  :0 
  * ^X-Spam-Status:.*Yes
  "$DEFAULT/.SPAM/"

  Which does as you'd expect. I also added it to /etc/skel/.procmailrc, but horde doesn't read in existing procmail rules, it just knows about the ones it created. I have to look into it - maybe I'll put it in the global procmailrc.
  
• mkdir -p /etc/skel/Maildir/SPAM; mkdir -p /etc/skel/Maildir/.LearnAsSpam; mkdir -p /etc/skel/Maildir/.LearnAsNotSpam - if the Maildir doesn't exist IMP default to mbox in $HOME/mail - which is not what we want, so have them created by default.
  
• Finally, I wrote a shell script that uses archivemail and sa-learn to clean out everyone's LearnAsSpam/NotSpam directories and add them to the global SA bayes filter (which is in the user amavis's home directory)

Well, I think that's about it. I might have tweaked something else... but I'll have to pick it up later.

Finally, I'm gonna have to do a major hardware upgrade - it doesn't look like 256MB is enough to run the OS and apache and php and amavisd and clamd and mysql and vsftpd and postfix - doing absolutely nothing it's swapped out 150MB. Needing more RAM and a new harddrive to replace the failed one... it's getting to the point where it's not worth using this 1Ghz P3 Coppermine system.

Installing horde...

After using the horde file manager for a project at work, I decided to give it a test drive at home. So far, so good. As this is just a dry run for the migration of tonns.org to home (as I don't have the right setup due to hardware failures). I was feeling kind-of lazy about it, so I followed this HOWTO to get postfix/SMTP-AUTH/TLS/dovecot going.

Key points on the install:

• It's all about rpmforge. Dag Wieers really takes the headache out of installing all this with the rpmforge repository. Remember to send him an email thanking him - I did.
  
  
• The list of packages I had to "yum install" is as follows:

  apr apr-util autoconf automake curl curl-devel cyrus-sasl-devel cyrus-sasl-gssapi dovecot e2fsprogs-devel gd httpd httpd-suexec ImageMagick krb5-devel libc-client libidn libidn-devel libtool libtool-libs libxml2-devel mysql mysql-server openssl-devel pam-devel perl-DBD-MySQL perl-DBI perl-HTML-Parser perl-HTML-Tagset perl-libwww-perl perl-URI php php-devel php-domxml php-gd php-imap php-ldap php-mysql php-odbc php-pear php-pear-log php-pear-mail_mime php-pecl-fileinfo php-pecl-memcache php-xmlrpc pkgconfig postgresql-libs rpm-build unixODBC vsftpd zlib-devel

  
  
• PHP and PAM don't play nice together. The pam_auth module for php exists, but damn I couldn't get it to compile as a module and fuck-no, I'm not recompiling PHP.
  
  
• Instead, setup IMP and MIMP before other modules, and use thier auth (i.e. imap auth) as the horde auth using this setup: http://wiki.horde.org/MIMPHowTo adding $conf['auth']['driver'] = 'composite'; as well.
  
  
• Getting the latest PEAR modules to install was also a PITA. IMP requires HTTP_Request and Auth_SASL modules, but to get them installed I had to lock-step upgrade modules to interim versions before everything would update. Specifically:
  

  pear upgrade Archive_Tar
  pear upgrade PEAR-1.3.3
  pear upgrade PEAR
  pear upgrade XML_RPC-1.4.0
  pear upgrade-all
  pear install HTTP_Request
  pear install Auth_SASL

  Which finally left me with:
  

  Installed packages, channel pear.php.net:
  =========================================
  Package        Version State
  Archive_Tar    1.3.1   stable
  Auth_SASL      1.0.2   stable
  Console_Getopt 1.2     stable
  DB             1.7.6   stable
  HTTP           1.4.0   stable
  HTTP_Request   1.4.0   stable
  Log            1.9.9   stable
  Mail           1.1.14  stable
  Mail_Mime      1.3.1   stable
  Net_SMTP       1.2.8   stable
  Net_Socket     1.0.6   stable
  Net_URL        1.0.14  stable
  PEAR           1.4.11  stable
  XML_Parser     1.2.7   stable
  XML_RPC        1.5.1   stable

Other than that, it was mostly following the horde INSTALL files. It's pretty simple.

Today, I'm gonna try to take a look at SpamAssassin, ClamAV, amavisd-new, CRM114, etc. etc. blah. blah.

got grub?

found this little gem in my inbox this morning:

This is an automatically generated mail message from mdadm
running on obiwan

A Fail event had been detected on md device /dev/md0.

Faithfully yours, etc.

oh yay! failed drives! luckily obiwan is still the "sandbox system" for now - it was supposed to be turned into my main externally-facing server once i was done with openwrt/dmz setup/etc. so much for good intentions - i'll never get this shit done.

so, i at least had the forethought to mirror the drives - it's dual 60GB ATA100 drives - good ol' hda and hdb. on each drive, i created two partitions - the first partition is /boot and the other is half of md0 - a raid1 device. i then built on md0 some logical volumes with LVM2, i usually name them /dev/linux/root, /dev/centos/usr, /dev/obiwan/home, or something like that. as far as the other partition, i thought i was doing the right thing by performing:

rsync -av --delete /boot /boot2

... to sync the kernel/initrd after a yum update included a kernel update, but that's only 1/2 of it. in today's failed case, it was hda that failed, which brings us to the crux of the problem - where's your bootloader now, eh? basically, nowhere. i'm screwed. so, i broke out the knoppix dvd and get to installing a bootloader on the second drive so i could bring the system up. how could i have prevented this from happening?

well, i think i have it worked out:

1. edit /boot/grub/device.map. make sure there's an entry for the second device there. in my case, it would be:

   (hd1) /dev/hdb

2. since grub-install likes to install in /boot of the grub root (very different from the system root - "/"), i gave it a little symlink hack:

   cd /boot; ln -s . boot

3. clean-up! get rid of all those old kernels that were installed with yum update:

   rpm -e kernel-old-version-blah

4. re-sync everything:

   rsync -av --delete /boot /boot2

5. now install grub on the second drive:

   grub-install --root-directory=/boot2 /dev/hdb

i think that should do it. i'm going to see if there's a way i can test this - maybe i'll pull some of the really 2GB drives out of the closet and get them in the test system to simulate failure.

Update: so much for that... i just got:

This is an automatically generated mail message from mdadm
running on obiwan

A DegradedArray event had been detected on md device /dev/md0.

Faithfully yours, etc.

ding-dong, the system's dead. if i'm gonna be using knoppix so much, maybe i should re-download the latest dvd. sigh

renewing certs

note to self: don't misplace the post-it with the passphrases for your CA. i ripped my whole apartment apart looking for it this morning. i need to update my openssl docs on how to renew a cert. back in 2001, i had no idea how to renew a cert. it's really as simple as just re-generating it with the same csr, and letting the serial number be incremented. however, without your CA passphrase, you'd be screwed. luckily, i found it and so i'm back in business. hopefully, the rest of the family using the site didn't notice.