step 2: anti-spam, anti-virus

ow. my head.

Not only is the spam problem on the internet horrible, but so is the how-to-implement-spam-prevention problem. There's sooo many walkthroughs, guides, howtos and different packages for different UNIX flavors that to attempt to accomplish the task. Here's the list of tools I'm starting off with:

amavis-new
spamassassin
clamav
razor-agents
pyzor

I started with the adminspotting walkthrough but that's debian based and my CentOS box needs additional configuration. I read over the SA wiki, but still didn't fit right. I think the closest is the HowToForge howto, but my virtual setup is different (file based vs. mysql based). I also added OpenProtect's sa-update channel and I built my own pyzor rpm using the fedora spec file. Below are some key config steps. I might have missed one or two, but I think I got "the big ones". Of course, there are more components that I could add (dcc, DomainKeys, spf, etc. etc.) but my VM is already wheezing on memory and thats with only 2 amavisd children and zero mail traffic.

Man, what a pain in the ass.

---8<---
# install rpmforge pkgs
yum install spamassassin
yum install clamav-db clamav clamd
yum install amavisd-new yum install razor-agents
rpm -ihv /www/src/rpms/pyzor-0.4.0-11.noarch.rpm

#
# for /etc/postfix/main.cf:
#

#
# amavis
#
content_filter=smtp-amavis:[127.0.0.1]:10024
receive_override_options = no_address_mappings

#
# for /etc/postfix/master.cf:
#

#
# amavis setup
#
smtp-amavis unix - - n - 2 smtp
-o smtp_data_done_timeout=1200
-o smtp_send_xforward_command=yes

127.0.0.1:10025 inet n - n - - smtpd
-o content_filter=
-o local_recipient_maps=
-o relay_recipient_maps=
-o smtpd_restriction_classes=
-o smtpd_client_restrictions=
-o smtpd_helo_restrictions=
-o smtpd_sender_restrictions=
-o smtpd_recipient_restrictions=permit_mynetworks,reject
-o mynetworks=127.0.0.0/8
-o strict_rfc821_envelopes=yes
-o smtpd_error_sleep_time=0
-o smtpd_soft_error_limit=1001
-o smtpd_hard_error_limit=1000


[root@vps1 ~]# cd /etc
[root@vps1 etc]# rcsdiff -u clamd.conf
===================================================================
RCS file: RCS/clamd.conf,v
retrieving revision 1.1
diff -r1.1 clamd.conf
72c72,73
< LocalSocket /tmp/clamd.socket
---
> #LocalSocket /tmp/clamd.socket
> LocalSocket /var/run/clamav/clamd

[root@vps1 etc]# rcsdiff -u amavisd.conf
===================================================================
RCS file: RCS/amavisd.conf,v
retrieving revision 1.1
diff -u -r1.1 amavisd.conf
--- amavisd.conf 2008/07/13 17:56:22 1.1
+++ amavisd.conf 2008/07/14 02:35:48
@@ -18,7 +18,7 @@
$daemon_user = "amavis"; # (no default; customary: vscan or amavis), -u
$daemon_group = "amavis"; # (no default; customary: vscan or amavis), -g

-$mydomain = 'example.com'; # a convenient default for other settings
+$mydomain = 'localhost'; # a convenient default for other settings

# $MYHOME = '/var/amavis'; # a convenient default for other settings, -H
$TEMPBASE = "$MYHOME/tmp"; # working directory, needs to exist, -T
@@ -46,7 +46,8 @@
$enable_global_cache = 1; # enable use of libdb-based cache if $enable_db=1
$nanny_details_level = 2; # nanny verbosity: 1: traditional, 2: detailed

-@local_domains_maps = ( [".$mydomain"] ); # list of all local domains
+#@local_domains_maps = ( [".$mydomain"] ); # list of all local domains
+read_hash(\%local_domains, '/virtual/etc/vdomains');

@mynetworks = qw( 127.0.0.0/8 [::1] [FE80::]/10 [FEC0::]/10
10.0.0.0/8 172.16.0.0/12 192.168.0.0/16 );
@@ -90,8 +91,8 @@
auth_required_release => 0, # do not require secret_id for amavisd-release
};

-$sa_tag_level_deflt = 2.0; # add spam info headers if at, or above that level
-$sa_tag2_level_deflt = 6.2; # add 'spam detected' headers at that level
+$sa_tag_level_deflt = 0.0; # add spam info headers if at, or above that level
+$sa_tag2_level_deflt = 4.0; # add 'spam detected' headers at that level
$sa_kill_level_deflt = 6.9; # triggers spam evasive actions (e.g. blocks mail)
$sa_dsn_cutoff_level = 10; # spam level beyond which a DSN is not sent
# $sa_quarantine_cutoff_level = 25; # spam level beyond which quarantine is off
@@ -132,7 +133,8 @@
$MIN_EXPANSION_QUOTA = 100*1024; # bytes (default undef, not enforced)
$MAX_EXPANSION_QUOTA = 300*1024*1024; # bytes (default undef, not enforced)

-$sa_spam_subject_tag = '***SPAM*** ';
+#$sa_spam_subject_tag = '***SPAM*** ';
+$sa_spam_subject_tag = '[SPAM] ';
$defang_virus = 1; # MIME-wrap passed infected mail
$defang_banned = 1; # MIME-wrap passed mail containing banned name
# for defanging bad headers only turn on certain minor contents categories:
@@ -143,11 +145,16 @@

# OTHER MORE COMMON SETTINGS (defaults may suffice):

-# $myhostname = 'host.example.com'; # must be a fully-qualified domain name!
+$myhostname = 'vps1.tonns.org'; # must be a fully-qualified domain name!

# $notify_method = 'smtp:[127.0.0.1]:10025';
# $forward_method = 'smtp:[127.0.0.1]:10025'; # set to undef with milter!

+$final_virus_destiny = D_REJECT;
+$final_banned_destiny = D_REJECT;
+$final_spam_destiny = D_PASS;
+$final_bad_header_destiny = D_PASS;
+
# $final_virus_destiny = D_DISCARD;
# $final_banned_destiny = D_BOUNCE;
# $final_spam_destiny = D_BOUNCE;


#
# NOTE: I also uncommented the clamav checks and commented out all the other
# AV checks, but that diff is too large to bother with here
#

# after following the OpenProtect update docs:
[root@vps1 ~]# cd /usr/share/spamassassin/
[root@vps1 spamassassin]# diff sa-update.cron.orig sa-update.cron
5c5
< /usr/bin/sa-update && /etc/init.d/spamassassin condrestart > /dev/null
---
> /usr/bin/sa-update --gpgkey D1C035168C1EBC08464946DA258CDB3ABDE9DC10 --channel saupdates.openprotect.com --channel updates.spamassassin.org && /etc/init.d/amavisd condrestart > /dev/null

#
# setup razor & pyzor
#
su -s/bin/bash amavis
razor-admin -create
razor-admin -register
pyzor discover

[root@vps1 ~]# cd /etc/mail/spamassassin/
[root@vps1 spamassassin]# diff local.cf.orig local.cf
9a10,22
>
> #pyzor
> use_pyzor 1
> pyzor_path /usr/bin/pyzor
>
> #razor
> use_razor2 1
> razor_config /var/amavis/.razor/razor-agent.conf
>
> #bayes
> use_bayes 1
> use_bayes_rules 1
> bayes_auto_learn 1

dive! dive! dive!

Looks like my server located at Dorsai is experiencing extended downtime, reason unknown. I've cutover the key websites, but not all of them and not mail yet. I need to really get the spam filtering, etc. going first. Time to hustle that setup, on the quick. More to come shortly...

Minor Correction

I said earlier that freshclam isn't an init script and I'd have to craft my own. Not required, as there's a script in cron.daily that updates it once a day.

Tweaking the anti-spam setup

1. Had to tweak the SA ruleset for LogWatch, specifically:
   

   header TONNS_LOGWATCH  Subject =~ /^LogWatch for /
   score  TONNS_LOGWATCH  -20.0

   Eventually, it should be a real whitelist entry, but since this is a dry-run, I'll just leave it at this.
   
   
2. The LogWatch amavisd script is out-of-sync with the logs that amavisd generates. This fixed things:
   

   # pwd
   /etc/log.d/scripts
   # diff -u services.orig/amavis services/amavis 
   --- services.orig/amavis        2005-08-22 21:16:47.000000000 -0400
   +++ services/amavis     2006-11-10 11:01:50.000000000 -0500
   @@ -60,7 +60,7 @@
            or ($ThisLine =~ /^cached [a-zA-Z0-9]+ /)
            or ($ThisLine =~ /^starting.  amavisd at/) ) {
           # We don't care about these
   -   } elsif ($ThisLine =~ /^Passed, /) {
   +   } elsif ($ThisLine =~ /^Passed CLEAN, /) {
          $CleanMsgs++;
       } elsif (($FileName, $From) = ( $ThisLine =~ /^BANNED name\/type \(([^\)]+)\)\, \<([^\>]*)\>/ )) {
    
   @@ -82,7 +82,18 @@
            $Viruses{$Virus}{$From}++;
          }; # if
    
   -   } elsif (($Fromspam, $Towards) = ( $ThisLine =~ /^SPAM, [\(\<]([^\>\)]+)[\)\>] -\> [\(\<]([^\>\)]+)[\)\>]/ )) {
   +   } elsif (($Fromspam, $Towards) = ( $ThisLine =~ /^Passed SPAM, \[[\d\.]+\] \[[\d\.]+\] [\(\<]([^\>\)]+)[\)\>] -\> [\(\<]([^\>\)]+)[\)\>]/ )) {
   +      $SpamMsgs++;
   +
   +      if ($Detail >= 5) {
   +        $Spamtypes{$Towards}++;
   +      }; # if
   +
   +      if ($Detail >= 10) {
   +        $Spams{$Towards}{$Fromspam}++;
   +      }; # if
   +
   +   } elsif (($Fromspam, $Towards) = ( $ThisLine =~ /^Passed SPAM, [\(\<]([^\>\)]+)[\)\>] -\> [\(\<]([^\>\)]+)[\)\>]/ )) {
          $SpamMsgs++;
    
          if ($Detail >= 5) {

   
   

Anti-spam setup

Man, setting up an anti-spam toolsuite is a PITA. Spammers suck.

Anyway, I started off with the this HOWOTO from howtoforge.com, skipping all the SQL stuff because I'm not using a virtual setup (for now - maybe the final setup will, I'm not sure yet). Some key
differences in my setup:

In the install:

• The HOWOTO is for Fedora, not CentOS. You need to add "clamd" and "razor-agents" to the install.
  
• There's no "pyzor" package for CentOS. Need to install from source.
  
• I didn't setup razor/pyzor exactly like the HOWTO. I can't recall all the details, but I just su'd to amavis and did it as a "standalone user". But since amavisd does all the SA processing as amavis, it's effectively global. I think I just followed the default documentation.
  
• There's no "freshclam" init script. I still have to write something that crons (or borrow the script from Fedora). I just ran it by hand for now.
  
• Installed "fetchmail". I'm majorly impressed by Horde on this one. IMP was working great and as soon as I installed fetchmail, all the options for remote mail retrival showed up
  

In amavisd.conf:

• $sa_tag_level_deflt = undef; - always print the SA headers
  
• @local_domains_maps = ( [".$mydomain", 'example.com', 'foobar.tld'] ); - if all your domains aren't here, the SA headers won't be added to the messages
  
• $sa_spam_subject_tag = '[SPAM] '; - damn, i hate the asterisks in the subject
  
• Uncomment the 'ClamAV-clamd' section
  
• Make sure you add amavis to the clamd group
  

In clamd.conf

• Add "LocalSocket /var/run/clamav/clamd"
  
• Comment out the TCPSocket and TCPAddr sections
  
• Make sure you add clamd to the amavis group, just to be sure.
  

In Postfix

• Got rid of the home_mailbox directive for Maildirs from yesterday's setup
  
• added "mailbox_command = /usr/bin/procmail"
  
• created /etc/procmailrc with:

  DEFAULT=$HOME/Maildir/
  LOGFILE=/var/log/procmail.log
  LOBABSTRACT=all

  The logging stuff is just temporary.
  
• The reason for the switch to procmail was the Ingo module for horde. It's pretty solid. It will update a user's .procmailrc via the FTP VFS backend, all using the single-signon password. My backend looks like this:

  $backends['procmail'] = array(
      'driver' => 'vfs',
      'preferred' => 'home.tonns.net',
      'hordeauth' => true,
      'params' => array(
          'hostspec' => 'localhost',
          'filename' => '.procmailrc',
          'port' => 21,
          'vfstype' => 'ftp'
      ),
      'script' => 'procmail',
      'scriptparams' => array(
          'path_style' => 'maildir',
          'variables' => array(
              'DEFAULT' => '$HOME/Maildir/',
          )
      )
  );

  That last variable is redundant with the /etc/procmailrc settings, but I don't want the users to try anything crazy.
  
• added to my .procmailrc using Ingo:

  ##### SPAM #####
  :0 
  * ^X-Spam-Status:.*Yes
  "$DEFAULT/.SPAM/"

  Which does as you'd expect. I also added it to /etc/skel/.procmailrc, but horde doesn't read in existing procmail rules, it just knows about the ones it created. I have to look into it - maybe I'll put it in the global procmailrc.
  
• mkdir -p /etc/skel/Maildir/SPAM; mkdir -p /etc/skel/Maildir/.LearnAsSpam; mkdir -p /etc/skel/Maildir/.LearnAsNotSpam - if the Maildir doesn't exist IMP default to mbox in $HOME/mail - which is not what we want, so have them created by default.
  
• Finally, I wrote a shell script that uses archivemail and sa-learn to clean out everyone's LearnAsSpam/NotSpam directories and add them to the global SA bayes filter (which is in the user amavis's home directory)

Well, I think that's about it. I might have tweaked something else... but I'll have to pick it up later.

Finally, I'm gonna have to do a major hardware upgrade - it doesn't look like 256MB is enough to run the OS and apache and php and amavisd and clamd and mysql and vsftpd and postfix - doing absolutely nothing it's swapped out 150MB. Needing more RAM and a new harddrive to replace the failed one... it's getting to the point where it's not worth using this 1Ghz P3 Coppermine system.