Complete Virtual Mail Server/amavisd spamassassin clamav
This article is part of the Complete Virtual Mail Server series, and may require previous parts to have been read or followed.
Spam is becoming more and more of an issue on the Internet and a robust and solid solution is required. There are paid services for even more spam protection but that should not be required and will not be discussed in this article.
Postfix
The first line of defense, is postfix itself. Postfix offers a few basic means to block spam, or rather spammers. Using smtpd_client_restrictions it is possible to use public DNS blacklists. There are 3 popular DNS blacklists of which one is incorporated into the other. These two lists are also the most accurate ones. The most important one is zen.spamhaus.org and as a backup bl.spamcop.net can be used. Using them with postfix is very simple. Add these domains as reject_rbl_client in main.cf:
/etc/postfix/main.cfUse DNS blacklists# Block spam using DNS blacklists
smtpd_client_restrictions = permit_mynetworks, permit_sasl_authenticated, reject_rbl_client zen.spamhaus.org, reject_rbl_client bl.spamcop.net
permit_mynetworks and permit_sasl_authenticated are added for good measure. If other restrictions are required or already in use, keep them in place.Amavis
Introduction
Spam-assassin and ClamAV are the tools to block spam and viruses, however amavis is required to tie this all together. Amavis will actually behave as a mail server in itself, accept mail, filter it, and send it onwards again. For this to work, postfix will need to actually listen for mail twice. The default port 25 is where mail initially is received on. From there on it is sent to amavis, which will be listening on port 10024. When amavis is done with the message, it will be sent to postfix on a different port, 10025. The reason for this should be obvious. If mail would be offered again on port 25, it would be passed to amavis again and thus in an endless loop. Obviously, postfix on port 10025 would only be listening to known hosts, like localhost and not check for spam anymore.
Because of this setup, it is quite easily possible, to have these 3 mail 'handlers' on 3 different hosts. It could be possible to have the primary mailserver, that listens on port 25 on the firewall, have it forward to a safe isolated server running only amavis and have amavis sent mail to an internal postfix server where it delivers mail to the mail storage.
Installation
Amavis should have been installed already, if not, emerge it. This should pull in spam-assassin and clamav as its dependencies:
root #emerge --ask mail-filter/amavisd-newBasic Configuration
amavisd offers an enormous amount of options and going over all them will take some time. However, the configuration file (/etc/amavisd.conf) is well documented and divided into clear sections. Each section will be examined as needed. Only options that will be changed will be mentioned to cut down the text for readability.
The configuration file is actually Perl code and proper precautions should be taken when editing this file.
For this example, amavisd will be running on host foo, but this could be any other host as well. amavisd does not need to run on the same host as Postfix. Also, the domain used is only used to identify the server itself, not the domains amavisd will be scanning.
With amavis being quite complex, troubleshooting can be difficult enough as it is.
The first step is to disable all actual checks and to enable logging. Also, some default values should be setup:
/etc/amavisd.confDisable anti-spam, enable logging@bypass_virus_checks_maps = (1); # controls running of anti-virus code
@bypass_spam_checks_maps = (1); # controls running of anti-spam code
# $bypass_decode_parts = 1; # controls running of decoders&dearchivers
$mydomain = 'example.com';
$myhostname = 'foo.example.com';
$MYHOME = '/var/lib/amavishome'; # a convenient default for other settings, -H
$log_level = 5; # verbosity 0..5, -d
$MYHOME is used as the prefix for several locations amavisd needs to work within, so setting it to /var/lib/amavishome should be safe. That path should have been created as the amavis user's home directory by acct-user/amavis, which is pulled in by mail-filter/amavisd-new. The location does not matter as long as the amavis has write permissions to it. Otherwise, the daemon will crash on start.
virusalert@example.com and spam.police@example can be renamed to undef if not wanting to receive any information on breaches. If not, these e-mail addresses should exist.
Normally, restarting Postfix should restart amavisd as well. For now, only amavisd should be started to see if there are any initial problems:
root #/etc/init.d/amavisd startLinking amavisd to Postfix
With amavisd working in bare skeletal mode, it should theoretically just pass mail through. Perfect for testing the postfix -> amavisd -> postfix binding.
First, a second postfix transport, where amavis will inject its mail, is added. A lot of options are defaulted to empty, since either they have been checked already, or interfere otherwise:
/etc/postfix/master.cfFilter mail through amavisdlocalhost:10025 inet n - n - 2 smtpd
-o smtp_dns_support_level=enabled
-o content_filter=
-o myhostname=foo.example.com
-o local_recipient_maps=
-o relay_recipient_maps=
-o smtpd_restriction_classes=
-o smtpd_client_restrictions=
-o smtpd_helo_restrictions=
-o smtpd_sender_restrictions=
-o smtpd_recipient_restrictions=permit_mynetworks,reject
-o mynetworks=127.0.0.0/8
-o strict_rfc821_envelopes=yes
-o smtpd_error_sleep_time=0
-o smtpd_soft_error_limit=1001
-o smtpd_hard_error_limit=1000
-o smtpd_client_connection_count_limit=0
-o smtpd_client_connection_rate_limit=0
-o receive_override_options=no_unknown_recipient_checks,no_header_body_checks
-o smtpd_authorized_xforward_hosts=127.0.0.0/8
With this transport in place, it should only listen on localhost and only accept mail from localhost. This should be extended if amavis is run elsewhere, but keep in mind anything is accepted.
maxproc' is set to 2 above. This can be increased if more listening daemons are required. However this number needs to match amavis's
$max_servers.Next another transport is added for, which could be considers 'being' amavis in a sense:
/etc/postfix/master.cfamavis transportamavis unix - - n - 2 lmtp
-o disable_dns_lookups=yes
-o lmtp_send_xforward_command=yes
-o lmtp_data_done_timeout=1200
After the transport for amavis has been added, smtp should be told to route all mail through amavis. For this two option need to be added to smtpd and change the maxproc to match amavis's:
/etc/postfix/master.cfrelay to amavissmtp inet n - n - 2 smtpd
-o content_filter=amavis:[127.0.0.1]:10024
-o receive_override_options=no_address_mappings
smtps inet n - n - 2 smtpd
-o smtpd_tls_wrappermode=yes
-o content_filter=amavis:[127.0.0.1]:10024
-o receive_override_options=no_address_mappings
Restarting both amavisd and postfix then should pass all mail through amavisd:
root #/etc/init.d/amavisd restartroot #/etc/init.d/postfix restartTesting
Sending a message to testuser@example.com remotely and locally should work fine. After the message has arrived, the headers should be checked.
When doing this test, either use webmail or a remote client using a different mailserver or mail through port 25 or 465. Do not test using mail submission on port 587 as that bypasses amavisd.
Looking at the mail headers, it should be noticed that it was sent through amavisd and re-delivered to postfix:
Received: from localhost (localhost [127.0.0.1])
by foo.example.com (Postfix) with ESMTP id 03ABA22E4C
for <testuser@example.com>; Wed, 28 Dec 2011 11:41:05 +0100 (CET)
Received: from foo.example.com ([127.0.0.1])
by localhost (foo.example.com [127.0.0.1]) (amavisd-new, port 10024)
with LMTP id 6N9l4nIQa620 for <testuser@example.com>; Wed, 28 Dec 2011 11:41:04 +0100 (CET)
Received: from ext.example.net (ext.example.net [1.2.3.4])
(using TLSv1 with cipher ADH-AES256-SHA (256/256 bits))
(No client certificate requested)
by foo.example.com (Postfix) with ESMTPS id C136021F97
for <testuser@example.com>; Wed, 28 Dec 2011 11:41:04 +0100 (CET)
Examining the above it is clearly visible that the mail was received by postfix via SMTPS even. It was then forwarded to amavisd on port 10024 via LMTP and finally redelivered to postfix using SMTP again.
ClamAV
Introduction
ClamAV is the de facto open source virus scanner for linux. Amavis can be linked to many different free and commercial virus scanners, but here clamav will be used. ClamAV is specifically designed for scanning e-mail. It consists of two parts, clamav itself, and freshclam, the clamav updating service. By default it updates every two hours, which should be enough for anyone.
Installation
ClamAV should have been installed already, if not it should be emerged:
root #emerge --ask app-antivirus/clamavThe
clamdtop flag lets clamav build the clamdtop utility, a nice monitoring tool for the viruscanner.Configuration
ClamAV will be configured to run in daemonized mode, e.g. it will be listening for connections (from amavisd). The other option (and the default fallback in amavisd) is to have amavisd use the commandline scanner, which is much slower and much much more resource intensive.
ClamAV does not have to be run on the same host, however it is recommended for performance reasons to keep it on the same host, depending on resource usage.
To be able to communicate, clamav needs to be part of amavisd's group:
root #gpasswd -a clamav amavisIt always helps to allow clamd to output some debug information:
/etc/clamav/clamd.confEnable Debug options# Enable debug messages in libclamav.
# Default: no
Debug yes
Also clamd needs some settings setup in its configuration file so that amavis can talk to it:
/etc/clamav/clamd.confAllow amavis to conenct to clamd# Path to a local socket file the daemon will listen on.
# Default: disabled (must be specified by a user)
LocalSocket /var/run/clamav/clamd.sock
# Sets the group ownership on the unix socket.
# Default: disabled (the primary group of the user running clamd)
LocalSocketGroup amavis
These options appear to be the default nowadays.
When running clamav on a hardened kernel, there will be warnings about certain operations not being permitted:
[LibClamAV] Bytecode: disabling JIT because SELinux is preventing 'execmem' access.
This is expected and okay. ClamAV can run fine without JIT.
Before starting clamav for the first time, the virus database needs to be downloaded. Freshclam is responsible for downloading and keeping the virus database up to date. Freshclam gets automatically started by the clamd startup script, but clamd will fail to start due to a missing database:
root #/etc/init.d/clamd startNow monitor the clamav log file to see freshclam download the initial virus database:
/var/log/clamav/freshclam.logFreshclam startup logfreshclam daemon 0.98.6 (OS: linux-gnu, ARCH: x86_64, CPU: x86_64)
ClamAV update process started at Sat Mar 21 16:30:16 2015
Downloading main.cvd [100%]
main.cvd updated (version: 55, sigs: 2424225, f-level: 60, builder: neo)
Downloading daily.cvd [100%]
daily.cvd updated (version: 20219, sigs: 1354642, f-level: 63, builder: neo)
Downloading bytecode.cvd [100%]
[LibClamAV] Bytecode: disabling JIT because SELinux is preventing 'execmem' access.
Run 'setsebool -P clamd_use_jit on'.
ERROR: During database load : LibClamAV Warning: RWX mapping denied: Can't allocate RWX Memory: Operation not permitted
WARNING: Database successfully loaded, but there is stderr output
bytecode.cvd updated (version: 247, sigs: 41, f-level: 63, builder: dgoddard)
Database updated (3778908 signatures) from database.clamav.net (IP: 200.236.31.1)
WARNING: Clamd was NOT notified: Can't connect to clamd through /var/run/clamav/clamd.sock: No such file or directory
Now that the database has been updated, restart clamd:
root #/etc/init.d/clamd restart/var/log/clamav/clamd.logClamAV startup logSat Mar 21 16:35:18 2015 -> clamd daemon 0.98.6 (OS: linux-gnu, ARCH: x86_64, CPU: x86_64)
Sat Mar 21 16:35:18 2015 -> Running as user clamav (UID 105, GID 206)
Sat Mar 21 16:35:18 2015 -> Log file size limited to 1048576 bytes.
Sat Mar 21 16:35:18 2015 -> Reading databases from /var/lib/clamav
Sat Mar 21 16:35:18 2015 -> Not loading PUA signatures.
Sat Mar 21 16:35:18 2015 -> Bytecode: Security mode set to "TrustSigned".
Sat Mar 21 16:35:29 2015 -> Loaded 3773304 signatures.
Sat Mar 21 16:35:30 2015 -> LOCAL: Unix socket file /var/run/clamav/clamd.sock
Sat Mar 21 16:35:30 2015 -> LOCAL: Setting connection queue length to 200
Sat Mar 21 16:35:30 2015 -> Limits: Global size limit set to 104857600 bytes.
Sat Mar 21 16:35:30 2015 -> Limits: File size limit set to 26214400 bytes.
Sat Mar 21 16:35:30 2015 -> Limits: Recursion level limit set to 16.
Sat Mar 21 16:35:30 2015 -> Limits: Files limit set to 10000.
Sat Mar 21 16:35:30 2015 -> Limits: MaxEmbeddedPE limit set to 10485760 bytes.
Sat Mar 21 16:35:30 2015 -> Limits: MaxHTMLNormalize limit set to 10485760 bytes.
Sat Mar 21 16:35:30 2015 -> Limits: MaxHTMLNoTags limit set to 2097152 bytes.
Sat Mar 21 16:35:30 2015 -> Limits: MaxScriptNormalize limit set to 5242880 bytes.
Sat Mar 21 16:35:30 2015 -> Limits: MaxZipTypeRcg limit set to 1048576 bytes.
Sat Mar 21 16:35:30 2015 -> Limits: MaxPartitions limit set to 50.
Sat Mar 21 16:35:30 2015 -> Limits: MaxIconsPE limit set to 100.
Sat Mar 21 16:35:30 2015 -> Archive support enabled.
Sat Mar 21 16:35:30 2015 -> Algorithmic detection enabled.
Sat Mar 21 16:35:30 2015 -> Portable Executable support enabled.
Sat Mar 21 16:35:30 2015 -> ELF support enabled.
Sat Mar 21 16:35:30 2015 -> Mail files support enabled.
Sat Mar 21 16:35:30 2015 -> OLE2 support enabled.
Sat Mar 21 16:35:30 2015 -> PDF support enabled.
Sat Mar 21 16:35:30 2015 -> SWF support enabled.
Sat Mar 21 16:35:30 2015 -> HTML support enabled.
Sat Mar 21 16:35:30 2015 -> Self checking every 600 seconds.
Linking amavisd to clamav
Amavisd should connect to the socket of clamd and thus clamav needs to be enabled as one of the main antivirus scanners. The fallback of invoking clamav from the commandline should not be changed. Also the virus check bypass needs to be disabled to be effective:
/etc/amavisd.confEnable clamav# @bypass_virus_checks_maps = (1); # controls running of anti-virus code
['ClamAV-clamd',
\&ask_daemon, ["CONTSCAN {}\n", "/var/run/clamav/clamd.sock"],
qr/\bOK$/m, qr/\bFOUND$/m,
qr/^.*?: (?!Infected Archive)(.*) FOUND$/m ],
After restarting amavisd, viruses should be able to detected and blocked:
root #/etc/init.d/amavisd restartTesting
To test whether the virus filter works, an anti-malware testfile exists, sending an e-mail using this string should trigger the virus scanner:
X5O!P%@AP[4\PZX54(P^)7CC)7}$EICAR-STANDARD-ANTIVIRUS-TEST-FILE!$H+H*
Again, it should be ensured that this is message is passed through the sending mailserver, and only be blocked by the receiving server (foo.example.com). Using a client that connects to port 25 and is allowed to relay should suffice.
Looking at the mail.log file, the following should be revealed:
/var/log/mail.logVirus infection testDec 28 14:05:33 7of9 amavis[7554]: (07554-01) Blocked INFECTED (Eicar-Test-Signature) {DiscardedInternal,Quarantined}, MYNETS LOCAL [10.0.0.2]:41144 [10.0.0.2] <root@test.example.net> -> <testuser@example.com>, quarantine: virus-WYHuLpwdzPVr, Queue-ID: 7FC7B22DF6, mail_id: WYHuLpwdzPVr, Hits: -, size: 407, 166 ms
Dec 28 14:05:33 7of9 postfix/lmtp[15452]: 7FC7B22DF6: to=<testuser@example.com>, relay=127.0.0.1[127.0.0.1]:10024, delay=0.41, delays=0.21/0.02/0.01/0.16, dsn=2.7.0, status=sent (250 2.7.0 Ok, discarded, id=07554-01 - INFECTED: Eicar-Test-Signature)
In theory, the virus scanner should be fully functional now.
Spam Assassin
Introduction
Spam Assassin is an excellent spam filter. It has become quite complex throughout the years and requires some effort to configure correctly.
Installation
There really should be no need to install spamassassin separately. The spamassassin USE flag should have pulled it in as a dependency of amavisd-new.
root #emerge --ask mail-filter/spamassassinConfiguration
The default configuration suffices for standard use.
If users don't e-mail via the submission port, the
trusted_networks variable should be setup properly to bypass spam filtering from known users.Updating
A key feature of Spam Assassin is its ability to self-update. Updates are handled via a so-called update channel.
Spam Assassin comes with the sa-update tool so updates can be fully automated. Spam Assassin updates can be done by using the --nogpg flag to ignore gpg keys, but should really only be done as a last resort. Adding the spamassassin GPG key is a simple 2 step process.
root #wget "http://spamassassin.apache.org/updates/GPG.KEY"root #sa-update --import GPG.KEYroot #rm GPG.KEYPossibly due to bug #396307 it is required to first create a directory for the keyring and set proper permissions prior to importing the keys.
root #mkdir -p /etc/mail/spamassassin/sa-update-keys; chmod 700 /etc/mail/spamassassin/sa-update-keysAfter adding the spamassassin update channel, it needs to be updated. After running this command, check for any errors:
root #sa-update -DOnce these updates have completed they need to be compiled for use with Spam Assassin. Also any errors should be spotted here:
root #sa-compileUnlike clamav, there is no 'freshassassin' and a cronjob is required to do updates. To keep Spam Assassin up to date, a cronjob should be created for the task:
/etc/cron.daily/sa-updatesUpdate Spam Assassin filter rules#!/bin/sh
sa-update --allowplugins --channel updates.spamassassin.org
sa-compile
Making the cronjob executable ensures it runs regularly:
root #chmod +x /etc/cron.daily/sa-updatesLinking amavisd to Spam Assassin
Actually, Spam Assassin does not need to be linked to amavisd, it is an integral part of amavisd. Enabling the spamfilter in amavisd does spam filtering:
/etc/amavisd.confEnable Spam Assassin# @bypass_spam_checks_maps = (1); # controls running of anti-spam code
With this change, amavisd needs to be restarted:
root #/etc/init.d/amavisd restartTesting
Testing is done again via a client connecting to port 25 that does not have its own spamfilter. For content GTUBE can be used. On that site there is also a suitable mail message in RFC-822 format.
XJS*C4JDBQADN1.NSBN3*2IDNEN*GTUBE-STANDARD-ANTI-UBE-TEST-EMAIL*C.34X
Checking in the testusers inbox, or junkbox more likly, the message can be found and its header examined:
Received: from localhost (localhost [127.0.0.1])
by foo.example.com (Postfix) with ESMTP id B7EDA22F62
for <testuser@example.com>; Wed, 28 Dec 2011 16:02:46 +0100 (CET)
X-Quarantine-ID: <BukBLFBE3OWo>
X-Virus-Scanned: amavisd-new at example.com
X-Spam-Flag: YES
X-Spam-Score: 1000.907
X-Spam-Level: ****************************************************************
X-Spam-Status: Yes, score=1000.907 required=6.2 tests=[ALL_TRUSTED=-1,
FH_FROMEML_NOTLD=0.18, GTUBE=1000, MISSING_HEADERS=1.207,
MISSING_MID=0.14, NO_DNS_FOR_FROM=0.379, TVD_SPACE_RATIO=0.001]
autolearn=no
Finetuning Amavisd
Amavis has a few more settings that can be changed to do some fine-tuning.
Recipient delimiter
When using postfix with the recipient_delimiter amavisd can be told to make use of this feature:
/etc/amavisd.confrecipient_delimiters for amavis# Delimiter must match the equivalent (final) MTA delimiter setting.
$recipient_delimiter = '+'; # (default is undef, i.e. disabled)
It might be interesting to add the following to /etc/postfix/main.cf, otherwise the user+foo@domain might not be delivered:
/etc/postfix/main.cfrecipient_delimiters for postfix# ADDRESS EXTENSIONS (e.g., user+foo)
#
# The recipient_delimiter parameter specifies the separator between
# user names and address extensions (user+foo). See canonical(5),
# local(8), relocated(5) and virtual(5) for the effects this has on
# aliases, canonical, virtual, relocated and .forward file lookups.
# Basically, the software tries user+foo and .forward+foo before
# trying user and .forward.
#
recipient_delimiter = +
Disperse quarantine
It is possible to disperse the quarantine over several sub-directories. For this directories need to be created first:
root #mkdir /var/amavis/quarantine/virus;
mkdir /var/amavis/quarantine/spam;
mkdir /var/amavis/quarantine/banned;
mkdir /var/amavis/quarantine/badh;
mkdir /var/amavis/quarantine/clean;
chown -R amavis:amavis /var/amavis/quarantine/*;
chmod -R gu+rwX /var/amavis/quarantine/*;
chmod -R o-rwx /var/amavis/quarantine/*
/etc/amavisd.confDisperse quarantine#$clean_quarantine_method = 'local:clean/%m';
$virus_quarantine_method = 'local:virus/%m';
$spam_quarantine_method = 'local:spam/%m.gz';
$banned_files_quarantine_method = 'local:banned/%m';
$bad_header_quarantine_method = 'local:badh/%m';
Also, setting a spam cutoff level helps in reducing stored spam. The cutoff level makes it that no spam is stored above a certain spam-score:
/etc/amavisd.confset a cutoff level for spam$sa_quarantine_cutoff_level = 25; # spam level beyond which quarantine is off
25 is the default. This value, aswell as the other levels can and should be tuned to organizational needs.
Spam Delivery
$final_spam_destiny is by default set to D_PASS, meaning that even with a high score at which it gets marked as spam, it is still delivered to the users mailbox. Modern mail-clients, which trust Spam Assassin, can then automatically move it to their SPAM folder.
Bayes database path
Set the bayes_path option in SpamAssassin's configuration file so tools such as sa-learn write to the correct database location:
/etc/spamassassin/local.cfset bayes_path to be within Amavisd's home directorybayes_path /var/amavis/.spamassassin/bayes
Virtual hosts
If this server handles more than one domain, telling amavis can help here:
/etc/amavisd.confAdd additional aliases to amavis@local_domains_maps = ( [".$mydomain", "mail.example.net", "mail.example.org", "mail2.example.com"] );
Cleanup
With Spam Assassin and ClamAV working as expected, debugging information can be reduced to the normal minimal:
/etc/amavisd.confDisable debugging in amavsd# Section III - Logging
# true (e.g. 1) => syslog; false (e.g. 0) => logging to file
$do_syslog = 1; # (defaults to 0)
#NOTE: levels are not strictly observed and are somewhat arbitrary
$log_level = 0; # (defaults to 0), -d
# Turn on SpamAssassin debugging (output to STDERR, use with 'amavisd debug')
#$sa_debug = '1,all'; # defaults to false
/etc/clamd.confDisable debugging in clamav# Enable debug messages in libclamav.
# Default: no
#Debug yes