Adventures in Email Security and Archival

Blog Merge

mkatz — Wed, 28 Mar 2007 15:02:55 +0000

I have merged all blogs into one consolidated blog. Thank you for your visit and please view the new location.

Avoiding Dead DNS Blacklists

mkatz — Wed, 28 Mar 2007 14:28:36 +0000

Came across and excellent link that gives the status of RBL sites. Using RBL’s to fight spam is absolutely crucial, but using the right ones is even more important. Personally, I only use zen.spamhaus.org, but there are many. It is a major mistake to use spamcop for rejection criteria in my opinion. DNS RBL Info on spamlinks.net

MPP Enters the Podcast Generation

mkatz — Fri, 09 Mar 2007 18:45:47 +0000

Multimedia has finally come to life here at Message Partners, thanks to the dynamite team of Pete and Prakash. So with no further adieu I introduce you to the inaugural podcast of the blog where I am interviewed by Peter Schooff. In this gripping and insightful interview, I postulate on MPP’s position in the email security market, the future of MPP, and explain why MPP is different than all of the other email filtering products around. download the file

Technorati Tags: MPP, Message Partners, Antispam, open source, spamassassin, email security

Is It Time for SpamAssassin to Die?

mkatz — Fri, 16 Feb 2007 21:17:00 +0000

We are flooded with requests by people that are tired of SpamAssassin. Tired of the slow performance, poor accuracy and endless time of administration. Small sites hate the poor accuracy, large ones hate the performance and admin time. SA has struggled to keep up with latest image spam and the increase in spam levels. Recent attempts at OCR plug-ins have only had limited and temporary success, while further reducing performance. SpamAssassin is probably the most widely deployed antispam filter in the world, yet, spam has only increased, which is testament to it’s failings. I guess the same could be said of the antispam industry in general, which has grown tremendously over the past 5 years. Even with the mercurial growth of this sector, only the growth in spam seems to outpace the antispam industry growth curve. A digression, sorry. On the topic of performance I always get a chuckle when a customer has to increase our SA timeouts to scan from our default of 6 seconds to about a minute to handle all cases of SA lameness. Our commercial scanners scan spam in less than a few milliseconds, while SA can choke for a minute at times. I have worked with customers who have replaced 8 SA servers with one MPP/Cloudmark server with room to spare. Of course SA can be fine tuned in the local.cf file, but the percentage of SA users that we deal with that understand this file is quite low. SA is a great tool for geeks and hard core admins with lots of free time, but to the rest of the world I think it is time to admit the failure of SA and look to new solutions. Even though I will be unliked for this, I don’t think that open source is the place to look for antispam technology unless you have tons of admin skills, expertise and time. Other projects like DCC are only more complicated than SA, more limited in implementation models and require much more human interaction to work effectively. There are open source projects like SPF, Domain Keys, Greylisting and Sender ID that can help, but these are only building blocks of larger architectures. In my view, the commercial software industry is where spam must be tackled for a few reasons. First of all, there are great solutions that exist already that are far more effecitve and scalable than SpamAssassin. My product MPP works with a few commercial content scanners (cloudmark, commtouch, mailshell), and all blow the doors off of SA in all regards except for memory utilization. I am always amazed when a customer will favor commercial av scanners over spending on commercial spam scanners. MPP allows admins to use open source or commercial spam and virus scanners, which is a big benefit, however, our experience shows that open source AV scanning is far more effective than open source spam scanning. For large environments open source av scanning is generally not feasible due to scaling and security issues, but for many small environments it is entirely adequate. So,,,is it time to retire SA and start looking for better alternatives. I say unless you have tons of time, great admin skills and a serious interest in constant tinkering, the answer is YES. Retire the old idea and think differently about new solutions. My .02 cents!

Technorati Tags: spam, antispam, antivirus, email security, email, spamassassin, clamav, cloudmark, commtouch, mailshell, mpp

New Antispam Defenses in MPP

mkatz — Thu, 25 Jan 2007 19:47:48 +0000

I usually don’t write about my own product, MPP, in this blog, but I am very excited about some of our new features. Our goal is to build as much spam detection capabilities as possible into smtp protocol level analysis and to defer as little email as possible to content scanning engines. With this in mind, we have introduced a few new features in MPP that help towards this goal. One features is ‘thresholds’, which allows providers to define characteristics of email streams in terms of message rate/time period and spam rate/message volume. Senders that either send too much mail or too much spam will be automatically blocked at the smtp level for a specified period of time. This is very helpful to stop drone attacks, but is also a great tool to stop outbound spam. Since both clean and spam mail can be measured this is a very useful tool to find abusive senders. The other feature of interest is spam-traps. With this feature admins can set-up address templates using regular expressions. Hosts that send to these addresses will be blocked at the SMTP level from sending. This is a great tool to stop dictionary attacks and in large settngs can be quite an effective and simple tool.

Technorati Tags: antispam, spam, outbound spam, email, email security

Incoming Address Verification - Critical Antispam Defense

mkatz — Tue, 09 Jan 2007 21:31:13 +0000

At Message Partners I work with many customers on antispam defense systems. Many of our customers use our software (MPP), on SMTP filtering proxies and I am surprised how many common it is to have no strategy for verifying incoming email addresses. The cost of accepting email for non-existant addresses is high. In the SMTP transaction there is a greeting that is followed by the actual data or email. It is desirable to stop as much spam as possible after the smtp greeting, before the actual email data is sent from the remote smtp client. If a remote smtp client tells you in the greeting that it is sending mail to asdf@yourdomain.com and you have no asdf at your site you are better off to reject the greeting than to accept it and then accept the email data. Having a list of valid email addresses in your organization will allow you reject email more efficiently. SMTP proxies or spam appliances can verify email addresses by checking lists, ldap directories, databases or using smtp verify transactions. SMTP verify is the simplest way to verify messages, however, there are scaling issues for large sites. Having a centralized LDAP directory of all valid email addresses will scale, however, this is difficult for many service providers. Active Directory or other user directories can be queried directly, but for non-msft shops there are good ldap directories to consider. OpenLDAP is the standard open source directory, but the GUI interfaces tend to me confusing and if you are not an LDAP pro it can be intimidating. Redhat has a directory server, http://www.redhat.com/software/rha/directory/, and a free version called the Fedora Directory Server, that are worth checking out. Needless Processing - Email to non-existant users should be dropped before the STMP Data transaction, i.e. before the message is accepted. If you process email for non-existant users you are wasting bandwidth, processor and storage resources if you quarantine spam. In summary, if you don’t validate incoming email addresses you are asking for trouble. If you are using spam quarantine you will fill up your quarantine with bogus emails and pollute your user tables. You are wasting bandwidth by processing junk email to fake accounts and you are wasting storage and processing resources. Centralized directories are a powerful antispam defense and they are worth the effort that they take to establish.

Technorati Tags: spam, antispam, email, ldap, directories, mpp, message partners, isp

IronPort Investors: Merry Late Christmas

mkatz — Mon, 08 Jan 2007 15:52:28 +0000

Wow! Cisco paid about 10x more than any antispam player that I am aware of for Ironport. Ironport certainly seems to have a great eroding customer base and sales infrastructure, but it is nothing compared to what CSCO already has. Not sure if there is a technology asset that I am not aware of that distinguishes them so much, but I am not aware of anything groudbreaking. I guess it is good to have good banking buddies. Congratulations Ironport investors.

Technorati Tags: Cisco, IronPort, Spam, Antispam

Lessons of Greylisting

mkatz — Thu, 04 Jan 2007 19:45:45 +0000

MPP v3.2 has introduced Greylisting via our implementation of a Postfix Policy Server. MPP is the only integrated pre and post-queue filtering solution available for Postfix, either open source or commercial. Since we have introduced greylisting we have learned some lessons that are worth sharing. First of all, a brief description of greylisting. The idea is to send a temporary failure message to every new smtp client that an smtp server sees. A real email server will retry their message after a temp fail message, but most spam drones go away and never retry. Supposedly spam drones are wising up to this, but so far we don’t see a lot of evidence of this. See http://greylisting.org has more details. In our very small environment we have found some astounding stats in 3 weeks; 47615 remote smtp clients have tried to send our email server. Of this number, 45149 clients never tried to resend again, in other words 94% of the remote clients were spam drones and just went away. Only 274 remote clients were legit, or less than 1%! That is an astounding number if you ask me, which you didn’t Not everyone likes greylisting because it delays email. Since 94% of clients simply give up after the first failure, even a retry delay of 1 second can still be very effective. The problem is that how long it takes for a remote email server to resend after a tempfail message is entirely up to the remote server. If you allow hosts to resend 1second after their first attempt and the remote host will retry in 20 minutes the recipient of the email sees a 20 minute delay and yells at the admin when they need something fast. The good thing for MPP users is that greylisting can be enabled on a per-policy basis so that it can be easily turned off for domains or users that don’t want it. This is a pain to do in most policy-server implementations and is a huge plus for MPP. Greylisting can work on a gateway or on an email server, but care must be taken when deployed on an email server since email clients use smtp to send email. It is a pretty simple workaround to not perform greylist checks if all hosts are on known networks, but it some environments this is not known information and problems can arise. Specifically, clients will see failure messages when sending email and will get angry. So if you are implementing greylisting on your email server and not on your smtp gateway be careful to whitelist local networks from greylist checks. Our solution uses MySQL as a backend to store remote host information and this database can become quite huge in large environments. We have seen about 1 million remote hosts in a few days in large environments and 95% were drones. This becomes a very large database, especially with the innodb storage engine so if you plant to use greylisting be very careful to properly size your database server in a large environment. The spam and phishes that passes greylist checks are sophisticated and are caught at about a 50% rate by content scanners. So while a content scanner may be 95% accurate on all traffic, it becomes about 50% accurate on spam that passes greylist checks. Mostly phishes and some image spam will get through the greylist checks. That’s all for now, please check out http://messagepartners.com for more on MPP and our greylisting capabilities.

Technorati Tags: MPP, antispam, postfix, greylisting, spam, email security, email

2007 - Spam is Back with a Bang

mkatz — Wed, 20 Dec 2006 20:10:58 +0000

Submitted by Peter Schooff Two years ago, Microsofts Bill Gates predicted that by the year 2006, spam would be finished. While even at the start of this year it looked like Gates prediction might still come true, since then spam has experienced a resurgence. 2006 now marks the year that spam has returned with a vengeance. In speaking with our customers at Message Partners, we have heard estimates of current spam levels ranging from around 70 percent to as high as 90 percent, or 9 out of every 10 email messages today are complete junk. While our intelligent email engine blocks out most of this spam so that it is nothing more than a nuisance, the recent surge in spam overall can be attributed to two main factors: the rise of botnets and image spam. Botnets refer to massive collections of computers that have been infected with malicious software, generally without the computer owners knowledge, so that they can be controlled by an external operator. Botnets are often created solely to spread spam. And where spam could once be fought by shutting down the source of the spam (and was probably the reason for Bill Gates optimism), botnets exponentially increase the number of computers used to send out spam. Botnets have been uncovered that range in number from a few thousand to more than one million infected computers, making shutting down every single one impossible. The other factor in spam’s recent rise is the switch from text to image-based spam. Where just a year ago filters were having great success screening out spam by searching for words like stock and sex and enlarge, the new spam comes with the sales pitch entirely embedded in an image, severely hampering the detection abilities of many spam scanners. And with spam flooding inboxes, this will inevitably lead to a shakeout in the industry as customers search for better methods to stay on top of this deluge. Message Partners has found that the only way to survive what has become a virtual tech arms race is by retaining the essential ability to adapt. Spams change in tactics has also brought with it a major change in content. While there are still plenty of spam emails promising easy riches or offering an impossible pharmaceutical miracle, Message Partners has kept close watch on several new and more dangerous types of spam that we expect to see much more of in 2007. The first is the stock pump-and-dump scam. While this scam has been around since the inception of the internet (and long before that), only recently has it come to encompass hyper-aggressive spamming. This type of spam is easy to spot, as the subject line just about screams out the latest hot stock pick. The fact that simply mentioning a stocks name to millions of people in an email inevitably drives up the stocks price, coupled with the fact that spammers can make money directly from buying and selling stock without having to rely on a secondary sales site, means this type of spam is likely to stick around for some time. The second and far more dangerous type of email that has come with the new wave of spam is known as phishing. Phishing is fake email that purports to be official email from a bank or legitimate website for the purpose of stealing password or financial information. Phishing emails, and the subsequent websites they link to, tend to steal their graphics directly from the sites they are trying to imitate, which means, visually, they are almost impossible to tell apart. While there have been countless warnings about never responding to, or clicking on a link from, any email asking for personal information, phishing has proven so effective that one can only conclude that all it takes is for a few phishing emails to make it into a few inboxes before someone gets hooked. Bill Gates prediction about spams demise shows just how tricky it can be to make predictions about technology, but it seems self-evident that both the amount of spam and the dangers it poses will only increase in 2007. Also, the ease with which these crimes can be committed, along with the cash windfalls that can be won by even modest success, means that cyber crime has become a major focus of organized crime. That is why Message Partners has never stopped focusing on building one of the most versatile and adaptable email engines available anywhere. We have also continually added to our state-of-the-art weapons to fight spam and viruses and phishing in whatever form they take. As James Joyce, the Director of Plug and Play computers, a major internet service provider in Australia, said, Without Message Partners our systems and customers would be overloaded with junk email. At Message Partners, we have never stopped believing that as long as we keep using email, email will keep trying to use us. And as email grows ever more important as a vital form of business communication, that makes choosing the right email platform all that much more essential. Tags: Spam, Spam 2007, Botnets, Image Spam, Phishing, Stock Pump and Dump

Lots and Lots of Spam

mkatz — Mon, 27 Nov 2006 13:19:32 +0000

According to Postini, 9 out of 10 mails are spam, http://www.cnn.com/2006/WORLD/europe/11/27/uk.spam.reut/index.html. This is totally believable given the fact that spam is a very inexpensive marketing tool, people seem to make trading decisions on spam email, people buy porn from spam, people give away thousands of dollars to scammers via spam, people buy drugs from spam and people generally react favorably to spam. Simply put, spam works. Botnets operators are far more sophisticated than most of the detection tools that exist and hackers are far more sophisticated than end-users. Message Partners has great spam filtering solutions, but in the end, spam is a human problem and technology can not fix human problems.