
I've spent the weekend tweaking and defining rules and filters for e-mail on two of the most popular e-mail programs - Apple's OS X Mail and Microsoft's Entourage.
A key part of both of my tests involved a means of statistical analysis called Bayesian modeling. Whew. This is complicated stuff. Essentially, it is a methodological tool for intelligent data analysis, knowledge discovery, and machine learning based on how many times a certain word or phrase or pattern appears. The more you identify certain messages as spam, the more this tool "learns" and keeps track of it in a "corpus," or list. Eventually, it automatically classifies the messages as spam and moves them or deletes them as you desire. Click here to learn more about the concept.
Anyway, Apple's Mac Os X Mail application comes standard with a Bayesian filter. After training it with a bunch of good and bad e-mail, it takes over and gets better day by day.
Keep reading for the results...
Entourage's junk mail filter works differently. It is not Bayesian. In other words, it doesn't "learn" as actively. You set it for a sentitivity level and it sort of guesses by content and domain and your "good mail, bad mail" designations.
Head-to-head, with just the standard junk filters, Apple's won every time in my preliminary tests this weekend with my six different accouts.
Using just those two basic filters Friday with 75 pieces of e-mail, Apple correctly identified 57 as spam and incorrectly classified 5 good messages as junk. On the highest setting, Entourage caught 41 real spam messages and mistook 18 good ones.
Then I set up a different test from Friday night through Sunday night in which my six accounts collectively brought in about 450 pieces of e-mail -- 325 of them spam.
But there are more spam-fighting tools. Both programs also allow you to set "rules," whereby messages are deleted, transferred to another file or marked as good. You can set these rules up as elaborately as you want and while it's tricky to get the parameters and conditions worded just right, they can be very powerful. Entourage, I found, has the most flexible rules.
Activating the Entourage junk filter and adding rules about equaled the Bayesian filter that is standard with OS X Mail, at least according to the results I saw. I found a lot of help on this FAQ page run my Entourage MVP volunteers. Though I also added roughly similar rules to Apple's Mail, Apple's didn't seem nearly as effective as Entourage's rules.
But my experiment included another element with Entourage. Through a third party software program called SpamSieve, you can also add Bayesian filtering to Entourage. It's tricky to set up. You have to mark and move and identify good e-mail and junk e-mail and you need to carefuly follow the instructions and tweak the program.
But when I added Spam Sieve to Entourage Rules and the junk filter, I clearly found it superior to Mail OS X alone.
Apple offers its standard Bayesian filter and user-defined rules. Entourage comes with a non-Bayesian junk rule filter and user-defined rules. To that I added SpamSieve. So, in a way, SpamSieve gives Entourage a three-pronged approach to fighting spam, compared to Apple's two-weapon arsenal.
So here's some bottom line results.
Of the 325 spam e-mails my accounts received this weekend, Mail OS X correctly filtered out - with the Bayesian filter and some basic spam rules configured - 233. Another 14 e-mails that turned out NOT to be spam also were filtered out. The rest of the spam got through.
With Entourage and Spam Sieve, using the standard junk filter and about the same rules as I had set for OS X Mail, 281 spam messges were grabbed. But 21 good e-mails were misidentified as spam.
This is just for a weekend. And I'm sure I wasn't neary as scientific as a testing lab would be an. But the results - to me at least - seemed to give a clear advantage to Entourage.
Either way, though, my conclusion is that filters and rules do work and Bayesian rules rock the best.
I would also have to include POPFile in the running. It works much better than SpamSieve and though it takes more effort to set-up, is well worth it. SpamSieve does work well, and I found it to be better than Mail, which seems to stop learning after a while. Entourage with POPFile is the answer for me...
(I do keep SpamSieve running as well since I paid, but it never gets to hit anything since POPFile seems to get it all.)
Posted by: Jonathan Greene at May 19, 2003 10:26 PMDoes POPFile handle IMAP???
Posted by: Mac Mike at May 19, 2003 11:26 PMThe biggest mistake you made is that Mail.app will get better over time, to the point where no legitimate email will be filtered out at all. I have gone months now without losing a single real email and having about 90% of the spam caught. This is with around 100 emails per day.
Posted by: Gerige at May 20, 2003 01:10 AMusing one's own software filters hasn't been effective for me so far... i got the best results with SpamAssassin, a spam-filter service with my email provider fastmail.fm :)
Posted by: greg at May 20, 2003 04:52 AMI am curious as to what you define "spam" as. Do you count any message you wish you didn't have to read? Or do you acknowledge that you may have given permission to some companies to mail you so while all of their messages aren't always welcome they aren't spam?
Posted by: Paul at May 20, 2003 05:41 AMBeen running SpamSieve with Entourage since september last year. Accuracy is now 94%. I've gotten 33 false positives in those nine months. Amazing...
Posted by: Peter N at May 20, 2003 07:26 AMI NEVER! give permission for anyone to send me advertisements. And personlly, I think spammers should be harassed, defamed, picketed, ridiculed, shunned and exposed as the slimy, morally bankrupt, pond scum people they are.
Posted by: Mac Mike at May 20, 2003 08:08 AMTell us how you *really* feel, Mike. :)
Posted by: Chimpee at May 20, 2003 11:40 AMSo if you buy a book from Amazon and then the author of the book writes another book and Amazon writes you an e-mail telling you about it then it is an advertisement and you do not want to receive it?
Posted by: Paul at May 21, 2003 12:16 AMI think the point is not that you want to recieve spam, legit or not, but if the spam filters are smart enough to decide what looks like spam and what doesn't. IE: Whether I told hotmail to send me ads (or sell them to their affiliates) is not the point (although I get about 100 msgs a day to my hotmail account which I have never used for anything or posted anywhere), I want to stop the email coming through.
Personally I use bogofilter on the server level, and it works well, though it does fall outside the realm of the test Mike performed :)
Posted by: Arcterex at May 21, 2003 11:51 AMMy point is just that if a reviewer doesn't define "spam" then their results are meaningless. If you say only 2% of mails were incorrectly tagged as spam it sounds very precise-- so I would assume there would be a somewhat precise definitition of what spam means to them.
If the filter tags everything that "looks like spam" then it will surely tag some things incorrectly.
I am not saying I don't like the spam filters. I use them only to tag spam and put it in a different folder.
I think it is terrible that some ISPs block spam for their customers without consulting them or even notifying them first.
Posted by: Paul at May 21, 2003 01:45 PMMike, I think you'll probably see better results after more training. You should be seeing much higher accuracy from good Bayesian filtering especially. I'll second the above endorsement of the filtering in the new Eudora beta -- it's excellent. I've used rule-based filters wiith hundreds of rules, SpamAssassin, Entourage's filtering, SpamSieve... Eudora 6 beats them all in accuracy, performance, and ease of use (which is the point, after all -- we hate spam because it wastes our time and attention).
FWIW here are my stats for the last seven days. Notice that I compute my false positive rate based on the number of good messages ("ham") I receive, rather than the total number of messages. I agree wiith Paul Graham that false positives are a much more serious failure than false negatives, and so 1.1% is really too high. But training will lower that -- it just takes longer because the volume of legit messages is so much lower.
Personal mail stats for last 7 days:
ham received 91
spam received 2639
total received 2730
false negatives (appx.) 49
false positives 1
false positive rate 1.1%
false negative rate 1.9%
Gentlemen,
I came to this site as a semi-computer-literate man with a lot of Spam to kill. Speaking as such, I can say that you have all been most helpful.
I have found that Spamsieve with Entourage seems to work best with the least effort (and $20 ain't bad). It is actually outperforming my wife's OSXMail now.
Brilliant.
Thanks for having this info out there.
Posted by: Collin at July 14, 2003 10:27 AM
Don't forget about the new Eudora 6 beta-- it also has a Baysian spam filter, and it has very little to learn. After about a week of helping it out a few times when it erred, I haven't had a false positive in two days. Very impressed with this new Eudora, though the rest of the version is hardly changed from 5.2, so I am not sure if it deserves a whole point update.
T
Posted by: ToeKnee at May 19, 2003 12:58 PM