Comparing Spam Filters: Gmail, Yahoo! Mail, and SpamCop

One false positive is too many in my book.

I'm OK with a few false positive, especially when you consider the amount of email I get. The important thing is that I haven't yet tried to get Yahoo or Gmail to recognize when a false positive happens. SpamCop, I've used for years, and I note there's some addresses on my whitelist that may be helping it. Later this week, I'm going to start whitelisting to see if Yahoo and Gmail do better on the false positive front because of that. But overall, Gmail did seem much more impressive out of the box than Yahoo on the filtering front, no special arrangements in place.

I don't really understand the point of the percent spam filtered figure - surely the way it has been done means that if you get more spam on gmail then it'll look better? It would be more relevant to see how much spam it filtered overall (i.e. spam_caught/total_spam) - the only downside is you have to do this manually.

I get about a 80% catch rate on gmail, which is appalling. But I also refuse to do personal training - a spam filter should just work, not make me work (I guess I'm biased though).

Is there a specific filter setting in gmail? In my gmail account, I get at least 10 spam emails every day. I have noticed that some of the emails are from the sender(s). Even after marking it as spam, gmail does not seem to catch it the next time it comes around.

Another scenario is forwarding stuff from yahoo to gmail. Does it make a difference if you forward stuff from a yahoo account into gmail? I guess that would not affect the quality of spam blocking, right?

av, of course forwarding will affect spam filters because it invalidates all checks on IP of incoming SMTP connection.

I've had a few false positives using gmail.

Most of my false positives on Gmail have been because of forwarding causing SPF failures. Since there's no way to whitelist forwarding IP addresses, I'm still trying to figure out a good solution.

I was at about a 33-50% false positive rate on Gmail, but now it's down to around 10-15%. Most of the false positives are from mailing lists (freebsd-questions, debian-user) but some are from friends and business contacts. I forward email to it instead of using the gmail.com address, which may have something to do with it. I dunno. Maybe I have the same problem as Keith Ivey.

I've contacted Gmail to ask them if there's a way they can let me tweak the filter myself for my account, or how to disable filtering entirely, but they won't do either.

I have both a Gmail account and a Y! (beta) account. I have a copy of mail sent to me at mypersonaldomain.com automatically forwarded to each account.

Y! counts about 15-20% of it as spam, despite me repeatedly clicking on "this is not spam." I've given up. Clearly, Y! is not built to work properly in this situation. :|

(oh, and for the record, I have not detected any false positives from Gmail in the last 6 months)

On the percentage spam filter, you are correct that it is not entirely accurate. I explain this more in the key to the stats. In short, let's say Gmail might catch 400 items it calls spam, from 600 sent. So we're looking at a spam catch rate of about 66%. Ah, but what if it's catching good stuff by mistake? If half of what it counts as spam is good stuff, it gets a higher spam catching rate than it deserves. That's why I manually check for false matches and subtract that from the overall spam figure -- plus give you the false match rate as well. OK, but what about the remaining 200 good items. What if there's spam in there. There is indeed spam in there. Counting that is harder, though Mailwasher can give me some stats I may report in the future. I'd say something like 10-30 percent of my "good stuff" from Gmail is spam it is not catching. I had the same issue, though slightly smaller percentage, with SpamCop. That's why I use Mailwasher in combination with server based filtering, as an extra solution. Ideally, Mailwasher would be able to report spam to Gmail in the way it can with SpamCop -- but that's not somethig it can do, at the moment.

Yahoo is a Giant Spam Magnet. Yahoo's SpamGuard Plus feature is a big Negative and a Big Joke. Spam emails deleted in Yahoo show up again the next day or sometimes several hours later in the same day. It is useless to use the spamguard. I just gave up trying to fight the spam in Yahoo. I just let it pile up in Yahoo's BulkMail folder.

So why the crappy selection of anti-spam tools?

I can understand Gmail and Yahoo because they are free and everyone uses them, but SpamCop?

It doesn't make any sense to me. They aren't considered a top anti-spam service. They haven't been for ages and ages. There are much better anti-spam tools/services out there now days.

Were you trying to cover good anti-spam tools/services? Or were you trying to cover stuff services that a wide variety of people use? I don't think SpamCop fits in either way.

(must disclose: I'm a former employee of an anti-spam company.)

Yeah, Ben has a point here.

Personally, I think POPfile -- although requiring some initial effort -- is the most amazingly accurate anti-spam tool around.

Oh, and Jeremy? What's with the "Stay topic?" command above? Is that like "Sit, UBU, sit!"? ;)

Why SpamCop? It's explained at the beginning of my series. I've used SpamCop for ages. But they had a glitch recently that blocked me from getting my email for a few hours. So I checked out Gmail and switched over temporarily, since they now have POP access. But I then decided that Gmail seemed to be doing nearly as good on spam filtering as SpamCop, so I was going to stick with it. But I was curious to know more that just on a gut feel, so I thought I'd do some stats for a few days.

There may indeed be better spam tools out there. I was really trying to compare with what I know to two major email services that are free to anyone.

Gmail, in particular, is a special case. I don't know of any other free email service essentially allowing you unlimited storage. Combine that with pretty decent spam filtering along with POP, and I'm finding it very compelling. Yeah, I'd love if Gmail caught the other 10-30 percent of spam it lets through. But Mailwasher on top of Gmail pretty much stops the rest of it. Overall, I get an archive of all my email plus decent spam filtering. Works well.

I'd love to see someone do an even more extensive test of a wide range of options out there. Just goes beyond what I was trying to show.

One other thing. Adam, I looked at POPfile. It seems somewhat similar to Mailwasher, in that it's going to hit your mail server, then filter what the server sends.

Ideally, I want my mailserver to keep the junk from ever showing up in my inbox at all. That doesn't happen, which is why I use Mailwasher. But until about a year and a half ago, I had no broadband. Spam wasn't just annoying. With the amount I get, downloading the junk over 56K was time consuming.

That's why I could never rely on something like Cloudmark (which I ran for years) or Outlook 2003's native spam filtering, at least as a first line of defense. I needed to keep spam out before downloading. Filtering after download was a last line of defense. If you fought through SpamCop, then through Mailwasher, finally Cloudmark or Outlook would help me -- but only few items got that far. I'm quite the Mailwasher fan.

So some of my narrowband habits still remain. I could just pull everything down and filter, but I still prefer to keep it out. And Mailwasher is a fast way to prefer what's made it past the mail server spam filters and delete easily (plus quickly preview mail). I only wish Mailwasher could report back to Gmail the spam that's getting through, so Gmail would learn. But that seems down to Gmail having an API for developers to use.

Thanks for collecting the data. I have a couple comments from my own use of Yahoo, Gmail, and Excite.

By far, Excite does the worst job of spam catching for me. I believe this is because I use that account to sign up for any mailing list or group I belong to. As a result I think that account gets more stuff (and thus a larger variety) thrown at it. Yahoo is my longest held account, although the excite one is only slightly younger. So my yahoo account has been out there for quite a while and also gets a variety of stuff thrown at it. Gmail is very new and doesn't get much.

Almost no spam gets through Yahoo for me. It all goes to the bulk folder. I do get some false positives and have marked them as allowed and that always seems to work. This same technique doesn't seem to help with Excite. The false positives that I see in the Yahoo bulk folder happen once every couple weeks to a month though, although I could be missing them. However I doubt they are happening on a daily basis.

I'm curious how much overlap in spam senders you have between the various email sites you used. Can you give some numbers indicating if Yahoo and Gmail are seeing the same spam or if they are getting different spam? This is probably hard to calculate since you need to really look at the body of the spam.

Thanks!

I signed up for a shiny new yahoo account two weeks ago, and didn't give the address out to ANYONE, and turned off all the marketing preferences. Yet within days spam started appearing. There's no doubt in my mind that yahoo is sharing new addresses with spammers. How else would spam wind up in a brand new account? It's a pretty unique account name too.

And their "spamguard plus" (yes I gave them $20) is worthless. I've flagged "ONLINE PHARMACEUTICALS" as spam the last five days in a row, yet it continues to show up in my inbox day after day.

Yahoo isn't trying very hard at all to stop the spam. My guess is that they're making money off of it somehow. It ain't that hard nowadays to stop the spam. I've done it on several mail servers with RBL maps, and spamassassin.

FWIW, my gmail account very rarely (practically never) gets spam.

Nick,

I understand your frustration, but I think you're making an unfair and unfounded accusation against Yahoo.

They may, indeed, be doing a worse job of filtering spam (at least for you), but the idea of such a major corporation selling addresses is IMHO just outrageous -- they'd get hammered by bad PR and/or sued up the wazoo.

So why/how are you getting such junk on a new address? Dictionary attacks. I've been shocked at how addresses I've created with a Web hosting account -- seemingly obscure addresses, not sales@ or webmaster@ -- started getting spam within minutes of creation.

I just realized I had an old Google alert on anti-spam and false positive, when I was also recently testing some anti-spam tools.

I finally settled on vqMe.com. It is comprised of suite of techniques that all work together. Recently they added some sort of smart filtering (they still show it as beta) that trains itself based on your use and the addresses in your white-list (which also builds automatically). Even thought the SmartFilter is marked beta it has worked well for me and I forward all my mail.

vqME also has a couple of really neat features (SurfMatch and SmartSubject) that also help train your experience. "I also agree with the above poster that training should be kept at an absolute minimum, I just want my anti-spam to work.

Just my 2.5-cents.
Jenn

Hi,

The one thing about gMail I would like (a new feature) is the ability to send an auto reply message when i am in vacation also to emails in SPAM folder.

When I set vacation on - the message goes out to all those emails that I receive in my inbox. The spam I receive is all from people who seem as if they want to sell the product they own or make some money.

So if I set the vacation on and place a small advertising ad they might receive it too. (At least they receive a message)... which can be profitable.

Karl

I would like to get the mail automaticaly from gmail to yahoo ( I am not able to check in all the mail id, i want together in one specify mail id)

This is an email complaint I sent to Gmail in June of 2006:

I got a gmail account about a year ago and never used it, never gave the address to anyone. I decided to start using it again, went into my inbox today for the first time in a year or more, and saw 60+ spam emails! Most of them were already in the Gmail Spam folder so I just deleted them all. But still! I find this shocking. I have two Yahoo email adddresses - that I HAVE given out to numerous people and websites - and I hardly ever get spam at those addresses! I check those accounts maybe once a month and there will only be 1 or 2 spams in there at most. How could I possibly have more than 60 spam emails in my gmail box when I've never given this address to anyone? If you figure that over a year, that's an average of maybe 6 spams every month. That's triple what I get at yahoo with two accounts that are widely used. Are you blocking any spam at all? If so, you're letting WAY too much crap through.

****

A followup email complaint to Gmail, sent a minute or two after the one you see above:

This is an addendum to the message sent just a minute ago. WOW - I just went in to the spam folder and saw the message that 30-day old emails sitting in the spam folder are automatically deleted. So the 60 emails I saw in my spam folder this evening were just from ONE MONTH??? That's nearly 60 times as much spam as I get at my widely-used yahoo addresses! I would like to start using gmail more, but I have some serious doubts now.

****

From then until this day (Jan 10, 2007), I have not used the gmail account. I checked it today for the first time in 6 months, and there were about 10 spams in there. My Yahoo accounts usually have zero spam, other than marketing emails from places where I already order stuff on a regular basis. No bizarre companies I've never heard of and no porn email either! Once in a while a spam may come in for one of my Yahoo addresses, but Yahoo puts it in the spam folder so I don't have to deal with it other than to click "Empty" next to the spam folder.

It's bizarre that some of the folks writing in this forum seem to have had the opposite experience from me. I wonder why that is? I wish my gmail account would deal with spam better - I would like to use it. But I just can't justify it.

Interesting that you say G-mail is in the middle, with Yahoo being the worst and SpamCop is the best.

I must get 10 Spams in Yahoo daily even though Spam Guard is ON but G-mail catches EVERY Spam coming in. I have NEVER had a Spam get into my In-box with G-mail but at the end of the month I have maybe 500 Spams in the Bulk folder.

Any thoughts ??

I find myself in the exact same boat as "LM" above and I need some advice. I set up a Gmail account over a year ago, have never used it and have never given the address out. I decided that I'd like to start using it but when I log in there are 256 spam emails in my spam folder! And those, apparently, are just the ones in the last 30 days!

How did these spammers get my info if I haven't given it to anyone? Does that mean that Gmail has made this address public somehow?! Help!!! I don't want to give up the email address...what can I do?

I LOVE how there's no way to ask Google these questions....

I have read most of these post, and I still do not know why my yahoo email has so much spam mail, I keep marking them span, and get more spam from the same person even if I do not open the email. I have had no spam on my gmail address, Today I have deleted over 50 spam emails already from my yahoo account. I have had a yahoo account for about 15 years, It is getting to be more and more spam all the time, where can I contact someone at yahoo who can tell me why the spam filter does not work at all.

Yahoo has merged with AT&T and forced their SpamGuard Plus on my account. It considered MOST of the mail that I received in last 3 weeks as Spam. No wonder I had not heard from friends. So today I sent an email to myself and did not receive it. Apparently their spam filter does not recognize its own sender. Their foreign help is another joke. The chat tech at who-knows where brought up the account but could not provide info on how to fix the dam spam filter. anyone have a good fast ISP?

I find these results quite intriguing. I have Hotmail, Gmail and Yahoo email accounts. I gave up on Hotmail as my primary account due to the large volume of spam. Gmail is pretty good with Spam. But I LOVE my Yahoo account because it's spam filter is very accurate. I have been on Yahoo for a few years. I get about 50-75 messages a day. It is very rare for a spam message to get through, maybe 1 a week. For False Positives, I get 1-2 a week, but since I only get about 6-7 spam messages a week, I can live with that.

I would like to know what there spam filter is doing so i can mimic it on a in house mailing system so we don't get so much spam. Right now we have so much spam it makes looking at your email a head ache. Any ideas on how to find this out so my filter is better. Thanks