It's been fun to watch the evolution of Gmail's spam filtering. In the early days, it was laughably bad. Even the most basic spam, the stuff that SpamAssassin would score in the 20+ range, would make it through.
But today things are quite a bit different. Based on my experience, the spam detection and filtering in Gmail is as good as Yahoo's SpamGuard. Though I don't regularly use Yahoo! Mail, we do have the benefit of SpamGuard on our corporate mail servers so I've developed a subjective notion of how much spam sneaks through both systems. At this point it's a draw. In any given day, I get maybe 15 – 25 spam messages on both systems. And I suspect they're both using virtually all the same techniques anyway.
But all along, I've been popping my personal email into Thurderbird from my own mail server, which runs a combination of spam-fighting stuff. And just the other day I realized how much less effective it is. Even after training Thunderbird's spam filter for months, the combination of SpamAssassin and Thunderbird are far less effective. I still end up with 75 – 100 spams a day in Thunderbiard, when Gmail is knocking out most of 'em.
The only logical thing to do was take advantage of Gmail's spam filtering and secure POP access.
Gmail makes it easy to turn on POP access to your inbox and even gives you smart choices about what to do with the Gmail copies once you've popped your mail:
- keep Gmail's copy in the Inbox
- archive Gmail's copy
- delete Gmail's copy
That, combined with the fact that you even get secure (meaning encrypted) POP access and the ability to use Gmail as your SMTP relay (once you setup SMTP AUTH in your mail client) tells me one thing:
The folks behind Gmail are trying to give everyone the sort of power and flexibility that used to come only with running your own mail server.
The results so far have been impressive. I'm spending less time dealing with the crap that used to end up in my inbox. Gmail is usable at home, on the road, and at conferences with unsecured networks.
This all makes me wonder if it's worth it for smaller organizations to bother running their own mail servers anymore. If Google offered small business mail the way Yahoo does, there'd be some serious competition in the market and it'd make a lot of people's lives much easier.
For the last 10 years I've helped run the various services for WCNet, a community ISP in northwestern Ohio. We have roughly 15,000 users and get a lot of spam. Even with greylisting, which helps a lot, we're still dealing with a lot of spam. (Postgrey kicks ass as a greylisting implementation for Postfix.)
Amusing side note: WCNet used to stand for "Woody County FreeNet" but the notion of a FreeNet is pretty old, and the service has grown to cover much of northern Ohio and southern Michigan. So we just call it WCNet these days. :-)
Posted by jzawodn at January 02, 2006 07:17 PM
Couldn't agree more. But Gmail's been more effective for me than it seems to have been for you. I get at most 1-2 spams a day now, down from 30-50 when I used SpamAssassin / Thunderbird. And not a single false positive. I'm finally to the point where I no longer need to check Gmail for the fpositives. And it's free.
Best aspect of it - going to the spam folder to read the spam recipe ads!
A nice review and "how to" Jeremy. I've now forwarded to gmail some old email addresses that were getting slammed with spam and it seems to work, in most cases, as a good filter, allowing me to keep the old addresses active without floods of spam coming in. Interestingly though I can't get one address to work in this fashion - in fact it was initially forwarding to the wrong gmail user even after many checks on my settings. Never figured it out.
I do a similar thing that I wrote about here http://jroller.com/page/thuss?entry=i_want_my_email_address
However, I forward all my mail through gmail and then onto my colocated Linux server so it really is like my personal spamassassin not to mention that it's a great backup of all my email when I accidentally delete a message.
Yes, Gmail's spam filter HAS dramatically improved, hasn't it? I even sent the team a kudos e-mail about that, especially since I ragged on 'em mercilessly earlier on about how bad their filtering was :P.
With that said, one thing that's kept me from just porting over all my Gmail is their annoying way of letting folks customize the "from" envelope. It still says "[mygmail address] on behalf of [the-address-I-want]" when people view my mails in Outlook. Blah :(
I certainly don't mind if my gmail address info is included in the headers, but in the actual visible from part... that's just confusing and annoying!
For alot of my mail, it goes from Gmail to Yahoo to the Mail program in my Mac at home. Zero spam ever makes it from Gmail to Yahoo to home but 2-3 per day go from Yahoo to the Mac. The other goofball trend I've noticed is the insane amount of Asian-language junk mail to my Gmail account.
I've been doing the same thing for about a year now. I send all my mail to my Gmail account and POP it into Thunderbird from there. You missed a couple of awesome points where Gmail kicks ass:
* It *automatically* puts mail you send through their SMTP server in your "Sent Mail" in the web interface.
* Any mail you send through the web interface will automatically be downloaded via POP next time you POP.
When I was first setting things up, I was going through the motions to add an automatic Bcc to my Gmail account from Thunderbird, assuming it would do the normal stupid thing. I was pleasantly surprised when my test mail showed up in my Sent Mail automatically.
One thing that annoys me: I wish the Labels information was available via some X-* header in the POPed mail, so that I could set up the "hard work" of the filters just once, and do the filtering in Thunderbird on the Label. (Or, alternately, maybe write a Thunderbird plugin that understands labels?)
Rock on, Gmail.
I don't do POP, but I do run my own mail server. What I'm thinking is sending email via gmail (ie. sender -> myserver -> gmail -> myserver -> myinbox).
Mail.app gets rid of 99% of my spam (about 1 slips through every 2-3 days, so higher than 99%), but that's at the client end, and it'd be nice to avoid that email ever reaching my laptop.
Woah -- WCNet! Back in high school (some 10 years ago now) I was slightly active on Grex(.cyberspace.org) and M-Net(.arbornet.org). I recall seeing WCNet users around those places. Yay for southeast Michigan freenets!
I do the same as Joe - I have a few old email accounts set to forward all mail to Gmail, where about 95% of the spam gets filtered out and the rest gets trashed by Thunderbird's filters. So far, no false positives out of several thousand spam emails.
It's meant that I can keep old accounts open (in case something important comes through) without being swamped by spam.
I use gmail then POP it from Thunderbird, and I have to say I'm not such a big fan of the Gmail approach. Both systems seem to do as well as you teach them to do, by marking spam for Dr. Bayes.
But the lack of folders in gmail means I can filter good mail in, so when I go a few days without cleaning out my gmail box it's very painful to sift through the bad mail.
With Thunderbird I have filters that put mail from people in my address books, mailing lists, etc. into a folder of clean mail. Then it's pretty easy to sift through my inbox to rescue the few non-spam messages (and add them to my white-filters), then select all and spam the rest.
With gmail, all my good mail is mixed in with the spam, so it's harder to avoid marking good mail as spam.
Gmail's spam filtering has never been as bad as Yahoo's, in my experience. Yahoo fails to filter so much spam, and mis-classifies so much legit email as spam, that it is unusable as an email client. Of the 25 most recent emails in my Yahoo inbox, 19 are spam. That goes back to Dec 18. There are 150 messages in my bulk folder for the same time period. 3 of those are false positives. That's 120/145, or 83% of spam being successfully filtered. And 3/11, or 27% of legit email being filtered. The 83% is really bad, but a 27% false positive rate is unbearable.
Gmail has been much, much better. Granted, my gmail address hasn't leaked out as much so it gets much less spam to begin with. But I'd say it's been about 95% successful at filtering the few spam messages I do get, and not a single false positive.
I've been very impressed with gmail's spam filtering too (I forward all my email from my server to a gmail account both as an offsite backup and because I'm too lazy to set IMP/Horde or whatever). My current spam favourite filtering solution is CRM114 (from http://crm114.sourceforge.net ). Once trained, it's been extremely accurate for me - far more so than Thunderbird's filter and better even than the SpamAssassin and blocklists setup at my workplace.
I have to agree with Doug on this. It's very easy for me to ID spam in my yahoo box because I use it only to receive yahoo group emails. A lot slip through, and the address is unpublished (by me, at least). Also, I've had a number of false positives, and I wasn't really checking that much.
With gmail, I have received few spam messages in the first place, and zero false positives.
As for my personal mail, I used to get over 60,000 spam messages per month, raw. I have a combination of personal procmail filters I built up in the pre-spamassassin era, spamassassin itself, and greylisting. Greylisting was the one that really cut it down. I don't use any RBLs because there was too much of a tendency to blacklist larger blocks to try to force ISPs to remove spammers by causing collateral damage, and I couldn't have legitimate email discarded. Greylisting has occasionally been inconvenient, but between it and the other measures, I'm down to maybe 5 per day - and a lot of them fit the same pattern, so if I wrote my own quickie procmail filters for them I could cut that at least in half if desired.
Well the oly weakness I've found in using a third party client like Thunderbird or Outlook is the lack of a reliable "Send as" field. Outlook has it in Exchange mode, but not POP/IMAP mode.
In Gmail proper, you can send as an email address (my personal) and so Emails would go out with
From: aaron@xxxxxxx.com
Reply-to: aaron@xxxxxx.com
But email sent from Outlook or Thunderbird go out as:
From: mygmailusername@gmail.com
Reply-to: aaron@xxxxxx.com
I still use the third party method (Outlook 2003) as well as Gmail proper but I just wish the third party vendors would securely figure that tid bit out.
I have also been really pleased with the Gmail/POP/SMTPAuth setup - much better than the creaky IMAP system I had running before.
However, I am plagued with false positives, so very surprised to hear all you people saying it isn't a problem.
Any invoice, registration info, automated mail of any kind goes straight into my spam folder - which is OK because I'm often expecting it and can fish it out.
But I get false positives from people I've actually sent mail to in the past (surely candidates for whitelisting?) and anyone who mails me for the first time is more than likely to end up in the spam box.
This means that I have to trawl through the spam box regularly to rescue the FPs from the 100s of spams and inevitably I miss a few.
I wonder why this isn't a problem for others (I don't deal in porn or mortgages by the way!).
I don't deal in porn or mortgages by the way!
We few, we happy few.
Henry V
Who paid you for this post?
It get about 100-500 Spam mails a day and maybe 1-3 make it past the Thunderbird spamfilter per week!
I have no idea how good or bad gmail or yahoo are but Thunderbird rocks!
Have you noticed that Gmail is technically still in beta?
I consider that a branding bug which doesn't significantly affect the product.
The single thing that stopped me from moving over to GMail in the initial days was its Spam filter. Nobody else could've been as much worse. Two mails that I'd forwarded from my own Yahoo! account (that id is in my GMail address book already) were sent to Spam. The only "spamworthy" attribute that I could see in that mail was the "Fwd:" prefix in the subject line. And several other genuine mails were sent to Spam folder even as they matched my filters for labeling. That put an end to my GMail usage. The foremost thing that I want from an email service provider is reliability, UI is among the least.
How is Google's spam filter working for the people who use it? How many spam emails can someone expect out of 1000 emails received?
Please reply.
Thanks in advance!
All great, so long as you don't mind GOOG having copies of all your mail, in perpetuity, I guess.
Google has released Gmail for your domain. It's the natural competitor to Yahoo Business Email - and it offers more powerful features at less cost.
Someone is sending spam via my virtual dedicated server and I can't find where or how to stop it via my Plesk Panel which is the only control panel I was offered by Godaddy.com They will only help me for $300
I want to find a way myself - anyone know how?
Hi all =)
It's true that Gmail does a great job, but sometimes an alternative solution is needed for corporation purpose because they won't change their habits to adapt to the Gmail system.
I finally found an anti-spam software that seems to work pretty effectively, and with a nice GUI and wide range of plugins it makes it one or the best of the anti-spam solution, and plus it is free. You can find it at www.spamihilator.com. In plus it is sponsorised by A-Squared antivirus and antispyware.
Try it and tell me what you think of it (maybe there a better solutions but for now I am at 98% filtering with no false positives now that it is well configured since 2 weeks of using).
Cya =)
"Software is like sex; it's better when it's free." - Linus Torvalds
Gmail is great, i use it but you forgot something
it's keeps a copy of EVERY mail and analyze it,
so i'am mooving to my personal mail server in order to keep
privacy rights
Before you run off and implement this (like I did), you should know of one serious issue. You can't POP a gmail account from more than 1 computer! Which, unfortunately, is really lame.
Until they allow for that possibility, using a POP client on a laptop and desktop to access your forwarded account, only gets you a collection of not the most recent emails not yet received by any computer.
Bummer.
You mean the gmail pop daemon doesn't follow RFC1939 and after a RETR it automatically issues a DELE? that doesn't sound ... right.
What makes you think that? GMail can keep all messages you've poped, archive them, or delete them.
I'm using ASSP (http://assp.sf.net) as an SMTP proxy for my mail server and for several companies I provide services to. And I have to say - one configured it is unbelievable accurate. Way better than my GMAIL box.
Check it out...
Our server is Bluhost. We have seen a large increase in spam over the past year. They use SpamAss. along with a new SpamFilter (not sure what it is). We still get 20-30 spams a day (each person on domain). I tried using fowarding everything to gmail, but seemed to have an increase in false positives. Had to stop using it because of this. Anyone else had problems with FP's on gmail?
>>You can't POP a gmail account from more than 1 computer!
Add recent: before your gmail user name, as in:
recent:john.doe
This will allow you to check from more than one computer.
I use gmail to filter about 6 different POP accounts and I find that it misses roughly 10-20 spams per 1000.
I think Gmail stinks. I just set up an account to do as you said, and Gmail's spam filter filtered a message that came from team Gmail. And it was a legitimate and important item of mail.
When I send mail from my Outllok via POP it automatically sends it to my Blackberry - like the SMTP server is sending a copy message out.
Do you know how I can stop this?
>>Add recent: before your gmail user name, as in:
>>recent:john.doe
>>This will allow you to check from more than one computer.
Smithee, that's a hot tip! You just saved me from extended misery and getting all my gmail emails duplicated every time I reset the "Turn on POP for all messages (even ones that have been downloaded already)" option in the Account Settings for gmail. It used not to be a problem - or at least I only noticed it being a problem recently. Anyway, Tbird is downloading some more messages for me now that I put recent:myusername@gmail.com into the Username box in Server Settings. Let's hope they're the right ones!
Hi, how can I configure gmail to filter emails sent to my personal email - i.e. user@domain.com filter through user@gmail.com then forward to user@domain.com?
Do I need to set it up in my Outlook 2003?
Thanks
Speeds
the spam on gmail all seems to follow the same pattern: gibberish in pseudo-proper sentence structure.
how come i never get this stuff on my hotmail? i get more sophisticated advertisement on my hotmail.
it's almost as if gmail writes these ads just to show that it's filtering something.
That's Great,
Since this post ranks so well for "cannot send from Outlook using Gmail" why don't you post a link to the fix here.
See You at SES
Thanks,
I've just moved our corporate mail over to gmail, initially the free 'standard edition' of the google hosted apps. I am seriously impressed. My main reason for moving was that my personal gmail seems to be extremely good at spam filtering, and our ISP's spam filtering solution (spam assassin) just wasn't doing the business anymore.
Overnight, I have gone from about 30 spams a day that reach thunderbird to zero (ok, only 2 days, but pretty stunning). No false positives yet. It has correctly identified 120 spams and 55 good messages. That is pretty amazing.
Contrast to my hotmail account which is deluged by the most obvious obnoxious spam and rarely puts anything in the junk folder and the choice between the Google and Microsoft offerings for corporate hosted mail was a no-brainer.
And with Gmail's 2gb storage, i can keep copies of all my mail on the account, as well as downloading it to Thunderbird.
I agree completely that with free services like this, there really seems no reason for any small businesses (even tech savvy ones) to run their own mail server. The price for using google is just some subtle sponsored links on the right of messages when viewing mail in the webmail. But no nasty footers or promo text on outgoing mail sent via the web or from the smtp server.
google's spam filtering WAS close to 100%, since about 2 weeks it is letting me down with many many pdf's & zip'ed BS.
I would guess that SpamAssassin works well if you keep training it so its bayesian stuff works (i.e. train it on false positives). Google maybe does teh same thing, but has teh entire community retrain it. Dunno what google uses. I've also heard it's about the same as yahoo in spam detection, but I guess yahoo doesn't allow for POP3 access :)
Is it possible that Gmail will blacklist a server if a user is forwarding all of his mail to Gmail in order to filter it for spam first?
I have a user who receives an enormous amount of spam to one account, so he forwarded the account to a Gmail address, which filters the email, and then forwards it to another email address which he checks on his desktop.
My concern is that all of the spam that Gmail is detecting now shows our server in the headers. Will Gmail blacklist my IP? That would be a huge problem, as no one on my server would be able to email any Gmail address!
I have recently come across in my own annoyance, SPAM addressed from myself to myself, so be sure to check your account and change your passwords often.
Here is your Google Anti-Spam idea with a twist:
Google as hosted spam filter:
http://www.iopus.com/guides/gmail-spam-filter.htm
Works well!
Lately I am so disappointed in my gmail spam filter I was doing a search to see what I could do to improve it. I'd say 3/4rds of my email currently making my in box is spam. It was good until a few months ago and now it is absolute garbage. I get 30-40 copies of spam with identical messages even after I've reported that kind of message as spam 5 or 6 times.
The gmail spam blocker is garbage. I am so disappointed because I switched to gmail as my main email and if it doesn't improve I'm going to be switching email again.
zero false positives and one in three thousand spam in inbox thankyou gmail!
I AM SO GLAD to benefit from the gmail spam filter! It is the most rational and satisfactory solution one could wish! I do have a question, out of nagging curiosity: how can it be that some 98% of the messages that I see assembled and discard in the spam dept. are encouraging me to increase the size of my penis and so please my sexual partner? Is there somewhere in the world where Kathryn is a masculine name? The other 2% is about getting rich quick, but that's not gender-specific, I guess.
In other words: why such a prevalence of one type of spam message? Not complaining, I just whisk 'em away anyway, thanks to gmail, but CURIOUS.
Best wishes,
Kate
bug : gmail inbox does not equal to email downloaded
I am using Gmail to filter the spam
my gmail inbox has 63 messages, my spam and trash are empty
I did not used my Thunderbird-webmail extension for gmail in 3 months
every time I click on the Get New Messages button
250 spam start to downloaded from Sept. 19 (last day I used the webmail extension)
How do I eliminated those thousand of spam and download ONLY the 63 messages my Gmail inbox has?
thank you
It seems the Gmail spam trap is not working as well anymore, spammers have figured out a way to go around it.
I've experienced a lot of spam coming through to my real mailbox, while they are in Gmails spam folder, they are still forwarded to real mail. Which is strange and started some weeks ago.
Well, I've just switched my domain's email over to Google Mail and the spam protection has been 100% with no false positives or negatives (switched a week ago). For the last few years I've been using SpamGuardian which was frankly rubbish, never seeming to learn from corrections to either false positives or negatives. I'm sure that spammers will up the ante at some point but I have more confidence in the google team finding a solution and deploying quickly than I do in a software vendor releasing a patch and my domain hosting company installing it.
You can also check new SPAMfighter for thunderbird: http://www.spamfighter.com/thunderbird_spam_filter.asp.
Would avoid the Gmail hassle
@bk: Yes, I believe this can prejudice GMail against your server. I am forwarding a bunch of mail for a friend with a high-spam-volume address to his new GMail account, and now if I create a new GMail account, mail from my server to that account is immediately considered spam.
My painful workaround is to send a message from my GMail account after the first time I contact a new GMail user from my domain. Once the GMail user marks one message as "not spam", subsequent messages from the same sender seem to be accepted.
I wonder if spammers who spoof my domain are the problem. My smtpd (postfix) doesn't seem to have an easy way to reject a message to my domain that pretends to be from my domain. Perhaps I can achieve it by using SPF, but that'll be a pain.
Why in the world would you willingly use Gmail's spam filter? You enjoy having real emails marked as spam?
gmail ought add
* captcha for unknown senders
* full imap support
* build up content for filter import/export
* user option: discriminate against non-SPF or non-DKIM servers
additionally:
revealing my gmail account in x-headers when I send mail as another address is a big no no and MUST cease
Sometimes inbuild spam filtering doesn't seem to work. In that case gmail provides a good way to get rid of spam. You can actually alter your gmail id before submitting to doubtful sites.
Here is how you do it.
> code..future
-1
Plus addressing does *little* to reduce spam. Since there is no MTA rejection action available to filtering you can but discard messages.
yahoo mail offers TRUE DEA: SpamGuard
mails send out from webmail as a SpamGuard address DO NOT reveal the underlying yahoo identity
..much UNLIKE privacy-leaky gmail
gmail needs a few antispam labs choices:
A user controlled SpamAssassin lab for gmail would be sublime.
Also allow user controlled filters for MTA rejection.
Add to gmail filer actioins “mark as spam”.
full implementation of TDMA for gmail would be wildly entertaining!
gmail filtering is pathetilame. We need be able to search:
any header (contents)
any header (exists)
wildcards