When I wrote When Better Isn't Good Enough, I had both features and design foremost in mind. I wasn't thinking as much about relevancy because I've seen enough data from blind comparisons to know that they're often too close to call. Tim and I talked a bit about relevancy in the comments and in a follow-up IM conversation.
The Search Engine Experiment is public blind test that's trying to sort out who is better than who. Seeing the results prompted Seth to write Can more than 60% be wrong?
About 65% of those tested said that Yahoo or MSN was the most relevant.
I won't go into the flaws with this method, since that's really not the point. He goes on to say:
Which reinforces my point that Google isn't "better" for most people if "better" means more relevant or deeper. Google is better because it feels better and quicker and leaner and easier to use. The story we tell ourselves about Google is very different, and we use it differently as a result. Think about that the next time you insist you need a "better" formula or a faster server or a stronger first baseman.
Music sounds better through an iPod because we think it does. Design matters. Stories matter most of all.
Bingo!
That's really what I had in mind in the "better isn't good enough" post. But Seth said it far better than I did.
Posted by jzawodn at November 25, 2005 07:15 PM
Hmm, I feel mixed about this one.
I agree that what people _feel_ when they use a service is paramount. It's some mixture of a bunch of ineffable factors, and the users aren't responsible for sorting them out --- they just know what they like.
On the other hand, the cool thing about blind tests is that it subtracts out the influence of the brand. If you've never done this, try the blind-taste test thing with a (willing) friend who's a (beer, scotch, cola) snob. I've been a beer snob in my day, and I was able to make some crude distinctions once when I tried it: stout vs. lager, sure, but subvarieties of Czech pilsners? Not so much.
Anyway, the interesting thing about the blind test is that, if you can't tell your favorite thing from the others, it means either that you're under the power of the brand, _or_ you're under that power of the intangibles of the user experience (the color of the label, the way the cap comes off, the usual serving temperature).
Seth's post munges the brand and the user experience together. Whether this matters depends on whether you believe that (in the long run) brand loyalty is correlated with how good the underlying thing is. Myself, I believe that brands are some sort of combo of current marketing and past product quality, and that when the product quality equation shifts, the brand will too, even if really slowly.
Whoa - holy dubious stats batman ...I certainly agree with your key points - that perception is important, Google is overrated vis a vis other search, and that Yahoo should shoot for "much better" and not "a bit better" results to win the search war. But statistically Seth seems to be making a very dubious interpretation of these results, which suggest Google is still the top engine by a significant margin. Applying his faulty logic the headline could also have read "Yahoo only serves the most relevant result one third of the time".
These results should be interpreted as indicating Google still wins the search wars, not the other way around.
Well, I did say "I won't go into the flaws with this method". :-)
seth godin is beating this "story" thing into the dirt. everytime i see him quoted/cited, he's beating everything in sight into the "story" pattern.
once again, the market has digested that google search is better, and the market doesn't care about a better search right now. google shares are rising because adsense is monopolizing the distributed ad network, not because of search.
To be honest, I gave yahoo search try from time to time... it happeneds once in month or two but every time search results were like "alltheweb" at the time 2002... no porgress, or et least not visible progress.
Hang on a sec - these stats *clearly* show that Google's results are more relevant than either Yahoo or MSN Search. To say otherwise is totally misleading. While stories and brands may count (Pepsi frequently wins in blind taste tests against Coke, but Coke sells better), these stats don't show that at all. I have a lot of respect for Seth, but he's clearly trying to make the stats fit with his preconceptions, and they simply don't.
You have a good point with the "better isn't good enough" argument, but in this case Google really is better. These stats don't support your argument at all.
I agree. Google has certainy tricked people into thinking that they are the be all and end all to not only searching, but technology in general. The greatest trick the Devil ever pulled, was convincing the world he didn't exist.
What argument? I'm not arguing anyhing. I'm simply agreeing with Seth--that the experience and brand matter.
Jeremy,
What I'm saying is that these stats don't show the triumph of branding, they show the triumph of better technology. Seth has tried to argue that Google's relevance is no better than that of Yahoo or MSN, and that we prefer Google based on the UX and branding. However, the stats show something else: Google is better in terms of relevance.
More on the stats here:
http://being-reasonable.com/index.php/weblog/comments/godin_google_and_gosplan/
Well, I hate to say the obvious but... that's obvious.
He just decided to interpret the stats a different way (Seth likes to do things differently) but he's not *wrong*.
Anyway, I suspect that if brnad and UI were factored in, the numbers would be quite different. That's the point, I think.
I think we'd agree that if brand and UI were factored in, Google would do even better than the other two - the gap would be wider, since Google has a better UI and a stronger brand. But I still find your title misleading: "Because it Just Seems Better" - the fact is that Google doesn't "just" seem better - it seems better and *is* better, in every sense.
Oh, I see.
I picked that title because it hints at how the "feel" of products can be as important than their actual quality. But I can see how you'd read it to think that Google only "feels" better.
Don't worry... I'm not that brainwashed. :-)
Every coin has two sides, no?
I made the test, I (blindly) picked Yahoo! as the most relevent on the first chance. Then I got to see the graphic of overall results and this is what I got:
Google: 42%
Yahoo: 32%
MSN: 26%
Applying Seth's faulty logic we could also say "Almost 75% of users said that either Google or Yahoo were the most relevant" or we could say "68% of users said that Yahoo had the worst results". The fact that the other two overall added more than Google alone doesn't mean Google isn't better. I totally agree with Pete Cashmore.
On the other hand, I liked the idea of a blind test. I won't deny that if I (or many other blind testers) would have known the brand in advance, the gap would have been wider. But as someone already mentioned, Google has a better branding, and has been giving more relevant search results for more years now. In fact, this is what prompted Yahoo and MSN to build/improve their own search engines.
As for me personally, I might be little bit biased toward Google, but I have thoroughly and regularly compared search results for my blog in both Y and G, and the G is always better (more results, more up to date). By the way, I never taste the M.
Oh, and I forgot to mention, that the testing tool should allow to rank the three competitors, not just let you choose which one was the best. An "always second place" competitor is certainly better than one that made a few first places and the rest as third place. Also what happens if two (or the three) search engines gave very similar relevant results in a given query? You still have to decide for one.
You know, there is a little bit of truth here, but as someone who, I like to think, dispassionately looks at software, I think it misses something about the Google v Yahoo thing.
Now, I have come back to Yahoo for a lot of reasons over the years. However, my Search Engine of choice has typically been HotBot, replaced by Google.
There are some very real reasons for this. If you are looking for general information on a "topic", yes, Yahoo and MSN are more "Relevant". However, most of the searching I do on a day to day, hour to hour basis is much more specific than that. If you are trying to run down a technical problem, Google simply does a better job of dealing with "quoted" searches that contain nonalpha characters, getting specific matches in sources that would not score high on "relevant" in most other contexts (bboards, etc) and being able to flip right to a usenet search on a tech issue is still useful -- it may be dead, but Usenet is still the greatest repository of answers to esoteric technical problems out there.
The real question is, what do you mean by "better"? More successful as a company? Better for the user, even if the user does not agree? What the user perceives is better?
Look at the Apple Newton -- Apple discontinued it in 1998, and yet there are die-hard fans of it (well, the article was in 2002, but the friends I have that swear by it still do). Does that mean the Newton is better than the clio, the razr, the palm, the handspring?
Generic drugs are a great way to look at this argument, too -- I can't tell the difference between NyQuil and generic NyQuil (CVS's brand), but my partner can, and he refuses to take the generic stuff. The ingredients are the same, but perhaps the recipe and/or concentrations are different. I've noticed with generic foods, too -- they're different. Sometimes I'm fine with it, like with pretzels, other times, I refuse to eat generic Cheez Doodles.
Generic cereal is the WORST. It just doesn't taste the same. As someone who also has been cutting calories, I definitely have things I'll skimp on (low-fat pudding, skim milk) and things I won't (full-fat cheese for me, all the way).
Do we like products better because they're cheaper? some folks swear by generics. Perhaps music from an iPod sounds better because it costs $100, $200 or $400, where a portable CD player costs $20.
And really, what is "better", and how do we define it? Usage, and # of hits? $$ the company generates? The general attitude about the company? Perhaps Google is "better" because it doesn't (yet) have the scandalous past that Yahoo! does (remember when Yahoo! bought Geocities and changed the TOS, to own everyone's content?).
Perception is key. Dean lost a HUGE bunch of support because they thought he was a nutcase, because of that yelling incident. Regardless of politics, Dean lost supporters because of PERCEPTION. And in the end, he did not win the bid for presidency. Perhaps the bottom line is really all that matters? We all know that Microsoft isn't all they hype themselves up to be -- but they're still the market leader, and for a decade or so have been the industry leader. There may be regime change in the future. . .
Take Oracle and MySQL, too. MySQL now has the features Oracle does. They're in their infancy, and Oracle has had them for decades, but right now, at this moment, it cannot be said that MySQL is not a full-featured database. Regime change may be in the future, but people still have the PERCEPTION that Oracle is better because (1) the price tag and (2) it HAS been 'better', for longer.
Have I started enough religious wars yet? But that's the point -- religious wars exist BECAUSE people have different definitions of "better". Sysadmins think vi is "better" because it's always there, and does what they need. Programmers think emacs is "better" because it does the tabbing, etc that they need. In the end, they are both right. (this reminds me of the recent discussion about Windows being more expensive than RedHat -- my comment is the 5th one down, and the point is that even "expensive" is relative.)
I agree that Google's lead is not just about the relevancy of the Top 3 results alone.
The results I got when I did the Search Engine Experiment didn't surprise me:
http://www.all-about-content.com/2005/12/rate-search-engines.html (Google was not the clear winner, even though it continues to be my favorite search engine.)
For me it comes down to:
1. Brand
2. Credibility
3. Usability
...in no particular order.
Brand - MSN is a too-litte-too-late player. Until they bring something new to the table, why bother switching? They were late developing their own algorithm, they were late introducing a toolbar, they are way late on delivering collaborative Web 2.0 tools, etc, etc. Even if they are now just as good, they are seen as always playing catch up. Plus, Google still has more good will than Microsoft so they'd need to be a LOT better to make people change existing habits.
And for the record, when I did a blind taste test and realized the cola I picked was not the one I had been drinking, I switched! :)
Credibility - Yahoo doesn't even trust its own algorithm. They hand pick the top results for very competitive searches. While that actually makes the results for those searches better, IMO, it creates a credibility problem.
Usability - Let's not forget that Google continues to have the cleanest interface. However good their results are, Yahoo and MSN are perceived as having more cluttered SERPs.
As a user, I'm also interested in the overall quality of all the results on the first page, and even how relevant the ads are. It says a lot to me if the level of relevance drops drastically after the first three results.
