Search engine relevancy test - Part 2 - Google vs. Yahoo vs. MSN

My first search engine relevancy test took some flak for not being scientific enough. This time around, I've set some strict guidelines that should make it a bit more fair to all the search engines. Yahoo won round 1, but I only tested one keyword and used only my interpretations as scoring guidelines. Here are the scoring guidelines for round 2:

1. Only sites returned on the first page of search results will be used (as opposed to the top 2 pages in part 1).
2. A result that directly links to content that fits my search objective will receive 3 points.
3. A result that links to a page that, while it may not exactly answer my question, has the information on its site, and it can be found quickly will earn 2 points.
   
4. Results that are Wikipedia entries, Yahoo Answers, About.com, etc. will receive 1 point. The reason for this is because these sites are so well known that if a user wants to search for the answer on them, they can go directly to those sites.
5. Any result linking to a manufacturer or retailer/wholesaler that does not provide the information I'm looking for will not receive any points.
6. Results returned to directories will receive -1 points.
7. Completely irrelevant results will receive -2 points.
8. Results from the same website will only be counted once.
   

Keyword #1 - "Fix a lawn mower engine"
The goal is to find instructions on how to fix a lawn mower engine. The keyword is long-tailed and fairly precise while the results might be pretty broad considering the number of different models and brands of lawn mowers out there. Note - I'm looking for answers on the internet with this keyword, and not for a book to purchase that helps me out.

• Google: 14 points
• Yahoo: 9 points
• MSN: 9 points

Keyword #2 - "XBox 360 game cheats"
Pretending for a second that I'm a 14 year old XBox 360 fan boy, and I want to find a website with cheat codes for lots of games. The results should have a good variety of XBox 360 cheat codes for a good share of all the XBox 360 games on the market. If the initial page that pops up is selling product or giving reviews, I'm going to give it zero points. I want cheat codes for this keyword. If I can't find cheats 1 click away from my landing page, I'm moving on. I'll be awarding three points to sites that have actual cheat codes, or a list of cheat codes on the landing page.

• Google: 16 points
• Yahoo: 19 points
• MSN: 22 points

MSN had an obvious advantage, considering that Microsoft owns both MSN and the XBox brands, and the results showed. One thing's for sure, there is WAY TOO MUCH advertising on video game sites. My eyes hurt.

Keyword #3: "Buy a used laptop"
I need a laptop, but can't afford a new one (okay, I know that with the price of new laptops at ridiculously low prices, it would be stupid to purchase used - humor me). The results that link directly to a listing for a laptop for sale will get the full 3 points (including manufacturers). A result that doesn't directly sell a laptop on that page, but has used laptops for sale somewhere on the site gets 2 points. 1 point will be given to sites that may not sell laptops, but link to other sites that sell laptops. Sites giving advice, reviews, or information on purchasing used laptops will not be given any points.

• Google: 13 points
• Yahoo: 10 points
• MSN: 14 points

I should note that while MSN had the most points for this keyword, 2 of their results were awful. One actually sold used laptops, but it was a Pakistani website selling to Pakistanis. Doesn't do me a lot of good. The other was a website with a directory like structure that didn't give me any information about what it actually did. There were some links to used laptop sales, once I dug into the site, but a normal is long gone before they reach that point. So even though MSN had the most points, it could have done better because I deducted two points for the crappy results.

Totals:

• Google: 43/90
• Yahoo: 38/90
• MSN: 45/90

Surprisingly, MSN edged out Google for the most relevant results in this test. But because the XBox 360 keyword gave MSN an edge, I'll withhold judgment until part 3. Stay tuned...