If you're a techie, like me, you've probably noticed that Microsoft has been making a lot of noise about competing with Google these days. Yahoo, too, has been active. In general, the search engine space is starting to heat up again after several years of google-dominance. And if you're running a website, for yourself or for someone else, you've probably seen the Microsoft search bot in your server logs. At least, you have if you pay attention to your server logs and Microsoft can find your site yet.
Now, don't get me wrong, competition is a good thing... but competition from Microsoft is not. We all know that competition from Microsoft involves things like "cutting off their air supply" and aggressively wielding their desktop muscle and huge cash reserves to get a foothold in the new market. Google's an established player, but it's facing a fair amount of competition and the current IPO can perhaps be taken as a hint that they are vulnerable -- that is, sufficiently in need of cash that it might affect their ability to respond to new players.
So it looks like Microsoft is trying to extend its overbearing self into the search engine space, and the target at the top of the list is Google. Now, as a Linux user, I understand two things about this situation: first, Microsoft is not my friend; and second, Google has been a very good friend.
So here's what I have to say: from now on, the Microsoft search bot will be specifically excluded from my web sites via the robots.txt file. The following entry should accomplish this:
User-agent: msnbot
Disallow: /
What that means is that I won't allow Microsoft's search engine to index my websites. Google can, yahoo can... but Microsoft can't. It's the website owner's version of an invisible boycott. And if enough people do it, the Microsoft search engine will find itself unable to search large segments of the web. And that means that people who notice this won't want to search using Microsoft's search engine.
So I'd like to invite anyone who cares about keeping the search engine market out of Microsoft's greedy little paws to take the same step with their own websites. If you do this, please drop a comment on this post so I know that you are doing it. And with luck, maybe we can cut off their "air supply" this time.
If you're thinking it's counterproductive to exclude a search engine from your website, then think about what will happen if you don't: imagine a web without Google. It's not pretty, but that's not the worst of it. Imagine a web where Microsoft decides what pages are indexed. Do you think they'll have any compunctions about developing "bugs" in their ability to index pages served by Linux? What about .php pages... maybe they'll just get indexed less often than the .asp pages. Or maybe the .asp pages will have custom parameters to make indexing easier, in the Microsoft tradition of hidden APIs. It doesn't take lots of imagination... just a knowledge of Microsoft's history.
The window of vulnerability is short. If we allow Microsoft to become an established player in the search engine space, they will command enough mindshare to make this impossible. We need to cut them off before they wield their market power to force their way in, and that means now. Any later will be too late to matter.
