The idea is that publishers could use their robots.txt as a ransom note, selling it to the highest bidder -- Bing or Google. (I suspect this idea takes fire and gets repeated a lot over the next couple months.)
Google composes, what, 70% of search referrals to news sites? More? For a business that relies on advertising and social distribution, it sounds like suicide to me, the same effect as putting all your content behind a registration wall.
posted by Andy Baio at 2009-11-09 19:19:48 
Disagree!<br /><br /> <br /><br /> It's probably 40%, and the online revenue is probably 5% of overall revenue. (In other words, for all the heckling that Murdoch has gotten for his blocking Google quote yesterday -- including me! -- it really doesn't matter what the hell they do from a revenue perspective... at least in the short term.) <br /><br /> <br /><br /> But the numbers aren't what matters. What matters is that no one has every really thought of their search traffic in these terms. We've always just put the search engine in the position of power, and no one has has ever positioned robots.txt as an asset.<br /><br />
posted by Rex at 2009-11-09 19:41:39
robots.txt is not an impermeable legal wall. It's just a webmaster's wish of how he would like to be treated. <br /><br /> <br /><br /> Thus far, the good engines have respected it, but if you're being a dick, they can ignore your suggestions and crawl right through.
posted by Andy at 2009-11-09 21:36:47
Oh cmon, you know it would be a PR catastrphe if Google ignored it. That sounds like a Republican slippery slope argument!<br /><br />
posted by Rex at 2009-11-09 21:44:11
No, the slippery slope argument probably starts with Bing stealing the top 10 publishers by doing face-to-face agreements. Then, it works well enough that they open a self-serve model: "All publishers who block Googlebot, register here and we'll send you a revshare check in the mail." Then Google does the same thing. Then Yahoo does the same thing, but it doesn't work because everybody blocks MSNbot, which powers Yahoo's results. Yahoo dies. Then there are two left, Google and Bing. Each have half the world's content and search is now retarded. I build an aggregator engine that puts the results of the two parties back together again, and I steal all their marketshare before anyone can block me. <br /><br /> <br /><br /> This should happen.
posted by Andy at 2009-11-09 22:03:43
I still don't get how this would cut into google revenues/market share all that much. I get to the WS Journal/CNN/NY Times more often than not through blog links. Google would still index these right? Wouldn't this essentially just funnel more of their revenues into the hands of rebloggers/aggregators.<br /><br /> <br /><br /> Am I missing something?<br /><br /> <br /><br /> <br /><br /> <br /><br />
posted by r l at 2009-11-09 22:36:09
To put it another way:<br /><br /> Isn't this akin to covering your eyes and shouting "I'm Invisible!"
posted by r l at 2009-11-09 22:40:33
Disallowing robots from crawling is not the same as restricting indexing. When Calacanis talks about blocking via robots.txt, he probably means using a meta tag like this on all pages:<br /><br /> <br /><br /> <meta name="googlebot" content="noindex">
posted by Andy at 2009-11-09 22:50:34
Anyway:<br /><br /> <br /><br /> meta name="googlebot" content="noindex"
posted by Andy at 2009-11-09 22:52:17
years ago "they" set out to kill gawker by posting erroneous celeb sightings.<br /><br /> <br /><br /> just saying.
posted by steve at 2009-11-10 10:22:50
hmm... if use wordpress engine, we can setting it...
posted by Lowongan Kerja at 2009-11-11 16:36:16
Calacanis, at 1:43 "I don't think the publishers are aggressive and smart and gangster like that. Rupert Murdoch probably is..." <br /><br /> <br /><br /> This is EXACTLY what Rupert Murdoch said in his interview with David Speers on <a href="http://venturebeat.com/2009/11/09/murdoch-says-he-will-remove-news-corp-sites-from-google/" target="_blank">Sky News</a>. And I quote: "We'd rather have fewer people coming to our sites, but paying." <br /><br /> <br /><br /> What a concept- paying for content on the Internet! Murdoch is about to throw a major spanner in Google's <a href="http://bit.ly/4mqHT6" target="_blank">Less Than Free</a> business model.
posted by buck at 2009-11-11 17:15:05
bing could steal NYT and washington post... and still have no market share. this would just translate into more page traffic for cnn, intl. tribune. etc.
posted by andy at 2009-11-11 17:33:12
They can't do it as a group due to anti-trust laws. Bing would have to approach each inevitably.
posted by John Hall at 2009-11-11 19:27:50
I feel like there is Nash equilibrium stopping this from happening. i.e. if wash post, la times, chicago tribune, all take the microsoft bribe, nyt would significantly increase its traffic by not opting out, they would anticipate that and ask for a much higher bribe than the value of the current revenue stream.
posted by John at 2009-11-11 21:40:23
Again, I need to repeat this. As much as we like to think the new currency is "pageviews," this is still a meaningless concept without a monetization strategy. <br /><br /> <br /><br /> A billion pageviews or a million dollars -- you pick.
posted by Rex at 2009-11-11 22:01:13
Rex is right ... the scenario might play out exactly like Andy paints it, but the probability of that isn't enough amo to keep the media from jumping on the randsom/opportunity, if presented. The only downside'd be that the Cory Doctorows of the world would H8 you forever.
posted by alesh at 2009-11-11 23:52:45
blocking google does nothing without hiding everything behind a paywall. If a bing referral was the golden key and opened up the paywall then yeah it could definitely steal market share.<br /><br /> <br /><br /> I subsribe to the notion paywall or not, information is liquid and will find the smallest of cracks to flow through.<br /><br /> <br /><br />
posted by ryanl at 2009-11-13 17:58:32
think its more than 70%. This is EXACTLY what Rupert Murdoch said in his interview with David Speers
posted by bukmacherzy at 2009-11-15 07:08:49
I'm with ryanl on this - I don't really see how this could ever be foolproof.
posted by John at 2009-11-16 17:21:16
Why not, blackmail was always good business. http://lifehacker.com/5409467/youtube-will-soon-block-access-from-set+top-devices
posted by T-Zombix at 2009-11-20 18:19:28
