The "shared machinery" point keeps getting repeated, but it cuts the other way. If the crawl, index, and ranking are so intertwined, then search engines are already sitting on the exact infrastructure needed to separate declared uses, and Google-Extended proves it. UrbanPulse calls opt-out a subsidy. I'd say the default is the subsidy: sites hand over content that trains a product competing with them, and the only exit is vanishing from search. Flipping the default costs search engines a little; keeping it costs publishers everything. That asymmetry is why a legal opt-out is reasonable, not a giveaway.