(Don’t) Bundle + Save | Google’s Multi-Use Indexing + AI Crawler

Google currently uses a single site crawler for both indexing + AI training, meaning that if you want your content to be found on Google search, you have to opt in to allowing their AI to scrape it. For companies + websites that are trying to avoid AI or are simply fed up with giving their content away for free, this bundled model is no longer cutting it. 

We’ll walk

Cloudflare hosts roughly 20% of all the websites in the world, and they’ve had it with Google forcing their hand with their “all or nothing” setup. They’ve given Google an ultimatum. If Google doesn’t provide a way for their content to be discoverable without having to give it away to Google’s AI for free by September 15, they’re prepared to walk away from Google search. 

USA Today Inc. + Beehiiv have joined Cloudflare, threatening to delist themselves from Google by blocking all “multi-use” crawlers from their content if they don’t work out a licensing deal soon. Google hasn’t agreed to anything yet, but competitors are starting to take note. 

USA Today Inc. has started to turn to other sources like newsletters, social media platforms, events, and other search platforms that *do* have licensing agreements when it comes to AI to keep their audience numbers up even as their search traffic starts to decline. Google’s commitment to their current practices gives competitors a weakness to take advantage of.

Should I stay or should I go

Mark Steinberg for Adweek writes, “​​The decision of when to make that call is mostly a matter of math: Once search traffic drops below a certain threshold, the value of appearing there becomes less than the value of withholding that content as a bargaining tactic.” Although he clarifies that the publishers would prefer Google to negotiate licensing terms over removing themselves.

Google has been King for a long time. These threats from publishers represent a threat to Google’s top position. Media companies are making models for projections of what their future would look like if they blocked Google to prepare for that possibility. While publishers don’t want to leave Google, they’re prepared to do it if their demands aren’t met. The fact that companies are making these (not empty) threats in the first place is a big deal.

decisions, decisions, decisions

This whole situation brings up the broader concept of choice as a business model. Consumer expectations are shifting, leaning towards a preference for more control and choice about content. If you ask a consumer to accept two unrelated things at once, without the option to separate the two things, the choice is a farce that doesn’t go unnoticed. Whereas before many consumers would look the other way + just accept the terms, now, they’re starting to look closer + consider more before clicking accept. 

Just like with these publishers + Google, if your audience is willing to walk away if their expectations aren’t being met, lack of choice will negatively impact your business going forward. The moral of the story? Give your customers + audience members options whenever you can. If there are multiple things you want someone to agree to, don’t lump them all together in a single “yes or no.” Break down each option + give them a real choice. 

Take TikTok for example. Back in January, they rolled out a new “local feed” and asked for permission to use your precise location using GPS. When it first came out, the pop-up message said it would now be tracking your location with no “decline” button – only an option to accept the new terms. Users even noted that if you closed the app to bypass the message, when you reopened the app, you’d be using it with the new terms regardless of whether you clicked accept or not, creating a similar “all or nothing” agreement – let us track you or you can no longer use the app. 

While it seems they’ve since added the option to opt-out of precise location (in an easier way than through your iPhone’s setting’s) and stick to using approximate location, many users were put off by this new privacy agreement, leading them to delete their accounts + never look back. 

what will trent said

For us, these decisions are important + should 100% be choices that each user can make for themselves. Why should you have to give you content to AI just so your website can be found on Google? Why should you need to give TikTok access to your location to scroll through silly videos? Proponents of ethical marketing, like ourselves, believe those choices shouldn’t be tied to one another. 

Learn more about our ethical marketing program, recreative good, here.