Google Search Snafu Can Have Huge Impact on Niche Blogs

Dear Google,
What happened? Where’s the love? You used to bring me flowers, you used to sing me love songs. You used to bring me traffic, at any rate…
Google, baby. Let me back in. Me and my pretty dumb things are shivering out here in the digital ether with no one but Ask.com and Yahoo to keep us warm. We need you honey. We miss you and the 500+ visitors a day you used to send to us. We’re lonely. Please, Google. Don’t be evil.
kissykiss,
chelsea girl

The public love letter from Chelsea Girl, who writes the erotic Pretty Dumb Things blog, was apropos just after Christmas when a Google bug made a handful of influential sex blogs “disappear” from Google search results. If you searched for “pretty dumb things,” you wouldn’t see Chelsea Girl’s blog on the first page of results. If you searched for “tiny nibbles,” you wouldn’t find the blog of the same name by SFGate columnist and sexuality author Violet Blue. If you searched for “Comstock Films,” you couldn’t find the arty adult film studio by that name. Instead, you would find less relevant sites.

So what if a few sex blogs drop down in Google search results? The problem is that with so much power concentrated in one company, Google, one small mishap has the potential to punish small independent blogs or web businesses that depend on Google-generated traffic. In late 2003, Google performed what was called the Florida Update on its search index, which caused small businesses such as FindGreatLawyers.com and Unforgettable Honeymoons to lose their ranking on relevant Google search terms. (Read about those case studies and more in this great story on SearchEngineWatch.)

In this recent case, Tony Comstock of Comstock Films noticed that his site’s Google-referred traffic had dropped precipitously after Christmas, and told blogger/author Violet Blue. She then notified Xeni Jardin at culture blog BoingBoing, who then wrote a high-profile post on the subject. Soon, Google’s head of web spam, Matt Cutts was in contact with Comstock and the blogs were largely restored to their previous place in Google searches.

Cutts later explained what happened to me in an email exchange:

This is relatively rare. In this case, a bug (plus an unfortunate side-effect that caused our internal tests to miss the problem) caused a very small number of sites (less than ten sites that we know of) to rank lower for about four days. We believe this situation is fully corrected and the very small number of impacted sites have been returned to their proper ranking. We’ve fixed the particular set of circumstances that led to this situation, and we’ve put changes in place to prevent it from happening again. We’ve also added more tests to our internal processes to spot similar problems to this in the future.

Google does adjust its algorithms to try to improve search quality, and that can cause changes in rankings for sites. Before new changes are deployed, they have to pass a number of tests. We can also look at factors such as the amount of “churn” [change in search results] caused by a particular change. In this case, the small number of sites affected meant that we didn’t detect the situation before it went live. As a fallback, we also monitor feedback from various online sources: blogs, forums, emails, etc. In a very short time, we noticed a couple blog posts, checked out their reports, and discovered the issue. After that, we were able to fix the issues involved quite quickly.

But was this really a rare case for Google? Search expert Danny Sullivan, editor in chief for Search Engine Land, told me Google often tweaks the way it indexes sites to weed out spam blogs or “splogs” that simply pump up key words and links to try to game the system so their blog is featured higher up in search results.

“In this case, it seems like Google might have been tweaking porn filters somewhat,” Sullivan said via email. “But you get spam in other areas, and sometimes tweaks seem to hit other industries. One hiccup can absolutely sink some businesses. And the lesson to take away from that is that businesses should not be basing themselves on getting free traffic from Google. Many sites learned this the hard way back during a major algorithm change in late 2003 known as the Florida Update. But then again, plenty of small businesses have continued to thrive and survive since then on Google.”

The Difficulties of Gauging ‘Good Porn’

One problem in the sex blog snafu is the nature of the blogs’ subject matter and the exploitation of sex searches online. Blogs like Tiny Nibbles and Pretty Dumb Things and ErosBlog try to give insight into human sexuality in a more artful way than the average glossy porn magazine. But so many unsavory characters are trawling the web trying to divert the huge amount of sex searches to their own businesses — even if the business has nothing to do with sex.

“There are so many people chasing the same dollars with search spam, blog spam, and the signal-to-noise ratio in the commercial Internet is so high [on sexual topics],” Comstock told me. “You type in open-minded sincere search strings about sex into Google and you get mostly garbage. Not just garbage because it’s not your taste in sex, I mean useless like a thousand windows pop up and it tries to install malware [a computer virus] on your computer.”

Comstock writes his own blog on the adult films he produces, and lately has been trying to optimize the blog for Google searches. He’s also been discussing the difficulties he’s had buying Google AdWords ads with the key word “romantica.” Google rejected his ad because it pointed to an adult site (his blog) but used a key word that wasn’t adult in nature (“romantica”), according to Google. These types of semantic arguments steam Comstock and make it difficult for Google to classify — by algorithm — which sites are for mature audiences, which ones are more arty and which ones are just spam.

Comstock told me that Google search results bring him a steady amount of traffic, and that he depends on that traffic in case of a media review. For example, after the British Metro newspaper mentioned Comstock, he got a flood of people Googling him because the paper didn’t run a web address for his site. If Google dropped his site from searches for “comstock films” then his business would be hit hard.

Author/blogger Violet Blue told me she thinks Google needs to have human liaisons to industries to minimize these types of glitches. I asked her if it was frustrating for a small business to lose its traffic from Google overnight.

“It’s more than frustrating for small businesses — it’s a death sentence,” she said. “Google has too much power…It’s something that we might not have ever seen in history. It’s not just small businesses but for someone like me as a blogger, I might be naive, but I have this principle of believing in self-publishing on the web. I count on a democratic dissemination of information rather than from the channels we’ve always been getting it from…When Google breaks, suddenly that’s gone and I don’t have that principle to rely on. We won’t have that diversity of voices if we can’t find anyone.”

Google Improves Feedback Loop

In this case, however, Google did act when it found out about the bug, and rectified the problem within a few days. Google’s Cutts was visible in comments he posted on sex blogs and in his highly public email conversation with Comstock — that Comstock posted in full on his blog.

Here’s part of what Cutts wrote to Comstock:

I worked on Google’s SafeSearch filter years ago, so I know that distinguishing between the “good porn” sites compared to the “regular porn” sites is a hard problem. I used to be able to reel off names like Jane’s Guide, Persian Kitty, The Hun, Greenguy, Luke Ford, etc. These days I haven’t worked on porn-related stuff in years, so I’m less familiar with the space compared to how I used to be.

In fact, I’d be curious to hear your take on what several of the highest-quality porn-related sites would be these days. I’m familiar with stuff like fleshbot.com or nerve.com, but less so with sites like tinynibbles.com or erosblog.com…I’d be curious to hear what some of the leading lights are in the porn industry these days, and I’d be able to point a few people at it to make sure that we work on distinguishing higher-quality sites from run-of-the-mill sites.

I asked Cutts if the company would consider using experts in certain industries who could help them suss out what the bad and good sites are, and help decide which sites should be penalized and which ones not.

“Google is open to using feedback from experts or other signals to improve our quality,” he said. “In this particular case, we worked with some of the sites involved to get their feedback about other high-quality sites in this industry. We’ve already incorporated that feedback to improve our search quality and testing as a result…Search engines get hundreds of millions of queries a day, and no search engine will be perfect for every query. But we try very hard, and that includes listening to feedback from users, webmasters, and the blogosphere. Based on the feedback we get, we try to improve our relevance for future searches.”

Sullivan noted that Google has improved its listening skills when it comes to indexing problems for sites, and has even set up an online repository for webmasters worried about search problems.

“Google has made huge leaps in supporting site owners, especially small site owners, over the past year,” he said. “The entire Google Webmaster Central service was created and offers tools that no other major search engine offers. There’s more Google can and should do, but I think it’s easy to forget how much they’ve done. On Christmas Day, you had at least two Googlers working to answer questions in public forums they monitor.”

Google is obviously trying harder to listen to the concerns of bloggers, publishers and shop owners online. But mistakes and glitches happen, and the company still holds vast sway over the way people find information and products online. So bloggers or site publishers will do well to track their Google-referred traffic and work as a community to publicize glitches and keep Google staffers in the loop.

What do you think? Does Google wield too much power in how people find things online? What solutions do you think would help the company combat spam but keep legitimate sites where they deserve to be in search results? Share your thoughts in the comments below.

UPDATE: A number of people have asked why there aren’t links to the sex blogs mentioned in this post. If Google had been blocking the blogs, then there would have been links included. But because anyone can easily find the
blogs through a search on Google, PBS.org felt it was not necessary to include the links here and risk offending some readers who might not expect to find links to explicit sites on PBS.org.

Currently, there are no hard-and-fast rules for the content we link to at MediaShift, and it’s a delicate balance of being open and also being aware of the adult nature of content we sometimes link to.

I ask that you as MediaShift readers please leave comments below explaining what you think the link policy should be here and elsewhere on mainstream
media sites and blogs. Should we link to explicit material and how should that be handled? Should we include a warning before the links? Which links are OK
and which are not? Your thoughts would be appreciated and I hope to return to this issue in a more in-depth way on the blog. PBS editors, who are involved in this issue, tell me they are very much open to your suggestions.