The Problem with Web Measurement, Part 2

This is the second part of MediaShift’s special series on web measurement. In the first part, I looked at the problem of inconsistent traffic numbers from panel-based measurement firms such as comScore and Nielsen//NetRatings, and the push by the Interactive Advertising Bureau (IAB) to standardize measurements. This week, I look at the philosophical differences between publishers and advertisers on measurement, consider a new solution by startup Quantcast, and dream the impossible dream of one universal online metric.

With all the web traffic numbers and metrics floating around — page views, unique visitors, time spent, sessions — it’s a wonder that anyone can agree to a simple advertising sale on a website. Complicating matters is that the advertising world is used to traditional measurement services such as Nielsen’s TV ratings that rely on usage by controlled panels of people. Online, those panel-based services can rarely gauge traffic on sites with less than 500,000 unique visitors per month.

Meanwhile, publishers use web analytics firms such as Omniture and WebTrends to directly measure traffic to their servers, and only use panel measurements when they have to compare their traffic to that of rival sites. That leaves a deep chasm with publishers and direct measurement firms on one side, and the marketers and panel-based firms on the other. Web analytics gurus slam the panel measurements as being old world and obsolete, while advertisers say they can’t trust data that comes directly from publishers.

IAB senior vice president Sheryl Draizen believes it’s time for advertisers to shift their thinking and start accepting direct-measurement numbers.

“There’s this philosophy in the entire media industry that it’s always better to use independent third-party numbers, because they’re independent and don’t have a vested interest,” she said. “I would argue that that’s not the case, because no one’s independent and everyone has a vested interest. We also have to change our thinking because we have a unique medium that could give us more accurate numbers than we have ever had before…I would challenge the agencies and marketers to stop thinking that the only numbers that are valid are coming from a third party. It’s just not the case in our industry. If there’s a certification process against those numbers, there’s no reason those numbers can’t stand.”

And those direct measurement numbers are only increasing in importance as publishers get used to seeing detailed traffic reports with the rise of free-of-charge Google Analytics and the coming of a free Microsoft web analytics product.

Web analytics software helps publishers see how traffic goes through the site, allowing them to optimize the site with better navigation. It differs from panel-based measurements that usually provide more demographic data on the audience of sites. Rather than use server logs, web analytics firms are now largely using JavaScript code on sites to measure traffic, because very few people turn off JavaScript code the way they delete cookies.

Chris Parkin, senior director of product and solutions marketing at Omniture, told me he thinks people get too caught up in “popularity” metrics rather than figuring out which parts of their site are the most engaging.

“What we do is help our media customers understand content velocity and affinity to help them sell their advertising space,” Parkin said. “What are the areas of content on your page that are helping people dig deeper into the site and view more? Time spent on the page could be relatively small, particularly if you have compelling headlines that influence people to dig deeper into the site. Using conversion metrics instead of popularity metrics is something we’ve been educating our users on for years.”

Many big publishers, such as Washingtonpost.com and Wall Street Journal Online, depend on Omniture for their web traffic data, and even a smaller operation such as the Tri-City (Ore.) Herald finds the service’s numbers to be reliable.

“[Omniture] doesn’t count bots, spiders, etc., that might otherwise show up in server logs,” said TriCityHerald.com’s online director Andy Perdue in a comment on MediaShift. “One issue is if a user has JavaScript turned off, but my experience has been that this is less than 1% of users. I have found Omniture to be extremely accurate, even if it isn’t always easy to drill down into some of the stats I’d like to see.”

While many of the web analytics firms use similar methodologies to measure traffic, there are still no standards or audits to certify the numbers they get. I talked to people at Omniture, Google Analytics and WebTrends, and none of them would commit to audits until after the IAB issues standards related to unique visitors and other traffic measurements.

Avinash Kaushik, a consultant, blogger and author of “Web Analytics: An Hour a Day,” hopes the IAB will help standardize the way firms measure unique visitors and visits, but thinks there’s less of a chance web analytics firms will submit to audits as the panel-based firms have done.

“The IAB will be able to convince the old world panel-based people to do it, because their data is so bad,” Kaushik said. “It’s less likely that it will happen on the web analytics side…It’s the bigger customers who will put pressure on web analytics firms to standardize on a couple metrics. I’m not sure that will come from a standards body. There’s one more dynamic that is underestimated. As I talk to the vendors, I find that they genuinely believe that their methodology is better than the others.”

‘Audacious’ Quantcast

While direct-measurement and panel-based firms battle it out to become the standard bearers for online metrics, a San Francisco startup called Quantcast is trying to combine the best of both worlds. Quantcast takes panel data it gleans from Internet service provider traffic, and matches that up with direct-measurement data it gets from web publishers who load its tags on their web pages. Not only does Quantcast combine panel numbers and direct measurement, but it also offers up all the data for free to publishers — and the public.

I visited Quantcast’s office to talk with CEO Konrad Feldman, who was eloquent about how his company might solve the web measurement problems rampant in the marketplace, but tight-lipped about how Quantcast would make money. Feldman said his firm would measure traffic for any site — from the largest media site to tiny blogs — for free, later figuring out how to charge publishers or advertisers for matching them up.

“Online, you need a different system [for measurement] because of fragmentation, there are so many different media choices that could be very useful for advertisers,” Feldman said. “With the multitude of niches, people self-select whatever media interests them, and because so many people have been empowered to publish content, that means you have a proliferation of niches. The problem is measuring and understanding those different niches. The panel-based approach has difficulty when the media choices become large relative to the sample size. We’ve seen that in television, where the accuracy of ratings have been questioned, especially in specialized programming like local TV.”

Not only does Quantcast aim to help measure traffic along The Long Tail of niche content; it also wants to build what it calls an “audience search engine” to help advertisers find the right audience online to market products.

“If you think about search being how consumers find advertising; for branding, what you really need is an audience search engine,” Feldman said. “It’s the way that advertisers and their agencies could go out and systematically find the audiences that are most important to them. There are other challenges with the way that agencies spend money with television. If you want to make similar buys online, you don’t want to make that difficult, you want intermediaries who can aggregate audiences for you.”

Bill Cook, senior vice president of research standards at the Advertising Research Foundation (ARF), was impressed with the ambition of Quantcast to change the game of web measurement with so many entrenched players.

“When I first saw these guys from Quantcast, I thought, ‘These guys are so audacious, they can’t be real,’” Cook told me. “But you meet them and they are so sincere and hard-working and even smart. I hope they are as smart as they think they are because it’ll be good for the world. Either the major players are going to get it right, or someone else will come along. There is an incredible amount of entrepreneurial activity. We get somebody in here once a week with a new way to do it. We’re in a temporary group grope, but we will get very smart a lot faster with all this energy and investment money out there.”

While Quantcast has signed up some big sites such as Fox, Gawker Media and Answers.com, the service hasn’t had much visibility yet with many other large media sites. I also heard complaints from publishers who say that Quantcast publishes inaccurate panel-based numbers for sites in order to push those sites to join their publisher network for more accurate direct-measurement numbers.

Gus Venditto, editor-in-chief of Jupiterweb, won’t load Quantcast’s code on his sites because he doesn’t want anything to slow his servers even a small amount. And Venditto doesn’t trust Quantcast’s panel-based numbers that are made public on its website. “Quantcast seems to be just guessing at traffic numbers for sites who aren’t using their web bugs,” Venditto said via email. “Perhaps it’s an educated guess, but the numbers they have as estimates for our sites are off by orders of magnitude.”

Quantcast says it’s not twisting arms to get anyone to use its service, and that its panel-only numbers are as inconsistent as those from other panel-based firms.

“We make our best efforts to show useful data on all properties,” Feldman said. “Panels are good for getting broad visability and a directional gauge. We want to provide a useful and broad service, and we hope that one day we’ll be able to provide quantified data for all sites. But, until that time, we, like everyone else, have to rely in part on panel-based estimates.”

One Unified Measurement?

Advertisers and marketers have been frustrated for years about the way measurement online is so different from the simplified metrics they’re used to getting with traditional media. Some of them dream of a day when there will be one unified measurement for online ratings that everyone can agree on — and that can be directly comparable to ratings in offline media. And eventually, if TV and Internet delivery are combined through wide broadband pipes into homes, won’t that create a more universal way of delivering ads as well?

As nice as it sounds, everyone I talked with — publishers and marketers and measurement firms alike — don’t think it’s possible to have a unified traffic measurement with the current complexity of the web and all the activities going on there.

“[Having one measurement] would oversimplify things, and wouldn’t provide one marketing person or advertiser with the information they need,” said Kate Downey, director of audience analytics and insights for The Wall Street Journal Digital Network. “It would be too much mushed together of everything you want to know. I like ‘visits,’ because it strips out things you don’t need to know. There are so many ways that people interact with your site that means something to some but not to others.”

Brett Crosby, senior manager for Google Analytics, noted that a unified measurement would be hard-pressed to cover the wide variety of websites and their functions.

“My personal opinion is that it’s going in the opposite direction [from a unified measurement],” Crosby said. “Like with FeedBurner [which measures RSS subscriptions], blogs have a different success metric than a retail site. And if you’re doing brand advertising, it’s not necessarily about trying to get people to buy something from your website; it’s about getting their brand of toothpaste into your head. It’s about applying the right metric to the right advertiser depending on the goal you’re trying to achieve.”

Someone who understands brand marketing as well as traditional media buying is Scott Symonds, who is executive media director at brand marketer AKQA. Symonds says he remembers the complaints about audience measurement in the TV industry 10 years ago, and notes how drastic the difference is in measuring a broadcast audience vs. an interactive one.

“When you buy a spot on a TV show, you get the whole reach in one shot,” he said. “If you buy one banner on Yahoo, you don’t get the whole reach of Yahoo, you’ve only gotten one person. The two-fold benefit of some kind of [universal] metric like that would be having a universal measurement that works across sites and also goes back to more traditional media reaches…It sounds like a noble idea but it’s a bit of a reach.”

Nielsen has tried to meld its online and TV metrics to come up with a “fusion” of data to understand how online video usage affects TV habits. Mainak Mazumdar, vice president of product marketing and measurement science for Nielsen//NetRatings, told me Nielsen has made some efforts to bridge various offline and online media, but he remains personally skeptical that one overarching measurement would work.

“It’s going to be extremely difficult to get to one measurement that will give you all the answers,” he said. “And I don’t know if it’s a worthy goal because the web is many things put together, and various measurements highlight its strengths. We want to see the web as both media and marketplace, and for media we have certain requirements, and on the market research side we do things differently. I don’t know if it’s a good idea to put everything under one grand scheme.”

Perhaps the only way a measurement firm like Nielsen can figure out what people are doing online is to physically look over their shoulder and note their media usage. That was actually done in an innovative (and expensive) research project in 2005 called the Middletown Media Study at Ball State’s Center for Media Design (CMD). Researchers literally followed people around during their waking hours to see which media they consumed and noted it on special electronic devices. Nielsen’s Council for Research Excellence recently collaborated with CMD using a similar method to capture exposure to cross-platform video, in and out of the home.

“What we would like to do is measure 100% of the consumers and that’s our goal, to measure everything Mark Glaser does each day, what he watches on TV, what he sees online, that he reads the San Francisco Chronicle every day,” Mazumdar said. “You might take a bus downtown each day and see certain billboards and advertising. My goal as Nielsen is to measure your activity; people are more important than technology habits. We would need to put a meter on your TV and on your laptop and your mobile phone. So we need to find people like you so we can make marketing more effective, and you would get more targeted ads so you don’t get bombarded with ads and spam.”

That Orwellian vision of knowing everything we do online is chilling, and it’s difficult to imagine the public accepting that level of surveillance. More likely, marketers will have to accept the fact that web measurement takes time, patience and education, and those who understand the most relevant metrics will be the ones who can connect an audience to products and services that interest them.

What do you think? Would standard measurements for unique visitors and page views help bring consistent traffic numbers? Should web analytics firms submit to audits of their methodology as panel-based firms are doing? Does Quantcast offer a hopeful alternative? What web measurement numbers do you trust? Share your thoughts in the comments below or in the Your Take question.

Photo of Avinash Kaushik by David McClure, and photo of Konrad Feldman by Jennifer Woodard Maderazo.