A few days ago I was snooping around Digg when I noticed a popular submission titled The Difference Between Digg and Reddit. I clicked, eager to learn, and was presented with an image juxtaposing two very distinct flavors of user-submitted comments surrounding the breaking news of Tony Snow’s death. The first comments shown at Digg offered generic words of respect that you might expect to hear about a public figure that passed away. The top comment at Reddit, however, was a bit more candid to say the least.
The discussion that followed ranged from folks saying “maybe I should join reddit…” to full blown conversations about what works and what doesn’t work in user-moderated comment systems. I have since been thinking a lot about what comments on media items are meant for, what makes a good system, and what tradeoffs exist.
The value of comments
I wasn’t around when the grand forces decided that comments were a must have for digital media outlets, but the conceptual benefits are pretty clear. From what I understand they provide an opportunity for readers to hear what others have to say, contribute their own two cents, show off their wit, and at times simply vent.
It seems like most organizations think of the feature in those terms, i.e. as a way to engage the individual and get some feedback. While that is nice, the real benefit of a comment system is the way it can drive collective intelligence. To put it in less geeky terms, comment systems should take the voices of a large group of people (the collective) and present what they have to say in a way that spreads knowledge and facilitates wisdom (the intelligence).
In particular a good comment system will…
- Harbor insight – chances are there is an expert on the subject who is reading the article. It is also possible there are readers who are aware of valid points either on their own or via another source.
- Reveal related information – there are probably resources out there where readers can learn more. Users will link to or otherwise share those resources.
- Challenge assumptions – if there are false claims, incorrect citations, rhetorical gaffes, or questionable perspectives in the article or in other comments then someone will identify and address the potential mistake.
- Answer questions – one article can rarely cover everything. Let the hive mind tie up loose ends.
- Entertain – in the same way that there is room for funny headlines and Jon Stewart there is room for funny comments. It makes the whole news experience more enjoyable and increases the odds that people will pay attention.
Common pitfalls
The magical transition from noise to information is where most comment systems fail, but there are lots of reasons why. Here are a few that I could think of:
- Ignorance – there will always be people who are outrageously closed minded or blatantly incorrect. I can’t tell you the number of times I have wanted to slam my head against the wall when reading comments on mainstream and local news sites alike. A system needs to account for this and prevent these types of comments from dominating.
- Group think – It is hard for groups of people to tell the difference between “wrong opinion” and “other opinion.” This means that systems have to prevent the larger group from taking over the conversation and squelching valid alternate points of view.
- Information overload – Once there are more than about 20 base-level posts it gets hard to take everything in. A good comment system will mitigate this effect and make sure that every useful contribution reaches somebody.
- Inability to respond – without conversation there is no way for individual ideas to mix and match. This means that users sit around isolated and unchallenged. As a rule of thumb, strictly linear layouts like the kind you see here on the IdeaLab won’t work in settings with an average of more than four or five comments per item.
A Possible implementation
Like any other piece of technology, there is no “best system” here. A local paper that has never had more than 3 responses on any article might not care much about implementing a robust collective intelligence suite. That being said, a good system on a decently trafficked site will get used.
I want to throw out a few ideas that might help the situation. Some are quite old, some are new, but none are used everywhere just yet.
- Threads – threads are standard on social media sites but they really ought to be standard everywhere. Users should always have the ability to reply to someone directly and have their replies appear in context. This naturally organizes information but it also helps to contain flame wars and hot debates that might otherwise take over and stifle new thoughts.
- Rating – it is difficult to find a perfectly pure way to rate subjective content such as comments. Despite this it is a worthwhile addition even if it is only used to get a general picture about where community stands on the issues.
- Randomization – you can’t view all of the posts all of the time, but by including an element of randomness when deciding on which comments to display at the top a system gives a chance to the ones that would have otherwise been lost in the crowd. This will help to prevent the lazy voting feedback loop where the top posts are rated higher simply because they are seen by more people.
- Categorization – it is hard to compare comments. Some are informative, some are funny, some are insightful, some are insulting, etc. In order to account for this you can allow users to categorize them into predefined buckets. This could be combined with rating by having users select reasons why they rated something up or down.
- Burial – some comments simply aren’t fit for print. Automatically hiding terrible comments helps clear away some of the noise, but how can you identify them? Ratings may be prone to group think but through categorization it becomes possible to see the difference between bad and unpopular.
- Geotagging – I haven’t mentioned geotagging in a while, but comments should absolutely be linked to location. This provides some degree of context to individual statements and also gives a sense of scope when looking at conversations. Furthermore you can apply some of the targeting practices that I’ve talked about in the past in situations of information overload.
Now that I’ve said all that I charge you with the task of going out and fixing the way comments work on your sites. Just let me know if you need any help!
Thanks Dan for your lucid writings. I’m getting the impression that you’re developing a alternative journalistic code for collective intelligence.
You’re absolutely right about collective intelligence. Human beings are social creatures, and much of our understanding comes from social/conversational interaction.
But the single most important factor leading to quality conversation has nothing to do with technology. It’s a shared sense of purpose.
This is missing from most general news sites, which typically do not even have a declared mission.
Some of the tools you mention — like threading, comment ratings and categorization — may actually interfere with the user experience on a typically cluttered, ad-supported news site. Threading only makes sense with high volumes of comments, where it helps with information overload.
Dan,
Thank you for writing so much on this issue. As we work on some new elements for our site, I’ve benefited a lot from what has been written here.
Keep up the good work!
Dan:
I was amused by your face palm imagery in response to ignorant posts.
However, I do think you need to distinguish between ignorance and the more insidious bigotry, hate speech and even the more cynical trolling behaviors that exist.
I been in a couple of the leadership development programs over the years and one session stands out in my memory. It was titled you are what you were when. It talked about the development of personality in childhood and the basic reality that unless the ideas we gained as children are challenged as we grow as adults; those beliefs will never change.
Hence, folks who were children during the depression were savers and pretty tight with money; those in the baby boom era are consumption oriented.
The point is that kind of ignorance … the ignorance of never having these kinds of ideas challenged … is subject to the dynamic of challenge, crisis and reassessment. That is a good thing.
Recognize also that some ignorance is more reflective of a cultural than rational thought.
One might class those who spout racial epithets as representative of skin head KKK cultural norms.
While I abhor hate speech, there are as many ways to show ignorance in this social area as there are to leave your lover.
In terms of the broadest definition of journalism I find it objectionable not to present the most accurate reality.
The plain fact is there are ignorant, inarticulate, poorly written people out in our communities whose ideas are ignorant but whose right to hold them is unquestioned.
There are two points here. First is a rather technical point and that is the data storage capacity for a site like mine is for all practical purposes, infinite. I can publish everyone’s comments … which is something one could not say about a newspaper with its finite number of column inches.
Since we can publish everyone, we then have the philosophical choice of whether we want an inclusive society or not. I personally vote inclusive.
That understanding (or call it a choice) comes from being the parent of a child with autism.
It would be such a joy for me for him to concoct a single sentence verbally or on paper. I honestly wouldn’t think that sentence would express any magnitude of intellectual brilliance but I do believe what ever it may be will be a compelling thought for him and I would be quite proud for him to express it in the same medium just like everyone else.
Why? Because not only is he a part of this greater society but, whether you like it or not, you need to know he’s there.