The Bizarre Wisdom of Crowds

At the spring STM meeting (that's the International Association of Scientific, Technical, and Medical Publishers) in Cambridge, Massachusetts this week, University of Chicago Law School Professor Cass Sunstein explained the somewhat counterintutive results of the accuracy of group predictions. For example, where more than 50% of a group has some knowledge about a topic, as the group gets larger, the average of the individual's predictions will approach 100% accuracy. Where fewer than 50% have the necessary expertise or information, the average prediction will likely fail. Examples of successes in this area include Oscar winners, presidential elections, the number of jelly beans in a jar, and horse races. A notable failure was predicting Bush's Supreme Court candidates.

Perhaps even more interesting is that prediction markets are more accurate than surveys, perhaps because people have to "buy into" them, so a self-selected group of people who think they know something participate, increasing the chances of hitting the greater than 50% expertise level.

And surveys are more accurate than deliberative processes, because people, even if they are not informed, are making their own predictions and not being swayed by group dynamics.

One of the surprising side effects is that boards of directors tend to be most effective when they are contentious! In such cases, presumably, the board members continue to think for themselves, despite the strong opinions of others.

A caveat of these markets is that they only work where the outcome is ultimately measurable. Companies are beginning to use prediction markets internally for sales forecasting and validating product launch dates.

This one one of several examples from the meeting of a valuable talk outside the typical scope of scholarly publishing presentations.