The Wisdom of Crowds and Bean Jar Experiments

17 September, 2013 at 14:19 | Posted in Economics | 7 Comments

“Who Wants to Be a Millionaire?” was a simple show in terms of structure: a contestant was asked multiple-choice questions, which got successively more difficult, and if she answered fifteen questions in a row correctly, she walked away with $1 million. The show’s gimmick was that if a contestant got stumped by a question, she could pursue three avenues of assistance. First, she could have two of the four multiple-choice answers removed (so she’d have at least a fifty-fifty shot at the right response). Second, she could place a call to a friend or relative, a person whom, before the show, she had singled out as one of the smartest people she knew, and ask him or her for the answer. And third, she could poll the studio audience, which would immediately cast its votes by computer. Everything we think we know about intelligence suggests that the smart individual would offer the most help. And, in fact, the “experts” did okay, offering the right answer—under pressure—almost 65 percent of the time. But they paled in comparison to the audiences. Those random crowds of people with nothing better to do on a weekday afternoon than sit in a TV studio picked the right answer 91 percent of the time.

WoCNow, the results of “Who Wants to Be a Millionaire?” would never stand up to scientific scrutiny … As it happens, the possibilities of group intelligence, at least when it came to judging questions of fact, were demonstrated by a host of experiments conducted by American sociologists and psychologists between 1920 and the mid-1950s … A classic demonstration of group intelligence is the jelly-beans-in-the-jar experiment, in which invariably the group’s estimate is superior to the vast majority of the individual guesses. When finance professor Jack Treynor ran the experiment in his class with a jar that held 850 beans, the group estimate was 871. Only one of the fifty-six people in the class made a better guess.

There are two lessons to draw from these experiments. First, in most of them the members of the group were not talking to each other or working on a problem together. They were making individual guesses, which were aggregated and then averaged … Second, the group’s guess will not be better than that of every single person in the group each time. In many (perhaps most) cases, there will be a few people who do better than the group. This is, in some sense, a good thing, since especially in situations where there is an incentive for doing well (like, say, the stock market) it gives people reason to keep participating. But there is no evidence in these studies that certain people consistently outperform the group. In other words, if you run ten different jelly-bean-counting experiments, it’s likely that each time one or two students will outperform the group. But they will not be the same students each time. Over the ten experiments, the group’s performance will almost certainly be the best possible. The simplest way to get reliably good answers is just to ask the group each time.

Advertisements

7 Comments

  1. Just two comments. First, I would think the quality of the response really depends upon the randomness (experience?) of the group. Secondly, the “millionaire” example is possibly skewed by the selection of the audience question. As a contestant you would want to ask the audience a pop culture question over a technical question. And I believe that translates to most groups.

  2. This is extremely misleading. The author says this:

    “And, in fact, the “experts” did okay, offering the right answer—under pressure—almost 65 percent of the time.”

    That makes it seem like, for example, if you get a question on a 1970s musical that you then call an “expert” on the field. That’s not true. The option on the show is called “phone a friend” and that’s ALL it is. It’s just a random friend that is not an expert at anything that you have standing by on the phone.

    Now, the friend — who is not an expert — gets it right 65% of the time. Not bad. And that goes up by 26% if you poll a crowd — who have, on average, the same expertise in the field as the friend. Again not surprising.

    This is NOT the victory of the group over the expert. But rather the victory of the group over the individual. The author goes on to say:

    “Now, the results of “Who Wants to Be a Millionaire?” would never stand up to scientific scrutiny…”

    Nor would his highly misleading book. Although it might, with its pseudo-democratic fables, make it to the bestsellers.

    • Philip, read Treynor’s Market Efficiency and the Bean Jar Experiment and I think you will see more clearly why a Post Keynesian might find this line of thought interesting …

      • It looks interesting. I’ll check it out, but it doesn’t change the fact that, as it is written, the above example is misleading.

        By this guy’s logic if you ask a crowd of 100 random people a question on, say, the structure of the current account or the premisses of Bayesian probability, they will have more of a chance of getting it right than you or I (provisionally taking us to be “experts”). That’s just rubbish.

        A crowd will guess better than an individual if the individual is not an expert on the topic. If the individual is an expert, the crowd will likely lose.

  3. […] that many people, acting independently, can often acurately estimate something like the number of beans in a glass jar. This is erroneously applied to groups that are in communication with each other such as the […]

  4. […] due, it was James Surowiecki in his book The Wisdom of Crowds who best made the point that groups as a whole know better than individuals, even if no one single individual knows anywhere near as much as the crowd collectively […]


Sorry, the comment form is closed at this time.

Create a free website or blog at WordPress.com.
Entries and comments feeds.