Lessons in decision-making: Confident people aren't always correct (except when they are)

Graeber cowrote the paper, which was published in American Economic Review in July 2023, with Benjamin Enke, an associate professor in economics at Harvard University, and Ryan Oprea, an economics professor at the University of California, Santa Barbara.

How does one measure confidence?

In the first phase of the study, the team invited more than 2,000 people to perform 15 classic cognitive bias tasks, including:

•     The “knapsack problem”—a strategic reasoning challenge that invites people to fill a bag with the right combination of weights to maximize the load without breaking it.
•     A coin flipping task that demonstrates the “gambler’s fallacy,” the faulty tendency to think that previous events influence a future random event.
•     A bidding task to show how people tend to pay more for an item in an auction than it’s worth because of emotions and other factors, a bias called the “winner’s curse.”

Through this effort, the researchers collected more than 70,000 decisions.

In the second part of the study, the test subjects participated in simple auctions, betting, and committees in which they could bid, bet on, or vote for the accuracy of their own solution to each task. These decisions served to find out whether the aggregate outcomes that emerge from people self-selecting into participating more or less aggressively in an auction, market, or voting process are better or worse than the average of everyone’s individual decisions. How robustly people participate was strongly predicted by people’s confidence in the correctness of their answer.

Whether auctions, betting markets, or committees created improvements was strongly predicted by the degree of the “confidence calibration”: are subjects with the correct answer more confident than those with incorrect answers, equally confident, or even less confident? When the right people are confident, they dominate the outcomes of markets or organizations, which is beneficial overall. But when the wrong people are more confident, wrong ideas dominate aggregate outcomes, making things worse overall.

Also read: Too many choices: A model to predict open-ended consumer decisions

That manager who loves to talk

While Graeber and his coauthors called for more research on meta-cognition, their most important finding has immediate implications: People are often poorly calibrated about whether they are right and wrong, and knowing when to pause and question your intuition or the bravado of a particularly overzealous employee might be a critical soft skill. But how does a manager find those right people and allow their decisions to be influential?

First, assess the calibration of an employee’s confidence. “Do those people who appear most confident also make the fewest mistakes? Do mishaps often happen even though the people involved seemed highly confident in what they did?” asks Graeber.

Then, evaluate whether, among which people, and in which tasks surprises happen.

“Let’s suppose a corporate decision is made, and it turns out badly. An undesirable outcome will be more surprising to the more confident people; people who had low confidence in the outcome to begin with wouldn’t be surprised,” says Graeber.

“The exact same argument goes for a surprisingly good outcome: If everybody were highly confident that things would go badly, but then they unexpectedly turned out much better, people should be surprised. Surprises are a sign that confidence is badly calibrated.”

Graeber and his colleagues say that further study of the calibration of confidence may be of “first-order importance” for understanding behavioral economics’ influence on social science.

“Although behavioral economists have put great energies into studying how nudges, frames, familiarity, and learning influence biases themselves, we know next to nothing about how these same drivers of choices influence meta-cognition,” they write.