How experts make complex decisions

Using an immense dataset of more than 200 million moves from an online chess platform, the researchers draw novel conclusions about how chess players find their way through the fog of complexity. They find that slowing down helps everyone, but that masters of the game benefit considerably more from extra decision time than less-expert players. And counterintuitively, they show that adding a mediocre option into the mix can actually be worse than adding a bad one.

How Chess Players Make Decisions

Chess has several features that make it perfect for studying byzantine decisions.

First, the quality of each move can be objectively ranked. Unlike picking an insurance plan or a marketing strategy, where accurately measuring and ranking alternatives is possible only with a well-functioning crystal ball, certain chess moves can be clearly identified as part of a winning strategy: these moves will (if followed up by subsequent optimal play) guarantee a win, no matter what one’s opponent does. Other moves can similarly guarantee a draw or would result in a loss.

(It may seem bizarre that the famously cerebral game can be boiled down to predefined winning and losing moves. But this was proved more than a century ago by the German mathematician Ernst Zermelo. “He basically said, ‘Chess isn’t an interesting game,’” Salant explains, “‘because either white has a winning strategy no matter what black is doing, or black has a winning strategy no matter what white is doing, or both of them can force a draw.’”)

Yet despite this ability to objectively evaluate every move, the nature of the game often makes it hard for even expert players to discern a good move from a bad one. “Even though there are famous theorems that say that at any point in the game either white or black has a winning strategy, you often cannot find those strategies because the game is so complex,” Salant says.

Consequently, the standard model of decision-making, in which a person evaluates every option one by one and then selects the best alternative, does not typically apply. The sheer number of available moves frequently makes it impossible to evaluate every option one by one (especially in speed chess, where players are limited in how long they can spend strategizing).

Also read: The long and short of it: What makes CEOs decide?

Thus, players typically consider only a small subset of all possible options and pick the first one that they consider good enough—that is, the first move that they believe will produce a win—a strategy that economists call “satisficing.”

Given that many real-world decisions also require a satisficing approach, Salant and Spenkuch thought that studying how players move on the chess board could help tease out the dynamics of complex decision-making.

A “Monumental” Database of Chess Moves

Their data came from Lichess.org, one of the world’s largest online chess servers. The database encompasses nearly eight years’ worth of games, capturing hundreds of millions of moves by hundreds of thousands of players, who range from hobbyists to grandmasters. Importantly, the data detail not only which piece moved where, but the full configuration of the board at each point in the game.

The researchers focused on what chess players call “endgames.” These are scenarios with a limited number of pieces left on the board in which both players are trying to achieve checkmate. In particular, endgames with fewer than six pieces were useful because they have been “solved” using computers—that is, for any configuration of six pieces on the board, computer scientists have cataloged all of the possible moves and how they fit into a strategy that can guarantee a win, a loss, or a draw.

But getting the data on actual moves was only the beginning. For each endgame, Salant and Spenkuch had to determine all of the possible moves that the player could have made, and whether each of those was a winning, losing, or drawing move.

“That was a monumental effort,” Spenkuch says, requiring 600,000 hours on Northwestern’s supercomputer cluster. The resulting dataset, containing 4.6 billion hypothetical moves, spans terabytes.

Salant and Spenkuch also obtained data on the number of moves required to obtain the guaranteed win, loss, or draw, a figure known in the chess world as “depth-to-mate.” They considered this metric a handy proxy for the complexity of a move (the logic being, it’s more difficult to correctly discern the value of a move that requires nine additional steps to obtain a win or loss than a move that only requires two steps.)

With the data wrangled, the researchers could begin testing some predictions.

Also read: Only fools rush in: Pitfalls of hasty problem-solving

Who Makes the Right Decision in Complex Situations?

They started by looking at how the degree of complexity affects which options get chosen.

The standard economic model of choice predicts that more-complex winning moves (that is, those that require more intermediary steps to obtain a checkmate) should be picked more frequently than less-complex winning moves. Salant and Spenkuch found that this was not the case in the chess data: The higher the depth-to-mate of a winning move, the less frequently it was chosen, presumably because the path to victory was fuzzier. “At least in this setting, the standard model is off in an important way,” says Spenkuch.

The researchers also found that a player’s skill level affected decision-making in nuanced ways. They were able to distinguish between “titled” players—those recognized as among the world’s best—and untitled players. Perhaps unsurprisingly, titled players were less likely than untitled players to make a mistake (i.e., picking a losing or drawing move when there was a winning option). This gap became most pronounced when the available winning moves were highly complex, suggesting that the best players’ expertise benefitted them most in especially tricky scenarios.

Players’ skill levels also influenced how they performed under time pressure. Predictably, titled players made fewer mistakes than untitled players regardless of how much time they were given to choose moves. But the difference was smallest in the most frantic varieties of speed chess that concluded within minutes. In slower matches, when decision time was much less limited, untitled players were considerably more likely than titled players to choose a bad move.

“It’s the titled players that benefit more from extra time to think,” Salant says.

Finally, the study looked at how changing the mix of available options affected decision-making, by comparing how players performed when they had slightly different sets of alternatives available. For instance, some players had to choose between 3 winning moves, 2 drawing moves, and 2 losing moves; other players had to choose between 3 winning moves, 3 drawing moves, and 1 losing move. Even though both groups have the same number of winning moves and therefore the same chance of accidentally stumbling into a bad strategy, the researchers found that the second group does significantly worse: substituting one losing move with a drawing move made players roughly 14 percent more likely to make a mistake, on average. The reason: “Drawing moves are more easily mistaken for winning moves,” Spenkuch explains.

Also read: Swept by intuition: Bad decisions of good leaders

What’s more, players did not seem to struggle with “choice overload” when facing a large number of possible moves: they actually made fewer mistakes when more moves were available, likely because a large portion of the alternatives in these scenarios were winning moves.

Putting these conclusions together, the takeaway is clear: “You can add as many winning moves as you want,” Salant says. “That will not result in more mistakes. If you want to make the problem as hard as possible, you need to add options that are not optimal but are pretty close to being optimal.”

Navigating Complexity in the Real World

Salant and Spenkuch see their paper as uncovering some fundamental features of complex decision-making. “For all sorts of business problems, it’s important to understand how people make decisions on a very basic level,” Spenkuch says.

He does urge caution before relying on these insights during your next strategy meeting though. “There’s a little bit of a leap of faith required to apply this to other settings,” Spenkuch warns. After all, most complex decisions are higher stakes than an online board game, and further studies are needed to test whether similar dynamics play out in other environments.

Still, there are lessons to be gleaned. Business leaders should not assume that a top-performing employee will make the right call in a split-second crisis, for instance, as the chess data suggest that the benefits of expertise fade when there’s a time crunch.

Businesses can also benefit from applying these findings to consumers, who may be asked to make decisions about complex products or services. When should you keep your marketing plan simple and when should you ramp up the complexity, perhaps by describing more facets of the product, or being ambiguous about potential use cases?

Part of the answer may depend on your market segment: flashy tactics are not likely to woo savvy consumers, whose experience will empower them to tell a great product from a so-so one.

And when facing multiple good options, Salant suspects that consumers, like chess players, may tend toward the alternatives whose benefits are immediately obvious to them— something that marketers would do well to keep in mind. “If you have a really good product, try to keep it simple,” he suggests.