Big banks like Goldman Sachs spectacularly failed to predict the World Cup winner - here's why

Matthias Hangst/Getty Images

France won the World Cup, but only one bank saw it coming.

• Five major banks predicted the outcome of the World Cup in Russia, but all but one got it wrong.
• Goldman Sachs forecast that Brazil would lift the trophy, while UBS backed Germany.
• Banks were afflicted by the sheer unpredictability of this year's tournament, which saw unfavored Croatia make it all the way to the final.

The 2018 World Cup, which ended on Sunday, was an almost unequivocal success. The soccer was scintillating, there was little trouble away from the pitch, and the controversial VAR system largely delivered as promised.

As with any World Cup, numerous individuals shined in Russia, with a breakout performance from France's Kylian Mbappe among the highlights.

One group that didn't shine, however, were big financial institutions. Before the tournament, banks including Goldman Sachs, ING, and UBS all took a stab at a forecasting what would happen, generally by applying techniques traditionally used in financial analysis and economic modeling to soccer.

All but one of them - Nomura - got the tournament wrong.

Of the five major banks to forecast a winner before the tournament, two chose Spain, one chose Germany, and one chose Brazil. Goldman Sachs even changed its modeling several times during the tournament and still managed to get it wrong. Only Japanese lender Nomura managed to correctly predict that France would leave Russia as world champions.

So what went wrong? To understand where lenders slipped up, it is first important to look at exactly how they tried to make their forecasts, and what they got wrong. Each bank is different, so we'll look at some of them individually.

Goldman Sachs

Goldman Sachs used perhaps the most interesting form of modeling, employing machine learning to run 200,000 models, mining data on team and individual player attributes, to help forecast specific match scores. The bank then simulated 1 million variations of the tournament to calculate the probability of advancement for each squad.

"We are drawn to machine learning models because they can sift through a large number of possible explanatory variables to produce more accurate forecasts than conventional alternatives," a group of strategists from Goldman's international research team wrote in a client note before the tournament.

"We capture the stochastic nature of the tournament carefully using state-of-the-art statistical methods and we consider a lot of information in doing so," they added.

"England meets Germany in the quarters, where Germany wins; and Germany meets Brazil in the final, and Brazil prevails," the bank said in its first forecasts, as evidenced by the chart below:

Goldman Sachs

In reality, Germany was eliminated at the group stage, England made it all the way to the semi finals, and Brazil lost to Belgium in the quarter finals. In total, just 11 of the teams Goldman expected to make the last 16 progressed from the group stage, only four of eight predicted quarter finalists made it that far, and just one of the four forecast semi finalists got there.

Goldman even forecast that eventual runners-up, Croatia, would be eliminated from the tournament at the group stage, exiting at the expense of Iceland.

After the group stage, Goldman decided to update its predictions, but consistently missed the mark, with its final forecast - delivered ahead of the semi finals matches between England and Croatia, and Belgium and France - saying we'd see a final between England and Belgium. The final, of course, was played between Croatia and France.

Goldman did caveat its failures before the tournament, saying that "forecasts remain highly uncertain, even with the fanciest statistical techniques, simply because football is quite an unpredictable game."

UBS

Murad Sezer/Reuters

Brazilian star Neymar's underperformance was one of the reason's for his side's early exit.

Swiss giant UBS' modeling was not as extensive as Goldman's, with the bank running just 10,000 simulations of the tournament.

UBS created its predictions by inputting a series of variables, including the ELO ranking of teams - which is an objective measure of how good they are - into a statistical model, and then running that model through a so-called Monte Carlo simulation. Here's the bank's explanation from before the tournament:

"To account for the many different roads to the final, we conduct what statisticians call a Monte Carlo simulation. What might sound fancy to some readers is in practice quite simple: Instead of mapping out all the different constellations, we draw a large number of random variables and use these to bring in a random component to our calculations and to simulate the championship. A better team is still more likely to win than its lower ranked opponent, but this is sport, and upsets DO happen. After we've repeated this again and again, we end up with the results of 10,000 virtual tournaments. We then simply count how many times each team won, made it to the semis, or was already eliminated in the group stage."

Rather than trying to predict an exact bracket like Goldman Sachs, UBS gave each team a probability of progressing through each round of the tournament, and a probability of winning it. In this model, UBS backed Germany to win the tournament. Germany, of course, went out in the group stage - an outcome UBS said had a less than 10% chance of happening.

Winner France was just the fifth favorite in UBS' model. But even more surprising is that Croatia were given a less than 1% chance of making it to the final, and only a 24% chance of even making it out of the group.

Staff at UBS Wealth Management, which compiled the forecasts, admit they were wrong, but pointed to getting around two-thirds of its group stage predictions correct.

"The big surprise for us was runner-up Croatia: they overcame Germany to finish top of their group, beating Denmark, Russia, and then England in the knockout stages," Chief Investment Officer Mark Haefele said in an email to clients seen by Business Insider.

Haefele also said investors can learn from the results, adding that they should always "be sure not to confuse probabilities for certainties - everything in investing (like in football) is only ever likely, probable, or possible, and never certain."

"Even the best looking investments can underperform."

ING and Nomura

Getty Images

Croatia's team celebrates a goal.

ING took a particularly novel - and ruthlessly capitalistic - approach to their forecasts, using the overall value of each team in the tournament to predict the winner. ING plumped for Spain, whose squad had a combined value of $1.187 billion, France came second in this forecast, with a value of just less, $1.183 billion.

Nomura, the only bank to pick the right winner, put forward this methodology pre-tournament:

"Being analysts, we have to apply some rigor to our World Cup predictions, so we've decided to apply portfolio theory and the efficient markets hypothesis to the World Cup. We look at the value of players in each team, the momentum of team performance and historical performance to arrive at three portfolios of teams to watch."

Why did they get things wrong?

Getty Images

France's breakout performer Kylian Mbappe.

Quite simply, the majority of banks were wrong because of how unpredictable this year's World Cup was. Germany's exit, for example, was characterised by UBS' Haefele as "probably one of the biggest surprises in the more recent history of the tournament." Germany had not been eliminated in the first round of the competition since 1938.

Away from Germany's elimination, the penalty shoot-out victory of Russia against Spain, England's progress to the semi finals, and Argentina's huge underperformance throughout the tournament were all massive shocks, and also flew in the face of the data.

Shocks in the group stage created a lopsided bracket in the second round, where one side of the tournament was stacked with the most favored teams, and the other filled with decent but unspectacular sides like England and Croatia. This helped to exacerbate the discrepancies between banks forecasts and reality.

Ultimately, with an event like the World Cup, relying on statistics is a fool's game. Football is simply too unpredictable to be reduced to a statistical model. Sure statistics can tell us a lot, but they cannot account for the burning pace of Kylian Mbappe, the impeccable game management of Luka Modric, or the once-in-a-generation vision of Kevin De Bruyne.

As Goldman Sachs said before the World Cup began: "This is, of course, precisely why the World Cup will be so exciting to watch."

Get the latest Goldman Sachs stock price here.