Debunking the “our banks are the best” meme

- September 5th, 2012

The World Economic Forum’s annual Global Competitiveness Index is out today. See the hi-lites and political implications here.

But can we trust this “Index”? To my layperson’s eyes, there seems to be some serious flaws with the methodology the WEF uses to arrive at these “rankings.”

First, the common understanding of a “ranking” is that the benchmarks being used to develop a ranking, the judge or arbiter of these rankings, and the objects being ranked all have something in common.

We rank the Olympic sprinters from 1 to 10 based on how fast they run 100 metres. That’s an objective test and it’s easy to slot each runner based one to 10.  The benchmark is 100 metres. That’s how far you have to run. The “judge” is the stopwatch. The objects being measured or ranked are the racers.

Rock music critics will rank their favourite all-time albums. Each rock music critic may have their own subjective evaluation criteria but for each individual list the judge is the common denominator. So for all the objects being measured — all the albums ever recorded —  the judge — each music critic — and the benchmark — whatever criteria used by each individual judge — are all in common.

But for the World Economic Forum’s ranking, it is impossible to “rank” countries because they are being ranked by different judges using different criteria.

How, for example, do you measure “Buyer Sophistication”? (We ranked 11) Or “Favoritism in decisions of government officials”? (We ranked 22nd) Or “Ethical Behaviour of Firms” (We ranked 7th) Or, for that matter, “Soundness of Banks”?

The WEF does this by polling a handful of business executives in each country. As I blogged in 2008, the WEF asked 75 business executives in Canada if they thought our banks were sound. It looks like most or all of them said yes. Meanwhile, the WEF found another group of business executives in, say, Finland, if they thought Finland’s banks were sound. Apparently many business executives in Finland were happy with their own banks’ soundness and Finland ranked 8.

But the Finnish business executives were not asked about the soundness of Canada’s banks and the Canadian executives were not asked about the Finnish banks. So we really can’t “rank” Finland against Canada in any meaningful way because the the Finnish and Canadian executives were being asked to make very subjective judgement — what does “soundness” mean to a Finnish exec and to a Canadian exec — about two different groups of objects, Canadian banks and Finnish banks.

And then there’s the sample size: 75 business executives? Just 75? And why only business executives? Business executives might have a very different view of the “ethical behaviour of firms”  compared to labour groups or NGOs. And why would business executives have any special insight that lets them rank “Public Trust of Politicians” (we ranked 21st)?

“If you look at . . . all these rankings, they’re meaningless,” Reuven Brenner, a professor in the Desautels faculty of management at McGill University told me in 2008.

So if a politician is boasting about our banks being number one or another is bashing a government for letting an index ranking slip, I’d take the WEF’s report with a heavy grain of salt. If anything, it’s seems like great PR for the Klaus Schwab’s Davos Summit and the World Economic Forum and dubious “globalization research.”

Categories: Economy

Subscribe to the post

1 comment

  1. Adrian Monck says:

    David

    As the report notes, the executive survey “gathers valuable information on a broad range of variables for which data sources are scarce or nonexistent. For this reason, and for the integrity of our publications and related research, improving the sampling methodology and comparability of data across the globe remain an essential and ongoing endeavor.”

    Section 1.3 of the Global Competitiveness Report (http://wef.ch/gcr) describes in detail how the executive survey data is handled. Survey data is notoriously hard to gather and assess, so we use partner institutes around the world, and a variety of statistical techniques to “clean” and “normalise” the results.

    We externally audited our process this year, and we describe in the report how the data is handled to enable academics and other researchers to use it.

    With all the caveats we detail, it remains the biggest survey of its kind and the data is used by Transparency International in both its Corruption Perceptions and its International Bribe Payers Indices.

    If you look at this year’s report, you’ll see that 103 executives were surveyed in Canada and the process by which they were chosen, the weightings attributed to them, response rates etc.

    Not all results country make it in…as the report makes clear:

    “…the 2012 Survey data collected in Ecuador, Georgia, Rwanda, and Sri Lanka deviate significantly from the 2011 results. The subsequent analysis revealed that this departure was not accompanied by a similar trend in indicators taken from other sources, and the recent developments in these countries do not seem to provide enough justification for the large swings observed. For these four countries, we therefore use only the 2011 Survey data in the computation of this year’s GCI. While this remains a remedial measure, we will continue to investigate the situation in an effort to improve the reliability of the Survey data in these countries. Last year, the same analysis resulted in the Survey data of six countries—Bosnia and Herzegovina, Kazakhstan, Morocco, Qatar, Saudi Arabia, and Slovenia—not being included in the analysis. This year, as an intermediate step toward the re-establishment of the standard computation method, we used a weighted average of the Survey data of 2010 and 2012 for these countries.”

    So please, take a look in detail before reaching for the shotgun. The survey isn’t perfect. We don’t claim it is. But in the absence of better, it can and does serve a useful purpose.

    Adrian Monck
    World Economic Forum

Comments are closed.