The World Economic Forum’s annual Global Competitiveness Index is out today. See the hi-lites and political implications here.
But can we trust this “Index”? To my layperson’s eyes, there seems to be some serious flaws with the methodology the WEF uses to arrive at these “rankings.”
First, the common understanding a “ranking” is that the benchmarks being used to develop a ranking, the judge or arbiter of these rankings, and the objects being ranked all have something in common.
We rank the Olympic sprinters from 1 to 10 based on how fast they run 100 metres. That’s an objective test and it’s easy to slot each runner based one to 10. The benchmark is 100 metres. That’s how far you have to run. The “judge” is the stopwatch. The objects being measured or ranked are the racers.
Rock music critics will rank their favourite all-time albums. Each rock music critic may have their own subjective evaluation criteria but for each individual list the judge is the common denominator. So for all the objects being measured — all the albums ever recorded — the judge — each music critic — and the benchmark — whatever criteria used by each individual judge — are all in common.
But for the World Economic Forum’s ranking, it is impossible to “rank” countries because they are being ranked by different judges using different criteria.
How, for example, do you measure “Buyer Sophistication”? (We ranked 11) Or “Favoritism in decisions of government officials”? (We ranked 22nd) Or “Ethical Behaviour of Firms” (We ranked 7th) Or, for that matter, “Soundness of Banks”?
The WEF does this by polling a handful of business executives in each country. As I blogged in 2008, the WEF asked 75 business executives in Canada if they thought our banks were sound. It looks like most or all of them said yes. Meanwhile, the WEF found another group of business executives in, say, Finland, if they thought Finland’s banks were sound. Apparently many business executives in Finland were happy with their own banks’ soundness and Finland ranked 8.
But the Finnish business executives were not asked about the soundness of Canada’s banks and the Canadian executives were not asked about the Finnish banks. So we really can’t “rank” Finland against Canada in any meaningful way because the the Finnish and Canadian executives were being asked to make very subjective judgement — what does “soundness” mean to a Finnish exec and to a Canadian exec — about two different groups of objects, Canadian banks and Finnish banks.
And then there’s the sample size: 75 business executives? Just 75? And why only business executives? Business executives might have a very different view of the “ethical behaviour of firms” compared to labour groups or NGOs. And why would business executives have any special insight that lets them rank “Public Trust of Politicians” (we ranked 21st)?
“If you look at . . . all these rankings, they’re meaningless,” Reuven Brenner, a professor in the Desautels faculty of management at McGill University told me in 2008.
So if a politician is boasting about our banks being number one or another is bashing a government for letting an index ranking slip, I’d take the WEF’s report with a heavy grain of salt. If anything, it’s seems like great PR for the Klaus Schwab’s Davos Summit and the World Economic Forum and dubious “globalization research.”