Wednesday, 29 April 2009

IMF In The Dark

Last week, the IMF published its updated global forecasts, as part of the IMF World Economic Outlook, titled ‘Crisis and Recovery’. All the media, all around the world, was completely full of it (FT, NYT, Bloomberg, China Daily, Gulf News, Jakarta Post, The Times of SA, etc., etc.).

Given the hype, it is striking how different this 2009 growth forecasts is from those published half a year and a year ago. The IMF's 2009 world economy forecast, as published in April 2008, foresaw a global growth rate of 3.77%. This was 14 months after the inevitability of the global crunch had gone from being a fringe topic to being top of the agenda. Then half a year later, the 2009 forecast was pushed down to 3.03%. More than a month into the severe crisis (counting from 16 September 2009, when after Lehman Brothers folded in US time, the World woke up to a very different reality). Clearly, the IMF forecasters did not think that the challenges would not be met. Their world economy growth forecast was shaved only by 0.74 percentage points.

In its World Economic Outlook Database update, published last week, the IMF sees the world growth rate in 2009 at -1.32%. That is an astonishing 4.35 percentage point correction on a variable that averaged at 3.45% during the last two decades with a standard deviation of 1.05. In other words, the forecast does not have any meaning, apart from the IMF announcing that it thinks, that it is going to be ‘kinda bad’…

No meaning whatsoever. No meaning.

This is not the first time that the world financial markets, the economic policy makers, and the global economics community, have fallen for a confidently put precise number. I would have thought though that the experience of the past years might incline us to guard against such follies. And make us double check.

Here is a graph with the successive forecast updates from the IMF World Economic Outlook Database: April 2008 in green, October 2008 in blue, and April 2009 in red, with the forecast bit being a broken line.

The first thing to notice is that the forecast changes after the data comes out... The IMF's very complex and sophisticated system yielded an 'adaptive forecast'.

The second thing to notice is that there is always a sharp return to growth (the end of the recession) forecast for the next period. This picture will be familiar to anyone who has spent time listening to economic policy makers around the world explaining away their particular crisis, and trying to convince you that the good times are really just around the corner... I have not seen one treasury presenter, minister or lowly analyst, who would argue otherwise. For their eyes (and tongues) the crises are always about to be over. This experience makes you rather sceptical about graphs with an exceptionally quick return to the previously predicted growth trend.

Here is the same graph zoomed in on the current period.

This helps to point out one more thing: that the April 2008 IMF forecast for 2009 growth assumed that the crisis would already be over by now, and thus 2009 would be the year of recovery.

The uneasy feeling one is left with is not only that the IMF forecasting system is completely (fully, entirely, wholly, utterly, 100%) unreliable at a time when it really matters, but also that they are somewhat dishonest about it. When investment bank analysts produce a rosy picture of their respective instruments, that's sales. So is your average Treasury's similar efforts. But the IMF? Did that red line really have to go back to the trend within a couple of years? Is there any reason why we should believe that the forecasting system (one of the best in the world in my view) that produced this miserable result is going to be any better for the next period? Or should we face the fact that the IMF, and probably all governments around the world are in the dark about what is going to happen?

However, trying to be constructive...

I was wondering, if there was a pattern to the IMF's failure to forecast the world economy's crisis behaviour in a credible way. Are there common characteristics among the countries where there was a large IMF forecast correction, as opposed to the countries where there was a small one?

The second half of this post is a toy study asking that question. There are two hypotheses.

The first hypothesis assumes that there was a common, global shock to the world economy, which hit every country. Then the IMF forecasters might have been wrong about the size of the overall shock, but right about the reaction. If this were true, it might be possible to discern some common characteristic of governance among the effective adapters, and the absence of that characteristic among those struggling.

This hypothesis would be anecdotally supported by the qualitative observation that, among a group of similar countries in Eastern Europe (the best economics lab of recent years), those that have deeply divided societies are doing much worse in the current crisis. (The Baltics, Hungary, Romania, Ukraine all have pilarised elites).

There are some really interesting studies out there connecting economic growth to pilarisation (or fractionalisation) in a convincing way. Jose Montalvo and Marta Reynal-Querol for instance, in their paper, provide a strong case linking economic development to religious and ethnic polarisation. An analysis connecting their polarisation and pilarisation data to the IMF growth forecast correction would be then one way of following up on the East European observation about divided society. (This is how I while away my time...)

Sadly, this yielded a non-result. Although all three forecasts for 2009 growth (April, October 2008 and April 2009) are explained well by the four fractionalisation and polarisation variables of Reynal-Querol, confirming their original finding linking societal division to bad growth performance, none of them have any significant explanatory power vis-a-vis the IMF forecast correction. (One possible reason is that, although the dataset has 131 countries, almost all post-communist transition countries are missing. Hence the anecdotal observation cannot be checked either.)

An alternative approach would be to take indices measuring aspects of governance directly. The Bertelsmann Transition Index, for instance, is a useful set of measures of the 'matureness' of developing country governments. The results (n=117, a few countries from the BTI had to be dropped as they are not reported by the IMF, all mature economies are missing) are more promising. At first, at least. There are quite a lot of factors that have strong and significant explanatory power towards the IMF forecast correction: out of 77 variables there are 15 that have their (second degree polynomial OLS) AR-squared above 10%, the highest being 18%.

Yet, they are all going in the 'wrong' direction. The more advanced the country is, the higher the IMF forecast correction was. This cries for checking these results against the wealth of the nations, proxied by GDP per capita. And there it is: almost all of the effect is picked up by the GDP pc, almost nothing is left. Three variables remain with significant, if rather small, explanatory power: 'stateness', 'no religious dogmas', and 'UN education index'. They are still going in the 'wrong' direction, though, suggesting that the more advanced a country is, the larger the IMF correction was.

Thus Hypothesis 1 is rejected. The IMF forecast correction does not seem to pick up the effect of the respective governments' ability to adapt their policy to the crisis.

However, the above points to a different, second hypothesis. What if the IMF errors are linked to how developed a country is? For this I used the GDP per capita and the UN education index (the latter being the only one from the BTI significant set that is available for all the countries) to explain the IMF 2009 economic growth forecast correction.

The variables are:

IMF: forecast correction size (difference in IMF GDP growth forecast for the year 2009, as projected in Oct 2008, and Apr 2009);
EDU: UN education index (log);
GDP: GDP per capita (IMF data, for 2008; log).
Altogether 171 countries.

Regression 1. IMF on EDU. The AR-squared is 11%. Negative coefficient.

Regression 2. IMF on GDP. The AR-squared is 8%. Negative coefficient.

Regression 3. EDU on GDP. The AR-squared is 55%. The residual is then taken from this regression, with the name 'RES'. Positive coefficient.

Regression 4. IMF on RES. The AR-squared is 3%. Negative coefficient.

Regression 5. IMF on GDP and RES. AR-squared is 11%, with both being significant (P-values under 0.01 for both). Both have negative coefficients.

(All the above are linear OLS, with occasional mild heteroskedasticity.)

Thus it seems that Hypothesis 2 cannot be rejected. The IMF forecast correction tends to be stronger in the case of high GDP and high education levels. (Although the overall explanatory power is not very high, thus pinches of salt should remain at hand.)

In other words, the IMF thought that the rich and well-educated countries would not end up with much trouble on their hands. And they were very wrong.

By October, the credit crunch had been on the agenda for 18 months, and we were well after the Lehman-bankruptcy landmark. Thus, the IMF was very aware that the crisis was upon the global economy. In this light, the above results perhaps suggest that the IMF was in an 'effective government myth': if you are a rich, educated, non-dogmatic country, then you have good economic policy institutions, thus you will be able to react. It seems that no economic policy institution really worked, which surprised the policy optimists. (But, incidentally, not those who argued that national level policy could not possibly suffice without an effective global umbrella.)

So what have we learned? First, that the world's best global economy forecasting system is really, really bad. Second, that early hopes for being saved by government eminence were misplaced. The door is open to new ideas, I presume.


  1. Great post. Thanks.

    I wonder, however, if this would ever be taken on by the IMF. How would that look? Hi, here are our new forecasts, but, hey, don't listen to them: they were proven to be meaningless random figures in the past...

  2. why don't you check them against past sovereign premia? would be funny if you could show that the imf was going after the oft-bashed markets

  3. Thanks for this. I will keep it as a permanent footnote for reading the front page news about the latest IMF forecasts.

  4. Tamas,

    You cannot possibly be right unless if you can show that the (hypothised) analytical problems are more than a mere one-off error. There is nothing in your post that would suggest that, however.

    In other words, just because the IMF forecasts were, perhaps, less than perfect this one time, does not necessarily mean that you have to throw away all future forecasts.