Blind Faith in Predictive (or any) Analytics is Dangerous
"Don't attempt to cross a river because it is on average 4 feet deep"
There was a recent article in the New York Sun which takes a poke at several experts because their so-called predictions did not come to pass. The criticism ranges from Superbowl predictions to unemployment numbers to the stock market. The real problem is we do not know who to blame: the experts because they made likely events seem like certainties or the lay audience for not exercising diligent skepticism. No matter what, we can argue that everyone was guilty of crossing the river quoted above!
How? Let me explain. Poor risk management, which is responsible for so much grief in the economy today and which relies heavily on predictive models, involves calculating the probability of the maximum loss a given portfolio can experience over a specified time horizon. This is the part which can be a slippery slope – because it partially rests on a critical assumption. Almost all conventional financial theory makes the assumption that returns are normally distributed.
In the case of the sub-prime mortgage crisis(seems like ancient history today!) all the major investment banks had an extensive arsenal of models to figure out risk. There were three fatal flaws with these models – limited historical information, an unfortunate choice of time horizon to set up the assumptions, and finally “incomplete” models!
The first flaw is (in hindsight) easy to recognize. There were limited sub-prime securities in the past and all previous models were constructed using prime mortgages.
The second flaw was that these models were fed sub-prime data at a point in time when the sub-prime defaults actually dropped from 10% to 5% (which is a good thing for banks of course). So in effect the models resulted in a self-fulfilling prophecy – start with an assumption of normally distributed results and then predict normally distributed performance for the future.
Continued on the next page
Follow Technorati