Hurricanes and errors of reasoning? Again...

This time they relate to Karen Clark's presentation.

First she makes the comment:

 "Models Can't…Predict Near Term Catastrophe Losses".

In my previous blog post Dice seriously underestimating risk, I noted that cat models do not attempt to "predict" actual losses. They simulate a range of possible losses around an estimated mean level. If there is plausible evidence that the mean loss is higher, for some reason, then it is right to reflect this in the model. The actual outcome may be higher or lower - that's what random means.

To give evidence for her claims Karen then shows the following graphic:

I believe this graphic is really evidence for the opposite point of view i.e. that  catastrophe models should take account of the medium term forecast.

From a quick look you might think that the graphic makes Karen’s point well - and it does show that actual outcomes are more variable than the forecast. To be honest I'm not sure what NOAA's ranges are in the graphic.  If they are inter-quartile ranges then the actual data might even be within the full range of variance.

In any case, if you look at the average, over the 9 years, of number of storms NOAA predicted would arise in excess of the long term average you get around 2 (17/9 if I’m reading the graph right). Now look at the actual number of storms in excess of the long term average, you again get 2 (well 15/9 so pretty close). 

In other words NOAA predicted an average of 2 storms in excess of long term averages over the period and that's what we got. 

This suggests that, whilst their annual hit rate might not be perfect, over the long term they do add value compared to background climate averages.

Another way to look at this is to consider how many times NOAA got the difference from long term average directionally correct.  Use the letter “S” when they get the direction correct, this is a “success” and “F” (for failure) when they get it wrong. Then in the 9 years shown NOAA’s success pattern is:    F,S,S,S,F,S,S,S,S

So they got the direction right 7 years out of 9.   If they were just guessing this success rate would be pretty unlikely - they'd have around a 9% chance of guessing that well or better.  To my mind this suggests their forecast has some skill and that is the reason for their success rate. 

So it looks like climate models aren't bad at telling us the average frequency over the medium term. Fortunately it is the medium term view that insurers currently want to reflect in their catastrophe models.

Our forthcoming paper discusses the use of forecasting in the insurance industry and considers these issues in more detail. 

Comments

Showing: 1 comments