[ExI] Forecasting experts’ simple model leaves expensive climate models cold

[Max More]

Another interesting piece from Armstrong and Green:

Forecasting experts’ simple model leaves expensive climate models cold

A simple model was found to be produce forecasts 
that are over seven times more accurate than 
forecasts from the procedures used by the United 
Nations Intergovernmental Panel on Climate Change (IPCC).

This important finding is reported in an article 
titled “Validity of climate change forecasting 
for public policy decision making” 
(http://kestencgreen.com/gas-2009-validity.pdf) 
in the latest issue of the International Journal 
of Forecasting. It is the result of a 
collaboration among forecasters J. Scott 
Armstrong of the Wharton School, Kesten C. Green 
of Monish University, and climate scientist 
Willie Soon of the Harvard-Smithsonian Center for Astrophysics.

In an earlier, paper 
(http://www.forecastingprinciples.com/files/WarmAudit31.pdf), 
Armstrong and Green found that the IPCC’s 
approach to forecasting climate violated 72 
principles of forecasting. To put this in 
context, would you put your children on a 
trans-Atlantic flight if you knew that the plane 
had failed engineering checks for 72 out of 127 
relevant items on the checklist?

The IPCC violations of forecasting principles 
were partly due to their use of models that were 
too complex for the situation. Contrary to 
everyday thinking, complex models provide 
forecasts that are less accurate than forecasts 
from simple models when the situation is complex and uncertain.

Confident that a forecasting model that followed 
scientific forecasting principles would provide 
forecasts that were more accurate than those 
provided by the IPCC, Green, Armstrong and Soon 
used a model that was more consistent with 
forecasting principles and knowledge about climate.

The forecasting model was the so-called “naïve” 
model. It assumes that things will remain the 
same. It is such a simple model that people are 
generally not aware of its power. In contrast to 
the IPCC’s central forecast that global mean 
temperatures will rise by 3 C over a century, the 
naïve model simply forecasts that temperatures 
next year and for each of 100 years into the 
future would remain the same as the last years’.

The naïve model approach is confusing to 
non-forecasters who are aware that temperatures 
have always varied. Moreover, much has been made 
of the observation that the temperature series 
that the IPCC uses shows a broadly upward trend 
since 1850 and that this is coincident with 
increasing industrialization and associated 
increases in manmade carbon dioxide gas emissions.

In order to test the naïve model, annual 
forecasts were made from one to 100 years in the 
future starting with 1850’s global average 
temperature as the forecast for the years 1851 to 
1950. This process was repeated by updating for 
each year up through 2007. This produced 10,750 
annual average temperature forecasts for all 
horizons. It was the first time that the IPCC’s 
forecasting procedures had been subject to a 
large-scale test of the accuracy of the forecasts that they produce.

Over all the forecasts, the IPCC error was 7.7 
times larger than the error from the naïve model.

While the superiority of the naïve model was 
modest for one- to ten-year-ahead forecasts 
(where the IPCC error was 1.5 times larger), its 
superiority was enormous for the 91- to 
100-year-ahead forecasts, where the IPCC error was 12.6 times larger.

Is it proper to conduct validation tests? In many 
cases, such as the climate change situation, 
people claim that: “Things have changed! We 
cannot use the past to forecast.” While they may 
think that their situation is unique, there is no 
logic to this argument. The only way to forecast 
the future is by learning from the past. In fact, 
the warmers claims are also based on their analyses of the past.

Could one improve upon the naïve model? The naïve 
model violates some principles. For example, it 
violates the principle that one should use as 
long a time series as possible, because it bases 
all forecasts on simply the global average 
temperature for the single year just prior to 
making the forecasts. It also fails to combine 
forecasts from different reasonable methods. The 
authors planned to start simple with this 
self-funded project and to then obtain funding to 
undertake a more ambitious forecasting effort to 
ensure that all principles were followed. This 
would no doubt improve accuracy. However, the 
forecasts from the naïve model were very 
accurate. For example, the mean absolute error 
for the 108 fifty-year ahead forecasts was only 
0.24 C. It is difficult to see any economic value 
to reducing such a small forecast error.

For further information contact J.  Scott 
Armstrong (http://jscottarmstrong.com or Kesten 
C. Green (http://kestencgreen.com/)]