Which metric to use for estimating accuracy of a climate model?

Question

Let's assume I have 3 different climate models for a specific region that project the temperature. I also have the observations of that region for the same time frame(the real temperatures). My readings are daily. Which metric should I use and why? For example :

• Mean Average is not really useful since because of the sign readings can cancel each other out
• Absolute Mean Average. I mostly think this is the most useful. However, I am puzzled because when I subtracted the projections - observations I noticed that around 500 readings out of the 13500 have a big difference (around 10 degrees of Celcius). Should I include these outliers or just delete them?
• Does Squared Mean Error and Root Squared Mean Error provide me with any valuable insight?

EDIT : Hey everyone. Thank you for the feedback. My question at hand is that I am given 4 different bias correction methods and I want to create a multi model with equal voting. I want to present which bias correction method is more suitable: quantile mapping and scaled distribution mapping (Gamma and Normal Corrections). In order to do that(and to show that bias correction is useful) I found the Mean Absolute Error, Squared Mean Error and Root Squared Mean Error to find the most accurate model. Then performed bias correction with all methods and used the same metrics again. Also I created the multi model(basically it s the average of the models). Thus I created a plot with a single model, a multi model and a multi model with bias correction. With these metrics, I saw that the most accurate was the scaled distribution mapping with normal corrections which makes sense since temperature follows a normal distribution over the years (basically it has the lowest Mean Absolute Error) I based the comparison on MAE because I read that we usually use it to compare models etc

Answer 1

The metric may not be the most important question and it rather depends on what you want to use the models for, which is not stated in the question. See the answer by @StephanKolassa .

An important issue is that it is difficult to perform a like-for-like comparison between model output and observations, and frequently this is done without understanding how the models work, which is more important than the statistical/forecasting considerations. Some relevant issues:

1. The model runs are projections rather than predictions. A projection in this context is essentially a conditional prediction - if the forcings (e.g. the mount of GHG in the atmosphere) in reality match those in the scenario then the projection is an estimate of what we should expect to see. So the result must consider how well the scenario matches reality.
2. Models do not directly predict weather (i.e. the day-to-day variation in temperature), they simulate weather that is statistically consistent with the conditions in the scenario, according to the physics of the model. This should come as no surprise, weather is chaotic (it is deterministic, but extremely sensitive to initial conditions). The best weather models have a useful prediction horizon of a matter of days, so there is no way a climate model (which is basically the same thing as a weather model) is going to be able to predict weather conditions years in advance. I suspect that is why you get very large errors. I makes no sense whatsoever to compare model output with observations at a daily timescale.
3. A persistence model is always going to beat a climate model hands down on a daily timescale, even if the climate model is perfect, because the climate model doesn't have the exact initial conditions (and in practice is also spatially and temporally quantised). The persistence model gets its initial conditions from yesterdays weather, so it has a much easier job.
4. A climate model is essentially a Monte Carlo simulation, so we don't have just one run with one set of initial conditions, we have an ensemble of model runs, each with different initial conditions. The weather from day to day may be radically different in each run (and radically different from the observations). The distribution of model runs however, gives an indication of the statistical properties of the weather that we can expect in future climate. This makes sense as climate is the statistical properties of the weather. This means all we should expect is for the observations to lie within the spread of the model runs.
5. The models are spatially quantised. It makes no sense to compare station level data with averages over the scale of the grid boxes used in climate models (typically several km or more). The OP mentions regional temperature, which is good. For a fair comparison, it would be best to estimate a gridded dataset that matches the grid used by the model? This may be an issue if the grid box contains a mixture of ocean and land, but the region in the observations is land only.
6. Compare anomalies relative to some sensible baseline (preferably 30 years or more). Individual model runs can be very variable in their average temperatures, but their projections of changes in temperature are much more reliable.

I mention these things because there have been journal papers written by experts in forecasting that were highly critical of climate models, but who unfortunately didn't take the time to find out how climate models work, how they are used, or how the model output is interpreted. All of these mistakes have been made, and are pitfalls for the unwary.

If you have any questions about climate or climate models, do ask at the EarthSciences SE (and tag me if you want me to see them as I don't check it that often).

Update: To explain in a bit more detail why comparing model output with observations on a daily timescale doesn't make much sense, consider a perfect climate model. How could we create a perfect climate model? Say we had a means of visiting parallel dimensions and observing the weather on alternate Earths. If there is an infinite number of parallel dimensions, then there will be a large number where the climate forcings are exactly the same as those in our reality. Will they have the same weather? No. Say on one parallel Earth a butterfly flapped its wings some time in the Cretaceous, but the version on our Earth didn't. That would mean that the initial conditions of the atmosphere on the two Earths at that point would be subtly different. As weather is chaotic, that means the pattern of weather on the two Earths would diverge, and you could easily get a day where it was scorching hot on one Earth and snowing on the other in the same region, if both conditions were consistent with the forcings (in the U.K. it has been known to snow in late Spring and to have warm Summer-like weather, so this is possible). We wouldn't expect the day-to-day weather on parallel Earths to be very similar. A climate model is attempting to be a simulation of such a parallel Earth, and we shouldn't expect the weather of the climate simulation to be any more similar than the weather on a real parallel Earth.

The good thing is that by having many parallel Earths, or many model runs, we can estimate the spread of temperature that is feasible for the forcings. So the proper way of performing model-observation comparison is to see where the observations lie within that spread.

Answer 2

By "Absolute Mean Average" I assume you mean the Mean Absolute Error: you take the difference of each separate forecast and its associated actual, then take the mean over the absolute values of these differences. Minimizing the MAE amounts to eliciting the conditional median of the future temperature distributions: Mean absolute error OR root mean squared error? and Why does minimizing the MAE lead to forecasting the median and not the mean? For temperatures, which one can usually assume to be symmetrically distributed, there should not be a lot of difference between MAE and MSE (more precisely, between the forecasts that optimize each).

You might be interested in the sections on accuracy measurement in Forecasting: Theory and Practice.

Whether you should remove very bad forecasts depends. I would rather try to use them to learn under what circumstances your model breaks down. Such information can be very valuable. Also, removing very bad forecasts tells your model that if it forecasts off a little bit, you will be concerned, but if it is badly off, you don't mind any more. Is this the message you want to send to your model?