8
$\begingroup$

Let's assume I have 3 different climate models for a specific region that project the temperature. I also have the observations of that region for the same time frame(the real temperatures). My readings are daily. Which metric should I use and why? For example :

  • Mean Average is not really useful since because of the sign readings can cancel each other out
  • Absolute Mean Average. I mostly think this is the most useful. However, I am puzzled because when I subtracted the projections - observations I noticed that around 500 readings out of the 13500 have a big difference (around 10 degrees of Celcius). Should I include these outliers or just delete them?
  • Does Squared Mean Error and Root Squared Mean Error provide me with any valuable insight?

EDIT : Hey everyone. Thank you for the feedback. My question at hand is that I am given 4 different bias correction methods and I want to create a multi model with equal voting. I want to present which bias correction method is more suitable: quantile mapping and scaled distribution mapping (Gamma and Normal Corrections). In order to do that(and to show that bias correction is useful) I found the Mean Absolute Error, Squared Mean Error and Root Squared Mean Error to find the most accurate model. Then performed bias correction with all methods and used the same metrics again. Also I created the multi model(basically it s the average of the models). Thus I created a plot with a single model, a multi model and a multi model with bias correction. With these metrics, I saw that the most accurate was the scaled distribution mapping with normal corrections which makes sense since temperature follows a normal distribution over the years (basically it has the lowest Mean Absolute Error) I based the comparison on MAE because I read that we usually use it to compare models etc

$\endgroup$
1
  • 2
    $\begingroup$ Firstly, you might want to look into "downscaling" methods. If a model run is more accurate on a daily basis, it is probably just random chance. If you want to determine which model is working better, you might want to see how accurate it is on timescales long enough for the "weather" to average out, e.g. seasonal averages. More skillfull models will get the statistical properties right (e.g. means and variances), they won't be good at predicting the noise (the weather on a particular day). $\endgroup$ Commented Sep 4, 2022 at 20:53

2 Answers 2

8
$\begingroup$

The metric may not be the most important question and it rather depends on what you want to use the models for, which is not stated in the question. See the answer by @StephanKolassa .

An important issue is that it is difficult to perform a like-for-like comparison between model output and observations, and frequently this is done without understanding how the models work, which is more important than the statistical/forecasting considerations. Some relevant issues:

  1. The model runs are projections rather than predictions. A projection in this context is essentially a conditional prediction - if the forcings (e.g. the mount of GHG in the atmosphere) in reality match those in the scenario then the projection is an estimate of what we should expect to see. So the result must consider how well the scenario matches reality.
  2. Models do not directly predict weather (i.e. the day-to-day variation in temperature), they simulate weather that is statistically consistent with the conditions in the scenario, according to the physics of the model. This should come as no surprise, weather is chaotic (it is deterministic, but extremely sensitive to initial conditions). The best weather models have a useful prediction horizon of a matter of days, so there is no way a climate model (which is basically the same thing as a weather model) is going to be able to predict weather conditions years in advance. I suspect that is why you get very large errors. I makes no sense whatsoever to compare model output with observations at a daily timescale.
  3. A persistence model is always going to beat a climate model hands down on a daily timescale, even if the climate model is perfect, because the climate model doesn't have the exact initial conditions (and in practice is also spatially and temporally quantised). The persistence model gets its initial conditions from yesterdays weather, so it has a much easier job.
  4. A climate model is essentially a Monte Carlo simulation, so we don't have just one run with one set of initial conditions, we have an ensemble of model runs, each with different initial conditions. The weather from day to day may be radically different in each run (and radically different from the observations). The distribution of model runs however, gives an indication of the statistical properties of the weather that we can expect in future climate. This makes sense as climate is the statistical properties of the weather. This means all we should expect is for the observations to lie within the spread of the model runs.
  5. The models are spatially quantised. It makes no sense to compare station level data with averages over the scale of the grid boxes used in climate models (typically several km or more). The OP mentions regional temperature, which is good. For a fair comparison, it would be best to estimate a gridded dataset that matches the grid used by the model? This may be an issue if the grid box contains a mixture of ocean and land, but the region in the observations is land only.
  6. Compare anomalies relative to some sensible baseline (preferably 30 years or more). Individual model runs can be very variable in their average temperatures, but their projections of changes in temperature are much more reliable.

I mention these things because there have been journal papers written by experts in forecasting that were highly critical of climate models, but who unfortunately didn't take the time to find out how climate models work, how they are used, or how the model output is interpreted. All of these mistakes have been made, and are pitfalls for the unwary.

If you have any questions about climate or climate models, do ask at the EarthSciences SE (and tag me if you want me to see them as I don't check it that often).

Update: To explain in a bit more detail why comparing model output with observations on a daily timescale doesn't make much sense, consider a perfect climate model. How could we create a perfect climate model? Say we had a means of visiting parallel dimensions and observing the weather on alternate Earths. If there is an infinite number of parallel dimensions, then there will be a large number where the climate forcings are exactly the same as those in our reality. Will they have the same weather? No. Say on one parallel Earth a butterfly flapped its wings some time in the Cretaceous, but the version on our Earth didn't. That would mean that the initial conditions of the atmosphere on the two Earths at that point would be subtly different. As weather is chaotic, that means the pattern of weather on the two Earths would diverge, and you could easily get a day where it was scorching hot on one Earth and snowing on the other in the same region, if both conditions were consistent with the forcings (in the U.K. it has been known to snow in late Spring and to have warm Summer-like weather, so this is possible). We wouldn't expect the day-to-day weather on parallel Earths to be very similar. A climate model is attempting to be a simulation of such a parallel Earth, and we shouldn't expect the weather of the climate simulation to be any more similar than the weather on a real parallel Earth.

The good thing is that by having many parallel Earths, or many model runs, we can estimate the spread of temperature that is feasible for the forcings. So the proper way of performing model-observation comparison is to see where the observations lie within that spread.

$\endgroup$
5
  • $\begingroup$ @alex as I said, comparing daily observations with model output really doesn't make a great deal of sense as the model output is not a prediction of daily weather. I'll extend my answer to try and explain why. $\endgroup$ Commented Sep 4, 2022 at 20:35
  • $\begingroup$ Hey @Dikran, thank you for you feedback. I read your comment carefully I m just stating what I have done. How should I proceed then with the comparisons of bias correction methods? $\endgroup$
    – Alex
    Commented Sep 4, 2022 at 20:44
  • $\begingroup$ @Alex it isn't a matter of bias. The model output simply isn't a prediction of daily weather so evaluating it's accuracy in that was simply doesn't make sense. It is a bit like having a real double pendulum (en.wikipedia.org/wiki/Double_pendulum) and a computer simulation of that double pendulum. How would you assess the accuracy of the simulation if you didn't have perfect knowledge of the initial conditions? The simulation could be perfect, but it won't match the trajectory of the real one because it is sensitive to initial conditions. $\endgroup$ Commented Sep 4, 2022 at 20:47
  • 1
    $\begingroup$ So it would be more useful to plot the outcomes from each method and demonstrate which method follows the distribution of the observations better perhaps? $\endgroup$
    – Alex
    Commented Sep 4, 2022 at 20:53
  • 2
    $\begingroup$ Yes, that sounds better. You might be better off asking at the EarthSciences SE as the issues here are more to do with the climatology than the statistics. I did some work with climate models (mostly weather typing and statistical downscaling), but I am not an expert on climate modelling, I only know the basics. You may get someone with better advice to give there. $\endgroup$ Commented Sep 4, 2022 at 20:55
7
$\begingroup$

By "Absolute Mean Average" I assume you mean the Mean Absolute Error: you take the difference of each separate forecast and its associated actual, then take the mean over the absolute values of these differences. Minimizing the MAE amounts to eliciting the conditional median of the future temperature distributions: Mean absolute error OR root mean squared error? and Why does minimizing the MAE lead to forecasting the median and not the mean? For temperatures, which one can usually assume to be symmetrically distributed, there should not be a lot of difference between MAE and MSE (more precisely, between the forecasts that optimize each).

You might be interested in the sections on accuracy measurement in Forecasting: Theory and Practice.

Whether you should remove very bad forecasts depends. I would rather try to use them to learn under what circumstances your model breaks down. Such information can be very valuable. Also, removing very bad forecasts tells your model that if it forecasts off a little bit, you will be concerned, but if it is badly off, you don't mind any more. Is this the message you want to send to your model?

$\endgroup$
5
  • 1
    $\begingroup$ +1 It probably isn't the model breaking down, it is more likely a lack of understanding of how the model works and how it is used. It is not meaningful to compare climate models to observations in this way as they are modelling the statistical properties of the weather rather than the weather itself. A more realistic question would ask if the observed weather is consistent with those statistical properties projected by the model. $\endgroup$ Commented Sep 4, 2022 at 18:15
  • $\begingroup$ @DikranMarsupial Is this generally true: Climate models are models of statistical properties of weather rather than models of the weather itself? Do you have a source for that? (If so, then I understand better why you said that this question might belong more on Earth Sciences SE than Statistics SE in the comments on your own answer.) $\endgroup$ Commented Sep 6, 2022 at 0:39
  • $\begingroup$ @DikranMarsupial Sorry! I think I understand what you mean, if I compare weather forecasting with climate models. Weather forecasts use data to predict events in a much shorter time frame than climate models which use weather data (be they averages or other statistics) over decades. Is that what you mean by your comment? $\endgroup$ Commented Sep 6, 2022 at 1:05
  • 1
    $\begingroup$ @EllieKesselman Climate models and weather models are basically the same thing, however weather models are operating on a short enough time (a few days) that the initial conditions are still relevant to the prediction. Note that modern weather forecasts are based on an ensemble of perhaps 50 model runs, so even then the uncertainty in initial conditions can lead to substantial differences in the outcome (if that were not the case, there would be no point in using an ensemble) $\endgroup$ Commented Sep 6, 2022 at 5:17
  • 1
    $\begingroup$ I think there is a section in the IPCC WG1 reports that explains the interpretation of climate model output. $\endgroup$ Commented Sep 6, 2022 at 5:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.