4
$\begingroup$

Perkins et al. (2007) introduce a "skill score" for measuring climate model output against observations. The score basically consists of measuring the overlap between probability density functions of the model (m), and the observations (o); for some variable (eg. maximum daily temperature). It is calculated as

$$S_{score} = \int^\infty_{-\infty} min[pdf(m),\ pdf(o)]$$

I'm trying to wrap my head around bayesian conditional distributions, and not getting far. This seems related, in that it's some measure of the likelihood of the model being a good estimate of the observations. However, I can't figure out if it's equivalent or not.

Given $P(m|o) = \frac{P(o|m)P(m)}{P(o)}$, is it correct that $P(m)=\int^\infty_{-\infty} pdf(m)=1$, and the same for the obs? Or am I missing something big here?

Improve this question
$\endgroup$
3
  • $\begingroup$ Hrm... I think I'm conflating the probability of the values within the distribution with the probability of the distribution occuring. But if all pdfs are somewhat possible, then isn't the probability of a particular distribution occuring $P(m)=0$? Or I need to find the pdf of all possible pdfs? $\endgroup$
    – naught101
    Commented Aug 7, 2012 at 2:14
  • $\begingroup$ In the paper, $S_{score}$ is defined differently. $S_{score}=\sum_1^n\min\{Z_m,Z_o\}$ "where $n$ is the number of bins used to calculate the PDF for a given region, $Z_m$ is the frequency of values in a given bin from the model, and $Z_o$ is the frequency of values in a given bin from the observed data". The expression that you are presenting is not related to the concept in the paper. $S_{score}$ is more like a goodness of fit statistic which they explain "This is a very simple measure that provides a robust and comparable measure of the relative similarity between model and observed PDFs." $\endgroup$
    – user10525
    Commented Aug 7, 2012 at 10:17
  • $\begingroup$ @Procrastinator: yes, I know, but if you take $n\to \infty$, then it is just a Riemann integral, isn't it? (the bins are equal [I just asked the author of the paper - She's in the same office :)], and it does state that the skillscore is a measure of "the common area between two PDFs"). I mean, it's still going to be calculated numerically, but writing it in a continuous fashion doesn't restrict one to using a particular numerical method.. $\endgroup$
    – naught101
    Commented Aug 8, 2012 at 0:39

1 Answer 1

Reset to default
3
+50
$\begingroup$

No, this is not related to conditional distributions. The measure you describe is a comparison of the distribution of two variables, without reference to their conditional relationship. In this sense, it's more similar to something like Kullback-Leibler divergence or the Kolmogorov-Smirnov statistic. So, to take your climate example, if the model gives precisely opposite results to the observations, i.e. high temperatures when they should be low, and vice versa, but gives the same distribution, it will have an $S_{score}$ of 1. Such a model would be excellent at climate forecasting, but poor at telling me whether to wear a coat tomorrow.

On the other hand, a measure of the conditional relationship between two variables, such as the correlation coefficient, or the mutual information, might tell you a lot about how the model and observations relate to each other, but without telling you how similar the distributions are.

Improve this answer
$\endgroup$
5
  • $\begingroup$ Interesting answer, thanks. One problem is "when they should be ..." is very difficult to define. I mean, ok, if the model is exactly right, except that summer and winter are reversed, then the pdf will be the same, and your point holds, but otherwise, it's pretty difficult to tell... $\endgroup$
    – naught101
    Commented Aug 10, 2012 at 5:44
  • $\begingroup$ Re "Without reference to their conditional relationship": I was thinking about this a bit, and I was kind of wondering about the validity of detrending the model/obs pair in some way (eg. take the global seasonal average (single value for each month of a generic year), and subtract that appropriately from the model and obs, and then check the PDF overlap. Is that somewhat related to what you mean by "conditional relationship"? (albeit only in a single dimension?) $\endgroup$
    – naught101
    Commented Aug 10, 2012 at 5:45
  • $\begingroup$ The conditional distribution describes what one variable looks like at a particular value the other variable. In this case it's the distribution of observed values over days when the model predicts a temperature T (p(o|m=T)) or the distribution of model values when we observe a temperature T (p(m|o=T)). More loosely, conditional statistics are about the properties of paired values: how each model value relates to its corresponding observation. What you suggest is "valid", but it's still a measure of how similar the distributions of those measurements are, not the measurements themselves. $\endgroup$
    – Martin O'Leary
    Commented Aug 10, 2012 at 13:39
  • $\begingroup$ Hrm. To my mind, climatology isn't interested directly in measurements, just in patterns. Monthly means and variances are more meaningful than daily measurements. Given that context, how can one look at conditional probabilities? $\endgroup$
    – naught101
    Commented Aug 15, 2012 at 2:46
  • 1
    $\begingroup$ To talk about conditional probabilities you need "pairing" between the two variables. There needs to be a correspondence where X=x when Y=y. If you don't care about that relationship, you don't care about conditional probability. One area where you probably do care about it though, is between different modelled variables. You might want to talk about the distribution of temperatures given a particular ENSO index, for example. Similarly, you might look at the distribution of T conditional on a model parameter, then use Bayes' theorem to get a distribution for that parameter, given observations. $\endgroup$
    – Martin O'Leary
    Commented Aug 15, 2012 at 3:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.