3
$\begingroup$

What is the validity of using a kernel density estimation to compare model x observed data?

In other words, if the KDE curve for the observed data looks like the KDE for the model forecast, can I use this result as a quality measure for the model? If not, why?

In this case, it's a weather forecast model - but I have some complex variables like wind direction, for which a simple correlation will not help too much. For example, the wind going from 360 to 5 is a huge numeric decrease, but it's almost the same direction!

Improve this question
$\endgroup$

1 Answer 1

Reset to default
3
$\begingroup$

I don't think you can do that. Suppose you have a vector of data [-1,0,1] and your model predicts [1,-1,0]. That has the same histogram, but you are off on all points. A marginal predictive improvement of [1,0,-1], where you get the 2nd observation correct, also has the same distribution.

Improve this answer
$\endgroup$
5
  • $\begingroup$ Yes, i thought about that. But besides this, is there something useful i can take from the KDE's? $\endgroup$
    – Fernando
    Commented Oct 3, 2013 at 0:42
  • $\begingroup$ I think the differences in KDEs can be useful. I often have data with a big spike at zero and if I don't see that in the model output, I know I am in trouble. Often I like to create bins and make a confusion matrix. If I say the wind was blowing NW, how often did I get it right and if I got it wrong, how? For your wind example, there might be some useful transformation that makes correlation-type measures or confusion matrices more interpretable. Maybe vector mean direction? Perhaps that would make a nice separate question. $\endgroup$
    – dimitriy
    Commented Oct 3, 2013 at 1:05
  • $\begingroup$ Here's an example of circular correlation coefficient. $\endgroup$
    – dimitriy
    Commented Oct 3, 2013 at 1:10
  • $\begingroup$ That's exactly what i'm using by now - the accuracy rate from the 8x8 confusion matrix (1 division at each 45 degree). Thanks for your insights, definitely going to check your refs, and maybe post a separate question later. $\endgroup$
    – Fernando
    Commented Oct 3, 2013 at 1:15
  • $\begingroup$ Another good reference is Fisher, N.I. 1993. Statistical Analysis of Circular Data. Cambridge: Cambridge University Press. $\endgroup$
    – dimitriy
    Commented Oct 3, 2013 at 1:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.