What is the validity of using a kernel density estimation to compare model x observed data?

In other words, if the KDE curve for the observed data looks like the KDE for the model forecast, can I use this result as a quality measure for the model? If not, why?

In this case, it's a weather forecast model - but I have some complex variables like wind direction, for which a simple correlation will not help too much. For example, the wind going from 360 to 5 is a huge numeric decrease, but it's almost the same direction!


I don't think you can do that. Suppose you have a vector of data [-1,0,1] and your model predicts [1,-1,0]. That has the same histogram, but you are off on all points. A marginal predictive improvement of [1,0,-1], where you get the 2nd observation correct, also has the same distribution.

| cite | improve this answer | |
  • $\begingroup$ Yes, i thought about that. But besides this, is there something useful i can take from the KDE's? $\endgroup$ – Fernando Oct 3 '13 at 0:42
  • $\begingroup$ I think the differences in KDEs can be useful. I often have data with a big spike at zero and if I don't see that in the model output, I know I am in trouble. Often I like to create bins and make a confusion matrix. If I say the wind was blowing NW, how often did I get it right and if I got it wrong, how? For your wind example, there might be some useful transformation that makes correlation-type measures or confusion matrices more interpretable. Maybe vector mean direction? Perhaps that would make a nice separate question. $\endgroup$ – Dimitriy V. Masterov Oct 3 '13 at 1:05
  • $\begingroup$ Here's an example of circular correlation coefficient. $\endgroup$ – Dimitriy V. Masterov Oct 3 '13 at 1:10
  • $\begingroup$ That's exactly what i'm using by now - the accuracy rate from the 8x8 confusion matrix (1 division at each 45 degree). Thanks for your insights, definitely going to check your refs, and maybe post a separate question later. $\endgroup$ – Fernando Oct 3 '13 at 1:15
  • $\begingroup$ Another good reference is Fisher, N.I. 1993. Statistical Analysis of Circular Data. Cambridge: Cambridge University Press. $\endgroup$ – Dimitriy V. Masterov Oct 3 '13 at 1:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.