I have ~20 datasets which contain data that I assume is, conditional on a single variable, binomially distributed. I have built a physical model of the process generating the data and can simulate data from this model. I am interested in quantifying how well this simulated data match the conditionally binomially distributed data that I have observed. Additionally, for each dataset the physical model generating the data is slightly different.

So what I essentially end up with is ~20 pairs curves like this

enter image description here

and I am looking for an appropriate and easily interpretable metric for assessing essentially how close these curves are to one another. My first thought is to calculate the Pearson-correlation between them but this is a terrible metric because I want to quantify exactly how well the data matches the observations, rather than some shifted and scaled version of this. My next thought is to just calculate the MSE between the two curves but this can be a bit hard to interpret without knowing the full data distribution.

I already know that my physical model does not perfectly reproduce the observed data, so doing a $\chi^2$ test or something like this will not be super productive.

The measure should ideally also accommodate sampling error in the values of the data curve, estimation error in the simulated curve (as pointed out in comment below).

What would be an appropriate measure in this case? Is there something very obvious I am missing?

  • $\begingroup$ Why not compare real data directly with the model you're simulating, rather than with a noisy simulated version of the model? $\endgroup$ – BruceET Jul 9 '19 at 23:14
  • $\begingroup$ Im not sure I entirely follow what you mean, sorry. Could you elaborate? Two potential answers to this comment though is because its not possible to get a closed form expression of the expected values under the model. Additionally, simulation is (computationally) cheap and I can get easily get convergent expected values by simulating. $\endgroup$ – user3235916 Jul 9 '19 at 23:23
  • $\begingroup$ Good reason. Then try to do enough iterations of the simulation that you know the hypothetical model to very good accuracy. $\endgroup$ – BruceET Jul 10 '19 at 0:38

If each curve is measured at the same value of $X1$, then you could perhaps apply some sort of distance metric? E.g. Cosine similarity, Euclidean distance, etc.

| cite | improve this answer | |
  • 2
    $\begingroup$ At a minimum, you need to accommodate sampling error in the values of the data curve, estimation error in the simulated curve, the possibility of dependence between them, and the fact that the simulated curve appears to be just a poor substitute for the estimates themselves. This all suggests such distance metrics would be inappropriate. $\endgroup$ – whuber Jul 9 '19 at 19:53
  • $\begingroup$ @whuber good points. Do you have suggestions for OP? $\endgroup$ – Demetri Pananos Jul 9 '19 at 19:55
  • 1
    $\begingroup$ @whuber Would that rule out KL-divergence? $\endgroup$ – Dave Jul 9 '19 at 19:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.