2
$\begingroup$

I have ~20 datasets which contain data that I assume is, conditional on a single variable, binomially distributed. I have built a physical model of the process generating the data and can simulate data from this model. I am interested in quantifying how well this simulated data match the conditionally binomially distributed data that I have observed. Additionally, for each dataset the physical model generating the data is slightly different.

So what I essentially end up with is ~20 pairs curves like this

enter image description here

and I am looking for an appropriate and easily interpretable metric for assessing essentially how close these curves are to one another. My first thought is to calculate the Pearson-correlation between them but this is a terrible metric because I want to quantify exactly how well the data matches the observations, rather than some shifted and scaled version of this. My next thought is to just calculate the MSE between the two curves but this can be a bit hard to interpret without knowing the full data distribution.

I already know that my physical model does not perfectly reproduce the observed data, so doing a $\chi^2$ test or something like this will not be super productive.

The measure should ideally also accommodate sampling error in the values of the data curve, estimation error in the simulated curve (as pointed out in comment below).

What would be an appropriate measure in this case? Is there something very obvious I am missing?

$\endgroup$
3
  • $\begingroup$ Why not compare real data directly with the model you're simulating, rather than with a noisy simulated version of the model? $\endgroup$
    – BruceET
    Commented Jul 9, 2019 at 23:14
  • $\begingroup$ Im not sure I entirely follow what you mean, sorry. Could you elaborate? Two potential answers to this comment though is because its not possible to get a closed form expression of the expected values under the model. Additionally, simulation is (computationally) cheap and I can get easily get convergent expected values by simulating. $\endgroup$ Commented Jul 9, 2019 at 23:23
  • $\begingroup$ Good reason. Then try to do enough iterations of the simulation that you know the hypothetical model to very good accuracy. $\endgroup$
    – BruceET
    Commented Jul 10, 2019 at 0:38

1 Answer 1

0
$\begingroup$

If each curve is measured at the same value of $X1$, then you could perhaps apply some sort of distance metric? E.g. Cosine similarity, Euclidean distance, etc.

$\endgroup$
3
  • 2
    $\begingroup$ At a minimum, you need to accommodate sampling error in the values of the data curve, estimation error in the simulated curve, the possibility of dependence between them, and the fact that the simulated curve appears to be just a poor substitute for the estimates themselves. This all suggests such distance metrics would be inappropriate. $\endgroup$
    – whuber
    Commented Jul 9, 2019 at 19:53
  • $\begingroup$ @whuber good points. Do you have suggestions for OP? $\endgroup$ Commented Jul 9, 2019 at 19:55
  • 1
    $\begingroup$ @whuber Would that rule out KL-divergence? $\endgroup$
    – Dave
    Commented Jul 9, 2019 at 19:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.