3
$\begingroup$

Suppose we have a linear model fit to real world data and we access the quality of the fit by examining $r^2$, the p-values of the regression coefficients, the normality of the errors, the Cook's distances,... All the usual metrics. Once we are reasonably satisfied with the fit, we use the model to interpret the data. Now suppose that we come across some randomly generated data, which when we fit the model to it, gives very similar results on all the diagnostic techniques as well as a similar regression line. A colleague of mine draws the conclusion that we can't trust any interpretations the model gives about the real world data because we could then apply the same interpretations to the random data and we know the interpretations of the random data to be wrong (since it's random and not from a real world process). Thus since the real world data looks like the random data we can't trust any conclusions the model gives. Every part of my colleague's reasoning feels wrong to me, but I am having a hard time making an argument to convince him of flaws in his logic. Can someone make some good arguments against his approach or point out any flaws in my own reasoning?

$\endgroup$
1
$\begingroup$

Your colleague's raised suspicions would be justified, depending on how this random data was sourced*: what is the probability that the next set of random data (with the same dimensions) would result in a model with the same fit if the process were repeated many times? Sure, by chance, random data could mirror any real-world dataset, so the question is with what frequency this random data generation procedure continues to result in a similar model.

If you are able to show that random data continues to result in this model, then perhaps at least one of the following is true:

  • the random data generation procedure itself needs to be questioned
  • you're working in a domain where it's expected that the model will have very little predictive power, and noise can easily be confused for signal
  • the model building process is flawed
  • you accepted a model that provides almost no predictive ability

At the very least, you'd want to re-run the trial as much as possible and see how often you repeat these results.

Caveat: this was vaguely worded:

"we come across some randomly generated data"

How did you "come across" this data? If it was through a single trial conducted to explicitly generate random data for a test, that's valid. If you selected the first random set of data from a large pool that demonstrated a significant result, that's another story.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ The problem is that he is only looking at one instance in which the random data matches the real world data, not multiple instance. $\endgroup$ – Wintermute Oct 9 '16 at 17:22
  • $\begingroup$ @Wintermute yes, so we need to understand the probability that this single observation implies something is wrong; if we start by assuming a prior probability of 90% confidence that we have a valid model, then this new information (one trial) should significantly reduce the posterior probability; the reduction in probability could be mathematically derived, but in practice is much easier to estimate through experimentation, I believe. $\endgroup$ – Brian Bien Oct 9 '16 at 17:28
  • $\begingroup$ I wonder if your initial sentence was supposed to read: "Your colleague's raised suspicions would not necessarily be justified". The bulk of your answer does not seem to support the idea that the colleague is ultimately correct in the OP's situation. $\endgroup$ – gung - Reinstate Monica Oct 11 '16 at 15:56
  • $\begingroup$ I do mean necessarily, but added an important caveat $\endgroup$ – Brian Bien Oct 11 '16 at 17:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.