Suppose we have a linear model fit to real world data and we access the quality of the fit by examining $r^2$, the p-values of the regression coefficients, the normality of the errors, the Cook's distances,... All the usual metrics. Once we are reasonably satisfied with the fit, we use the model to interpret the data. Now suppose that we come across some randomly generated data, which when we fit the model to it, gives very similar results on all the diagnostic techniques as well as a similar regression line. A colleague of mine draws the conclusion that we can't trust any interpretations the model gives about the real world data because we could then apply the same interpretations to the random data and we know the interpretations of the random data to be wrong (since it's random and not from a real world process). Thus since the real world data looks like the random data we can't trust any conclusions the model gives. Every part of my colleague's reasoning feels wrong to me, but I am having a hard time making an argument to convince him of flaws in his logic. Can someone make some good arguments against his approach or point out any flaws in my own reasoning?
1 Answer
Your colleague's raised suspicions would be justified, depending on how this random data was sourced*: what is the probability that the next set of random data (with the same dimensions) would result in a model with the same fit if the process were repeated many times? Sure, by chance, random data could mirror any real-world dataset, so the question is with what frequency this random data generation procedure continues to result in a similar model.
If you are able to show that random data continues to result in this model, then perhaps at least one of the following is true:
- the random data generation procedure itself needs to be questioned
- you're working in a domain where it's expected that the model will have very little predictive power, and noise can easily be confused for signal
- the model building process is flawed
- you accepted a model that provides almost no predictive ability
At the very least, you'd want to re-run the trial as much as possible and see how often you repeat these results.
Caveat: this was vaguely worded:
"we come across some randomly generated data"
How did you "come across" this data? If it was through a single trial conducted to explicitly generate random data for a test, that's valid. If you selected the first random set of data from a large pool that demonstrated a significant result, that's another story.
-
$\begingroup$ The problem is that he is only looking at one instance in which the random data matches the real world data, not multiple instance. $\endgroup$ Commented Oct 9, 2016 at 17:22
-
$\begingroup$ @Wintermute yes, so we need to understand the probability that this single observation implies something is wrong; if we start by assuming a prior probability of 90% confidence that we have a valid model, then this new information (one trial) should significantly reduce the posterior probability; the reduction in probability could be mathematically derived, but in practice is much easier to estimate through experimentation, I believe. $\endgroup$ Commented Oct 9, 2016 at 17:28
-
$\begingroup$ I wonder if your initial sentence was supposed to read: "Your colleague's raised suspicions would not necessarily be justified". The bulk of your answer does not seem to support the idea that the colleague is ultimately correct in the OP's situation. $\endgroup$ Commented Oct 11, 2016 at 15:56
-
$\begingroup$ I do mean necessarily, but added an important caveat $\endgroup$ Commented Oct 11, 2016 at 17:16