Suppose we have a linear model fit to real world data and we access the quality of the fit by examining $r^2$, the p-values of the regression coefficients, the normality of the errors, the Cook's distances,... All the usual metrics. Once we are reasonably satisfied with the fit, we use the model to interpret the data. Now suppose that we come across some randomly generated data, which when we fit the model to it, gives very similar results on all the diagnostic techniques as well as a similar regression line. A colleague of mine draws the conclusion that we can't trust any interpretations the model gives about the real world data because we could then apply the same interpretations to the random data and we know the interpretations of the random data to be wrong (since it's random and not from a real world process). Thus since the real world data looks like the random data we can't trust any conclusions the model gives. Every part of my colleague's reasoning feels wrong to me, but I am having a hard time making an argument to convince him of flaws in his logic. Can someone make some good arguments against his approach or point out any flaws in my own reasoning?
Your colleague's raised suspicions would be justified, depending on how this random data was sourced*: what is the probability that the next set of random data (with the same dimensions) would result in a model with the same fit if the process were repeated many times? Sure, by chance, random data could mirror any real-world dataset, so the question is with what frequency this random data generation procedure continues to result in a similar model.
If you are able to show that random data continues to result in this model, then perhaps at least one of the following is true:
- the random data generation procedure itself needs to be questioned
- you're working in a domain where it's expected that the model will have very little predictive power, and noise can easily be confused for signal
- the model building process is flawed
- you accepted a model that provides almost no predictive ability
At the very least, you'd want to re-run the trial as much as possible and see how often you repeat these results.
Caveat: this was vaguely worded:
"we come across some randomly generated data"
How did you "come across" this data? If it was through a single trial conducted to explicitly generate random data for a test, that's valid. If you selected the first random set of data from a large pool that demonstrated a significant result, that's another story.