Like we cannot take the beeps of a faulty metal detector seriously, can we do the same in case of a poorly fit model which shows significant results? Is this analogy correct?
What can be the most objective perception about the significant outcomes from a poorly fit model?
-
$\begingroup$ Good question, but a bit too general. I would rather consider particular cases: How do you define "poorly fit" model? Bad observations for the regressors? Wrong values of the target variable? How "wrong" are them? Do those errors follow some distribution? $\endgroup$– DavidCommented May 27, 2019 at 7:13
-
$\begingroup$ Hmm. The definition of 'poorly fit' is solely based on r-squared (or adjusted r-squared). $\endgroup$– Deepan DasCommented May 27, 2019 at 7:48
1 Answer
I post now an answer based on the response to my comment (let's assume we are talking about linear regression, even though these consdierations apply to any kind of regression model):
We can say that a model with a low $R^2$ is a "poorly fit" model, in the sense that the data does not really adjust to the formula $Y=a+bX$, this happens sometimes because the relationship between $Y$ and $X$ is not actually linear, in which case you should try a different model altogether.
However, what happens often is that the "target" $Y$ has a lot of variance (either because of pure randomness or because of dependence on variables that are not included in the model as regressors) If we can not get our hands on those uncontrollable variables, then the model is just OK and you can use it for inference on which factors can be "removed" from the equation.
In order to know what type of case you are facing, the best thing I would try would be a residuals plot (in fact, most validation techniques for regression models are related to residuals). If it is a single-variable model, you can plot residuals over X, otherwise, plot residuals over "target" $Y$ and try to figure out patterns in that plot.
If you can spot any patterns (maybe residuals are negative for low values of Y but positive for high ones, or maybe they are high for medium values of Y but low in the extremes...). If you can see those patterns, you've most likely chosen the wrong type of model. If the residuals over Y (or residuals over X) plot looks like random noise, that's a good sign for your model.
Other tools to validate your model are QQ-plots, autocorrelation tests (for "ordered" data), heteroskedasticity/normality tests...
-
$\begingroup$ Upvoted for excellence of reply. $\endgroup$ Commented May 28, 2019 at 10:43