Is it possible to compare between these two types of model? I have a set of data that involves 6 independent variables and 1 dependent variable. It is based of a questionnaire for social science students. I ran the assumption tests for linear regression and the linearity shows vague results, but somehow passes as a linear model. Thus, I would like to propose that fitting it into a modified Gompertz model gives a better fit and better predictions. I calculated the RMSE for both the linear and nonlinear model, and results shows that the Gompertz model has a smaller RMSE. Is this acceptable or is this impossible even to begin with?
$\begingroup$
$\endgroup$
3
-
1$\begingroup$ Smaller RMSE is a good signal unless it is due to overfitting. You could compare the AIC or BIC values of your models to get a more objective measure of the improvement in RMSE. $\endgroup$– Richard HardyCommented Mar 3, 2015 at 8:06
-
$\begingroup$ How many parameters in each model? $\endgroup$– Glen_bCommented Mar 3, 2015 at 8:26
-
$\begingroup$ parameters as in unknown? seven. $\endgroup$– jojoCommented Mar 3, 2015 at 9:10
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
3
I presume the Gompertz model you fitted has more parameters than the linear model (you should be explicit about the exact model, since there's more than one model I might call a Gompertz model).
1) If that's the case you would expect the model with more degrees of freedom to have smaller RMSE.
2) It looks like the form of the nonlinear model was chosen after seeing the data. Even if the number of parameters were the same, you would expect it to have smaller RMSE.
So I wouldn't recommend RMSE for either of those reasons.
If you hadn't seen the data before choosing to fit the Gompertz model, you might compare AIC, or BIC or some similar criterion, but this would still have an issue of model selection which can impact your inference.
You might look to assessing out-of-sample predictive ability -- say via cross-validation (of MSE or RMSE perhaps). That will reduce, but not completely eliminate the effect of having seen the data, and avoid the issue of different numbers of parameters.
-
$\begingroup$ I guess it should be a separate question, but before I post it, maybe I will have some instant luck here. Curiously, Diebold (2013 working paper) (see sections 5 and 6) suggests that full sample should be used on most occasions and out-of-sample assessments are little more than a waste of data. (This sounds like a very general statement and may be misleading, but Diebold puts its very clearly in just a few pages.) I would be very curious to hear your opinion... I will post an excerpt from section 5 in the next comment. $\endgroup$ Commented Mar 3, 2015 at 8:56
-
$\begingroup$ [Can] any pseudo-out-of-sample model comparison procedure [...] compete in terms of consistency with the full-sample procedures [...][?] The answer turns out to be yes, but simultaneously there are simpler procedures with the same asymptotic justification and likely-superior finite-sample properties (SIC being one example). A second key question [...] that drives the pseudo-out-of-sample model comparison literature [...] is whether pseudo-out-of-sample procedures can help insure against in-sample overfitting, or “data mining,” in finite samples [...]. The answer is no (and the result is not new). $\endgroup$ Commented Mar 3, 2015 at 8:58
-
$\begingroup$ @Richard Your first impulse is the correct one - it would be a good question to post. $\endgroup$– Glen_bCommented Mar 3, 2015 at 9:12