0
$\begingroup$

I have a simple linear model that predicts an outcome based on a few input variables (e.g. y = a*x + b), which are based on theory (psychology). None of the variables are free parameters, meaning there is no need to fit the model. I would like to assess the (predictive) quality of the model by comparing it to the data (and possibly also to a more complex model containing a logarithmic term that I have fitted to the data first). However, since the model is not fitted, regular measures of model quality (e.g. R2, AIC, BIC) are not applicable. I calculated the RMSE, but it is not very informative as a standalone value. So here is the question:

(1) Is there a more informative way to answer the question whether the model with zero free parameters is able to predict the data well?

Similarly (2) if I want to compare the simple model with zero free parameters to the model with 1 free parameter, is there a better way than simply saying one model has a lower mean RMSE according to a cross validation? Especially as the simple model seems to be at a disadvantage because it is not fitted.

$\endgroup$
0
$\begingroup$

I have tried to find a solution to this myself and have come up with the following partial answer:

(1) To assess the relative model quality, a model comparison approach that includes plausible alternative models can be used. With this, it is possible to say that the model containig zero free parameters is better (or worse) than these alternatives at least.

(2) To have a more informative model comparison, the model quality in terms of RMSE can be assessed on the participant level. For example, it can be said that for 28% of the study participants, one model makes the most accurate predictions and for 72% of the participants, the other model makes the most accurate predictions.

Still missing: An appropriate way to mitigate the disadvantage of the model with zero free parameters in comparison to a more flexible model that has been fitted to the data before.

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.