Can I trust a model if I do not check the prediction error?

My main field is machine learning and 90 % of what I do is to try to improve the prediction error.

Recently I have started to work with a medical group. They are mainly doctors so I do not know how much I can trust their statistical knowledge.

What they commonly do is to fit logistic regression and without any consideration of the model performance they start to look to the coefficient in order to understand which one is more important or stuff like that.

In a similar way I fitted a random forest model and the I have done the importance plot something like this even if the predicted performance was very poor.

How much can I trust these kind of results in model where the prediction performance is poor?

• Models have more than one possible purpose. Is the purpose of the model prediction? – Glen_b May 11 '14 at 10:17
• no necessarily. Now I was mainly interested in how much a variable influences the output. The random forest ranking was perfect but I do not know if I can trust it due the fact that the prediction error is so large – Donbeo May 11 '14 at 10:28
• Related to Glen_b's comment: darden.virginia.edu/web/uploadedFiles/Darden/Faculty_Research/… – boscovich May 11 '14 at 10:31
• Is it possible that they are using a scoring rule or some likelihood measure like generalized $R^2$ instead of prediction accuracy? Different measures don't always agree on which model is the best. – user44764 May 24 '14 at 2:21