3
$\begingroup$

My main field is machine learning and 90 % of what I do is to try to improve the prediction error.

Recently I have started to work with a medical group. They are mainly doctors so I do not know how much I can trust their statistical knowledge.

What they commonly do is to fit logistic regression and without any consideration of the model performance they start to look to the coefficient in order to understand which one is more important or stuff like that.

In a similar way I fitted a random forest model and the I have done the importance plot something like this even if the predicted performance was very poor.

How much can I trust these kind of results in model where the prediction performance is poor?

Improve this question
$\endgroup$
  • 3
    $\begingroup$ Models have more than one possible purpose. Is the purpose of the model prediction? $\endgroup$ – Glen_b May 11 '14 at 10:17
  • $\begingroup$ no necessarily. Now I was mainly interested in how much a variable influences the output. The random forest ranking was perfect but I do not know if I can trust it due the fact that the prediction error is so large $\endgroup$ – Donbeo May 11 '14 at 10:28
  • 1
    $\begingroup$ Related to Glen_b's comment: darden.virginia.edu/web/uploadedFiles/Darden/Faculty_Research/… $\endgroup$ – boscovich May 11 '14 at 10:31
  • $\begingroup$ Is it possible that they are using a scoring rule or some likelihood measure like generalized $R^2$ instead of prediction accuracy? Different measures don't always agree on which model is the best. $\endgroup$ – user44764 May 24 '14 at 2:21
2
$\begingroup$

Depends on what you are looking for.

First of all, we have to define what "trusting a model" is. From a mathematical point of view, this mean : "are you using the correct model according to your data ?". In this case, you should not be using the prediction error to assess the quality of your fit. The only thing we can do is "checking that we have no evidence that the assumptions we made for this model are violated". In the case of linear regression, we usually check the normality of the error distribution, linearity of the relationship between covariates and response variable, homoscedasticity and the independence of errors.

As Glen_b said, models have more than one possible purpose. If you think about prediction performance, I'm pretty sure we can find an example where they are good but the model is not. This case could be very problematic if you think that you are using the correct model. This could lead to wrong prediction for data that are far away from your original data.

There are many other things you have to be careful about (i.e. number of observations, variance of your observations, etc).

Improve this answer
$\endgroup$
  • $\begingroup$ Thanks for the answer. I have a dataset that does not respect any assumption. I have something like 10 cathegorical variables and 3 continuous variables. The output is binary. In my knowledge there are no particular assumption on random forest. How can I ensure that I can trust these results? $\endgroup$ – Donbeo May 23 '14 at 21:03
  • $\begingroup$ My answer was more general than answering for random forest. Sorry about that. I have no experience with random forest, but I found this stackexchange's answer that might interest you Random Forest Assumptions $\endgroup$ – Julien D. May 23 '14 at 21:19
  • $\begingroup$ This is interesting but it's not the answer yet I give you a point $\endgroup$ – Donbeo May 23 '14 at 21:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.