3
$\begingroup$

I am using RandomForestClassifier from scikit-learn and I get the following results:

  1. "accuracy of 77%" with a train/test split 75/25%.
  2. "accuracy of about 76%" with cross-validation (cv=5) using the entire sample.

I am happy so far...

However, what do I do now? I want a final shippable model to pickle and use it on data, give it to a friend and say "this is the best I could do...". So,

  1. Do I train on all X,y? and joblib.dump(clf, 'final_shippable_supermodel.pkl')
  2. Do I train on using X_train, y_train from a random split?

ALSO, note that I understand 77% might not meet the standards of those who answer. I am just trying to understand the steps. An, yes, I have good precision and recall scores.

|cite|improve this question
$\endgroup$
  • 1
    $\begingroup$ If an answer solves your problem, consider to mark it as accepted (see stats.stackexchange.com/help/accepted-answer). If other users see you have a positive acceptance rate (if possible, of course ;)), then you increase the probability of getting more answers. $\endgroup$ – steffen Mar 14 '18 at 12:55
  • 1
    $\begingroup$ This is a sensitive issue i cant vote that it solved my problem yet. i am going though literature still. if it is adequate of course i will mark as accepted $\endgroup$ – George Pamfilis Mar 15 '18 at 15:30
4
$\begingroup$

Allow me to paraphrase your question:

Do I re-train on all $X, y$ before handing over my final model?

Yes. And you could report the performance measures you found: test set performance and cross-validation.

These estimates do not suffer from optimism bias with respect to out-of-sample performance.$^\text{1}$ They are preferred to in-sample measures of fit that can, and often do, suffer from optimism bias.

Lastly, note that random forests come with a natural estimate of out-of-sample prediction performance: the out-of-bag estimator (OOB). Simply set oob_score = True. I suspect, that its value will be similar to the ones you already obtained.

$\!^\text{1}$ In fact, they are even (slightly) pessimistic with respect to the performance of the model trained on the entire sample ("all data"), as a model's ability to generalize improves upon "seeing" more examples.

|cite|improve this answer
$\endgroup$
  • $\begingroup$ What if I were to use one, or the best model from cross validation? statistically, it gave me consistent scores so what is wrong with this new idea? By the way i do agree with you answer. $\endgroup$ – George Pamfilis Mar 5 '18 at 12:00
  • $\begingroup$ @GeorgePamfilis the problem would be that that specific forest is evaluated (validated) at only $1/5$th of the data, thereby making its generalization estimate more variable than using each observation for evaluation (validation) at least once. So that possibly, say, the third CV split looks like the best forest, while actually it was the result of chance (random variation). Concluding, I would go for OOB: it uses the entire sample and you get it for free. $\endgroup$ – Jim Mar 5 '18 at 17:01
1
$\begingroup$

Once you tested your model and saw that the accuracy (or whatever metric you use) is good enough, then you can re-train your model with all your data before shipping it.

Think, for instance, in a simple split training/test. You keep a test set because you need to know how well your model generalizes to unseen data. Otherwise you would be fooling yourself, since the model might be overfitting (memorizing) the training set. However, once you know that you have a good model, you can use your test data to make your model more robust.

|cite|improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.