The error percentage of regression changes with change in the train and test data which I am deciding randomly. Cross validation can overcome this but how do I apply it for my regression model?


2 Answers 2


If I understand the question, you're looking to use a cross-validation for tuning your random forest parameters, resulting in two holdout sets:

  • one for cross-validation // model tuning
  • one for a final test (from which you generate an estimated overall performance, RMSE, MAE, etc)

Is that correct?

Assuming it is, I would suggest first splitting your dataset into two sets -- train and the rest, then split "the rest" again into two additional datasets, thereby resulting in a CV and Test dataset.

Example (Python 3.x && sklearn's train_test_split)

from sklearn.model_selection import train_test_split  

X_train, X_test, y_train, y_test = train_test_split(X_all, y_all, test_size=0.3, random_state=10)

X_cv, X_test, y_cv, y_test = train_test_split(X_test, y_test, test_size=0.5, random_state=10)

I've used a seed so the datasets would be repeatable across experiments // iterations. Note that the CV and Tests datasets are derived from the first test and that I elected to make X_Train 70% of the set and a 15% / 15% split on CV and Test.


That may be due to overfitting. Normally there is an 80 - 20 rule that advices to assign 80% of your data as a train set and the rest as a test set - so you don't have to partition them with random percentage.

another cross validation method, which seems to be the one you are suggesting is the k-fold cross validation where you partition your dataset in to k folds and iteratively use each fold as a test test, i.e. training on k-1 sets. scikit[1] learn has a kfold library which you can import as follows:

from sklearn.model_selection import KFold

[1] http://scikit-learn.org/stable/modules/generated/sklearn.model_selection.KFold.html


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.