I've been using cross_val_score in the Scikit-Learn package, along with Pandas dataframes and Numpy to find a 5 fold cross validation error for training a Linear Regression model on a sample data. However, I am also required to run this in combination with Best Feature Subset selection for Linear regression using Backward Stepwise selection - which I have implemented by hand (simply using loops). My main concern arose when I had to evaluate the cross validation error for each model obtained from each round (that is, each model has a reduced number of features).

I know that generally, I can find the cross-validation-error for a dataset in the following way:

seed = 7 
kfold = StratifiedKFold(n_splits=5, shuffle=True, random_state=seed) 
linreg = LinearRegression()
models[p-1]["model"] = linreg.fit(X,y) 
models[p-1]["error"] = cross_val_score(models[p-1]["model"], X, y, cv = kfold)

However, if I were to reduce the set of features and train my model accordingly like this:

models[p-1]["model"] = linreg.fit(X.ix[:,0:1],y)

Then what should be the data that I provide to cross_val? Should I do this

models[p-1]["error"] = cross_val_score(models[p-1]["model"], X.ix[:,0:1], y, cv = kfold)

or this:

models[p-1]["error"] = cross_val_score(models[p-1]["model"], X, y, cv = kfold)

Because they provide me different cross validation errors. The first one gives me an error of 0.590917074397, while the second one gives me 0.910187691851. I can't seem to understand why the difference is so huge. Also, I can't understand whether cross_val_score is selecting the proper attributes from the data when I provide the full set X to cross_val_score after training it on the subset of features.


1 Answer 1


"fit" fits a model against some training data so you can later do a predict with some different data.

"cross_val_score" splits the data into say 5 folds. Then for each fold it fits the data on 4 folds and scores the 5th fold. Then it gives you the 5 scores from which you can calculate a mean and variance for the score. You crossval to tune parameters and get an estimate of the score. This includes fitting, in fact it includes 5 fits!

In your case the fit is not used. The first crossval is on only 2 features. The second crossval is on all features. The high score could mean more features is a better model; or it could mean overfitting.

  • $\begingroup$ Thanks for the reply Simon. I actually figured out the problem later - turns out, the parameter for "model" in cross_val_score is simply an object of the classifier/model, and nothing else - irrespective of whether it has been used for fitting or not. So, all I needed to do was pass an object of the classifier, the desired dataset, and the target values to the function to get my cross validation score. $\endgroup$ Nov 24, 2016 at 17:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.