How to use cross_val_score in Scikit-Learn?

Question

I've been using cross_val_score in the Scikit-Learn package, along with Pandas dataframes and Numpy to find a 5 fold cross validation error for training a Linear Regression model on a sample data. However, I am also required to run this in combination with Best Feature Subset selection for Linear regression using Backward Stepwise selection - which I have implemented by hand (simply using loops). My main concern arose when I had to evaluate the cross validation error for each model obtained from each round (that is, each model has a reduced number of features).

I know that generally, I can find the cross-validation-error for a dataset in the following way:

seed = 7 
np.random.seed(seed) 
kfold = StratifiedKFold(n_splits=5, shuffle=True, random_state=seed) 
linreg = LinearRegression()
models[p-1]["model"] = linreg.fit(X,y) 
models[p-1]["error"] = cross_val_score(models[p-1]["model"], X, y, cv = kfold)

However, if I were to reduce the set of features and train my model accordingly like this:

models[p-1]["model"] = linreg.fit(X.ix[:,0:1],y)

Then what should be the data that I provide to cross_val? Should I do this

models[p-1]["error"] = cross_val_score(models[p-1]["model"], X.ix[:,0:1], y, cv = kfold)

or this:

models[p-1]["error"] = cross_val_score(models[p-1]["model"], X, y, cv = kfold)

Because they provide me different cross validation errors. The first one gives me an error of 0.590917074397, while the second one gives me 0.910187691851. I can't seem to understand why the difference is so huge. Also, I can't understand whether cross_val_score is selecting the proper attributes from the data when I provide the full set X to cross_val_score after training it on the subset of features.

Answer 1

"fit" fits a model against some training data so you can later do a predict with some different data.

"cross_val_score" splits the data into say 5 folds. Then for each fold it fits the data on 4 folds and scores the 5th fold. Then it gives you the 5 scores from which you can calculate a mean and variance for the score. You crossval to tune parameters and get an estimate of the score. This includes fitting, in fact it includes 5 fits!

In your case the fit is not used. The first crossval is on only 2 features. The second crossval is on all features. The high score could mean more features is a better model; or it could mean overfitting.