So, I've been using cross_val_score in the Scikit-Learn package, along with Pandas dataframes and Numpy to find a 5 fold cross validation error for training a Linear Regression model on a sample data. However, I am also required to run this in combination with Best Feature Subset selection for Linear regression using Backward Stepwise selection - which I have implemented by hand (simply using loops). My main concern arose when I had to evaluate the cross validation error for each model obtained from each round (that is, each model has a reduced number of features).
I know that generally, I can find the cross-validation-error for a dataset in the following way:
seed = 7
np.random.seed(seed)
kfold = StratifiedKFold(n_splits=5, shuffle=True, random_state=seed)
linreg = LinearRegression()
models[p-1]["model"] = linreg.fit(X,y)
models[p-1]["error"] = cross_val_score(models[p-1]["model"], X, y, cv = kfold)
However, if I were to reduce the set of features and train my model accordingly like this:
models[p-1]["model"] = linreg.fit(X.ix[:,0:1],y)
Then what should be the data that I provide to cross_val? Should I do this
models[p-1]["error"] = cross_val_score(models[p-1]["model"], X.ix[:,0:1], y, cv = kfold)
or this:
models[p-1]["error"] = cross_val_score(models[p-1]["model"], X, y, cv = kfold)
Because they provide me different cross validation errors. The first one gives me an error of 0.590917074397, while the second one gives me 0.910187691851. I can't seem to understand why the difference is so huge. Also, I can't understand whether cross_val_score is selecting the proper attributes from the data when I provide the full set X to cross_val_score after training it on the subset of features.
Thanks in advance for your help. Also, please do guide me if this question has already been answered, since I have tried to look for this problem and was unable to find it.
