# How is ShuffleSplit supposed to work with greedy feature selection?

I'm studying machine learning and wrote some code using the ShuffleSplit method in Python with linear regression using a greedy feature selection algorithm. However, my code isn't working and I believe the problem stems from a lack of understanding I have regarding cross-validation and feature selection. Here's the code:

for i in range(X_dev.shape[1]) :
train_error_ith = []
valid_error_ith = []

if ((X_dev_fs.shape == (0,)) or (X_dev_fs.shape == (900,))):
X_dev_fs = np.append(X_dev_fs, X_dev[:, i])
X_dev_fs = X_dev_fs[:, np.newaxis]
else:
X_dev_fs = np.concatenate((X_dev_fs, X_dev[:, i].reshape(-1, 1)), axis = 1)

# For cross validation
for train_index, test_index in cv.split(X_dev) : # train_index.shape = 720, test_index.shape = 180, 5 iterations
X_train, X_valid = X_dev_fs[train_index], X_dev_fs[test_index]
y_train, y_valid = y_dev[train_index], y_dev[test_index]

model = lin_reg.fit(X_train.reshape(-1, 1), y_train.reshape(-1, 1))
lin_train_pred = model.predict(X_train.reshape(-1, 1))
lin_test_pred = model.predict(X_valid.reshape(-1, 1))
train_error_ith.append(mean_squared_error(y_train, lin_train_pred))
valid_error_ith.append(mean_squared_error(y_valid, lin_test_pred))


Ultimately, I'm trying to implement greedy feature selection with linear regression.

My algorithm is as follows:

1. Among the total number of features we have (i.e. X_dev.shape[1]), we build and train models using each one.

2. We derive the training and cross validation errors for each model corresponding to each feature.

3. We selected the model that performed best (i.e. smallest cross-validation error) and save it.

4. We continue the loop, but this time with the best feature from the previous iteration saved.

My understanding of using ShuffleSplit in this manner is that it will split the data into a specified number of splits, and we derive the training and validation errors by calculating the average of these errors across the splits.

Is the way I'm implementing it incorrect?

Any feedback is appreciated. Thank you.