I am comparing two feature subsets using a linear regression on a dataset. I want to see whether feature subset 1 can generate better (more accurate) linear regression models than feature subset 2.
Currently, my method is that I shuffle the data several times (say 1000 times). Each time, I use a 5-fold cross validation to fit the linear regression models on the training set only using Feature Subset 1 and test the models obtained on the testing set, and average the 5 testing MSEs as 1 observation of the MSE. So I have 1000 MSE samples for feature subset 1. I repeat the process for feature subset 2, getting another 1000 MSE samples for feature subset 2.
Then I apply t-test to the two 1000 MSE samples to see whether they are significantly different. The dataset has around 200 samples.
Is there any problem with this method?