I am comparing two feature subsets using a linear regression on a dataset. I want to see whether feature subset 1 can generate better (more accurate) linear regression models than feature subset 2.

Currently, my method is that I shuffle the data several times (say 1000 times). Each time, I use a 5-fold cross validation to fit the linear regression models on the training set only using Feature Subset 1 and test the models obtained on the testing set, and average the 5 testing MSEs as 1 observation of the MSE. So I have 1000 MSE samples for feature subset 1. I repeat the process for feature subset 2, getting another 1000 MSE samples for feature subset 2.

Then I apply t-test to the two 1000 MSE samples to see whether they are significantly different. The dataset has around 200 samples.

Is there any problem with this method?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.