1
$\begingroup$

I am comparing two feature subsets using a linear regression on a dataset. I want to see whether feature subset 1 can generate better (more accurate) linear regression models than feature subset 2.

Currently, my method is that I shuffle the data several times (say 1000 times). Each time, I use a 5-fold cross validation to fit the linear regression models on the training set only using Feature Subset 1 and test the models obtained on the testing set, and average the 5 testing MSEs as 1 observation of the MSE. So I have 1000 MSE samples for feature subset 1. I repeat the process for feature subset 2, getting another 1000 MSE samples for feature subset 2.

Then I apply t-test to the two 1000 MSE samples to see whether they are significantly different. The dataset has around 200 samples.

Is there any problem with this method?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.