Return to Answer

formatting to make text more readable

Source Link

edit approved Jun 24, 2014 at 15:51

rdorlearn

3.6k
6
29
29

Any situationAny situation: Use k-fold cross validationk-fold cross validation with some suitable number of repeats (say 5 or 10).

Splitting the data into 1 half, training on the first half and validating on the other is one step in 2-fold cross validation anyway (the other step being repeating the same exercise with the two halfs interchanged). Hence, rule out 'splitting the data into half' strategy.
Splitting the data into 1 half, training on the first half and validating on the other is one step in 2-fold cross validation anyway (the other step being repeating the same exercise with the two halfs interchanged). Hence, rule out 'splitting the data into half' strategy.
Many machine learning and data mining papers use k-fold cross validation (don't have citation), so use it unless you have to be very careful in this step.
Many machine learning and data mining papers use k-fold cross validation (don't have citation), so use it unless you have to be very careful in this step.
Now, leave one out method and other methods like 'leave p out' and 'random split and repeat' (essentially bootstrap like process described above) are defintely good contenders.
Now, leave one out method and other methods like 'leave p out' and 'random split and repeat' (essentially bootstrap like process described above) are defintely good contenders.
If your data size is N, then N-fold cross validation is essentially the same as leave one out.
If your data size is N, then N-fold cross validation is essentially the same as leave one out.
'leave p out' and 'bootstrap' are a bit more different than k fold cross validation, but the difference is essentially in how folds are defined and the number of repetitions 'k' that happen.
'leave p out' and 'bootstrap' are a bit more different than k fold cross validation, but the difference is essentially in how folds are defined and the number of repetitions 'k' that happen.
As the wiki page says, both k-fold and 'leave p out' are decent estimators of the 'expected performance/fit' (although the bets are off with regards to the variance of these estimators).
As the wiki page says, both k-fold and 'leave p out' are decent estimators of the 'expected performance/fit' (although the bets are off with regards to the variance of these estimators).

Your situation:Your situation: You only have a sample size of 200 compared to number of features (100). I think there is a very high chance that there are multiple linear models giving the same performance. I would suggest using k-fold cross validation with > 10 repeatsI would suggest using k-fold cross validation with > 10 repeats. Pick a k value of 3 or 5.

Reason for k value: generic choice.
Reason for k value: generic choice.
Reason for repeat value: A decently high value for repetition is probably critical here because the output of a single k-fold cross validation computation may be suceptible to fold splitting variability/randomness that we introduce.
Reason for repeat value: A decently high value for repetition is probably critical here because the output of a single k-fold cross validation computation may be suceptible to fold splitting variability/randomness that we introduce.

Maybe I would also employ 'leave p out' and 'bootstrap like random split repeat' methods (in addition to k-fold cross validation) for the same performance/fit measure to check if my k-fold cross validation method's outputs look alright.
Maybe I would also employ 'leave p out' and 'bootstrap like random split repeat' methods (in addition to k-fold cross validation) for the same performance/fit measure to check if my k-fold cross validation method's outputs look alright.
Although you want to use all the 100 features, as someone suggested, pay attention to multicollinearity/correlation and maybe reduce the number of features.
Although you want to use all the 100 features, as someone suggested, pay attention to multicollinearity/correlation and maybe reduce the number of features.

Any situation: Use k-fold cross validation with some suitable number of repeats (say 5 or 10).

Splitting the data into 1 half, training on the first half and validating on the other is one step in 2-fold cross validation anyway (the other step being repeating the same exercise with the two halfs interchanged). Hence, rule out 'splitting the data into half' strategy.
Many machine learning and data mining papers use k-fold cross validation (don't have citation), so use it unless you have to be very careful in this step.
Now, leave one out method and other methods like 'leave p out' and 'random split and repeat' (essentially bootstrap like process described above) are defintely good contenders.
If your data size is N, then N-fold cross validation is essentially the same as leave one out.
'leave p out' and 'bootstrap' are a bit more different than k fold cross validation, but the difference is essentially in how folds are defined and the number of repetitions 'k' that happen.
As the wiki page says, both k-fold and 'leave p out' are decent estimators of the 'expected performance/fit' (although the bets are off with regards to the variance of these estimators).

Your situation: You only have a sample size of 200 compared to number of features (100). I think there is a very high chance that there are multiple linear models giving the same performance. I would suggest using k-fold cross validation with > 10 repeats. Pick a k value of 3 or 5.

Reason for k value: generic choice.
Reason for repeat value: A decently high value for repetition is probably critical here because the output of a single k-fold cross validation computation may be suceptible to fold splitting variability/randomness that we introduce.

Maybe I would also employ 'leave p out' and 'bootstrap like random split repeat' methods (in addition to k-fold cross validation) for the same performance/fit measure to check if my k-fold cross validation method's outputs look alright.
Although you want to use all the 100 features, as someone suggested, pay attention to multicollinearity/correlation and maybe reduce the number of features.

Any situation: Use k-fold cross validation with some suitable number of repeats (say 5 or 10).

Splitting the data into 1 half, training on the first half and validating on the other is one step in 2-fold cross validation anyway (the other step being repeating the same exercise with the two halfs interchanged). Hence, rule out 'splitting the data into half' strategy.
Many machine learning and data mining papers use k-fold cross validation (don't have citation), so use it unless you have to be very careful in this step.
Now, leave one out method and other methods like 'leave p out' and 'random split and repeat' (essentially bootstrap like process described above) are defintely good contenders.
If your data size is N, then N-fold cross validation is essentially the same as leave one out.
'leave p out' and 'bootstrap' are a bit more different than k fold cross validation, but the difference is essentially in how folds are defined and the number of repetitions 'k' that happen.
As the wiki page says, both k-fold and 'leave p out' are decent estimators of the 'expected performance/fit' (although the bets are off with regards to the variance of these estimators).

Your situation: You only have a sample size of 200 compared to number of features (100). I think there is a very high chance that there are multiple linear models giving the same performance. I would suggest using k-fold cross validation with > 10 repeats. Pick a k value of 3 or 5.

Reason for k value: generic choice.
Reason for repeat value: A decently high value for repetition is probably critical here because the output of a single k-fold cross validation computation may be suceptible to fold splitting variability/randomness that we introduce.

Maybe I would also employ 'leave p out' and 'bootstrap like random split repeat' methods (in addition to k-fold cross validation) for the same performance/fit measure to check if my k-fold cross validation method's outputs look alright.
Although you want to use all the 100 features, as someone suggested, pay attention to multicollinearity/correlation and maybe reduce the number of features.

Source Link

created Jun 20, 2014 at 12:43

Theja Tulabandhula

Please refer to the wikipedia page for the method definitions (they do a far better job than I could do here).

After you have had a look at that page, the following may be of help to you. Let me focus on the part of the question where one wants to pick one of these methods for their modeling process. Since this is pretty frequent choice that one makes, and they could benefit from additional knowledge, here is my answer for two situations:

Any situation: Use k-fold cross validation with some suitable number of repeats (say 5 or 10).

Splitting the data into 1 half, training on the first half and validating on the other is one step in 2-fold cross validation anyway (the other step being repeating the same exercise with the two halfs interchanged). Hence, rule out 'splitting the data into half' strategy.
Many machine learning and data mining papers use k-fold cross validation (don't have citation), so use it unless you have to be very careful in this step.
Now, leave one out method and other methods like 'leave p out' and 'random split and repeat' (essentially bootstrap like process described above) are defintely good contenders.
If your data size is N, then N-fold cross validation is essentially the same as leave one out.
'leave p out' and 'bootstrap' are a bit more different than k fold cross validation, but the difference is essentially in how folds are defined and the number of repetitions 'k' that happen.
As the wiki page says, both k-fold and 'leave p out' are decent estimators of the 'expected performance/fit' (although the bets are off with regards to the variance of these estimators).

Your situation: You only have a sample size of 200 compared to number of features (100). I think there is a very high chance that there are multiple linear models giving the same performance. I would suggest using k-fold cross validation with > 10 repeats. Pick a k value of 3 or 5.

Reason for k value: generic choice.
Reason for repeat value: A decently high value for repetition is probably critical here because the output of a single k-fold cross validation computation may be suceptible to fold splitting variability/randomness that we introduce.

Additional thoughts:

Maybe I would also employ 'leave p out' and 'bootstrap like random split repeat' methods (in addition to k-fold cross validation) for the same performance/fit measure to check if my k-fold cross validation method's outputs look alright.
Although you want to use all the 100 features, as someone suggested, pay attention to multicollinearity/correlation and maybe reduce the number of features.