1
$\begingroup$

Suppose we have a training data set. We want to learn some hypothesis using some algorithm. Would we divide up the training set differently if we used, for example, logistic regression as opposed to support vector machines?

So if we divide the training data set into: $70 \%$ in-sample data and $30 \%$ out-of-sample data, would this hold if we used logistic regression or support vector machines?

$\endgroup$

3 Answers 3

1
$\begingroup$

If this is your standard scenario: trying a bunch of algorithms to see which one performs best in the test data, you obviously need to use the same kind of split. Your comparison would be useless if different algorithms were tested with different amount of data. The one with the largest training data would have an advantage.
Just pick k-fold or leave-one-out. They are both fine and easy to implement.

Now, you could add some more considerations if most of the algorithms you want to test are similar. Support vector machines take a long time to fit so if you are only testing those, stay away from leave-one-out since that's the one with the most fittings. On the other hand, if you have very little data you don't want to remove too many elements from the training data then leave-one-out would become preferrable.

$\endgroup$
0
$\begingroup$

No, it wouldn't change. In fact, testing one method against another using the same test/training split is kind of the point, so you can determine which algorithm generalizes better.

$\endgroup$
0
$\begingroup$

Depending on the method you use, the variables you set and the whole environment in general you run a bigger or lower risk of overfitting. So actually, splitting your data into training and test set is a good practice to verify how 'general' is the estimate your model gives for new input, this is: how well does it understand the relation between features of your data and how far is it from being to biased to your initial observations (training set).

So if you use a linear regressor with a 40-degree polynomial the obtained model would probably be more biased onto your training data that a 3-degree polynomial (this depends hugely on your dataset size and complexity). The same does for SVM: the kernel you use and in general the way you transform data will impact the results. It can be said that for most cases SVM's are a very robust method against overfitting, at the contrary of linear regressors.

The result does not depend that greatly on how you split you dataset, but on how good is data, how well is represented for the algorithm you use and how good are your experiments to find near-optimal parameters.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.