2
$\begingroup$

I split the training data into a train (75%) and test data (25%). I got an AUC for test data less than training data. does this mean overfitting ?

$\endgroup$
  • $\begingroup$ It's no surprise that the model predicts the data it was fit to more effectively than a new data set. $\endgroup$ – gammer Mar 25 '17 at 17:01
1
$\begingroup$

It's hard to detect overfitting with just one data point. Overfitting happens when the performance on the training set is improved but the performance on the test data gets worse. What you got is consistent with overfitting but it could also be due to some other random difference between the training and test data.

$\endgroup$
  • $\begingroup$ So its would be not a problem if I got an AUC for test data greater than AUC for training data ? $\endgroup$ – user44677 Mar 25 '17 at 17:07
  • 1
    $\begingroup$ Not necessarily. You don't have enough information to know if you are overfitting or not. $\endgroup$ – Aaron Mar 25 '17 at 18:26
1
$\begingroup$

The training set should in general fit the data better than the testing set. This is because this set is what your model was built on, so it has already "seen" that data, and fit itself to it. The testing set, by contrast, has not yet been "seen," and because of this should nearly always have a worse fit. In fact, this is the point of having a testing set--to test your model against unexpected data it may encounter when it is applied.

To verify that your model is good, consider running it multiple times with the data partitioning re-generated each run. This is one way to "cross-validate" your model's fit and robustness. If the results are stable, you likely didn't overfit. If they change significantly, your model does indeed do this.

As to partition size, it does not directly indicate overfitting. Partitions of 50-50, 60-40, and 80-20 are all common. However, your hunch is right in that you do make a trade-off. A larger proportion of data going into a set will always yield a more accurate model in relation to that set. By having a larger proportion of data in the training set, you allow your model to examine more cases and be more "prepared" for what it may encounter. However, yes, this can potentially create overfit as the models begins to mimic the training set too closely.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.