4
$\begingroup$

I know the goal of machine learning is to create generalizable models and therefore overfitting is undesirable.

However, I wonder if it could be desirable in some cases. For example, let's say I want to predict if a student will dropout a course, and I want to do this before the end of the course by using a proxy label, their assignment submission status. In this scenario, I would not mind if the model trained on the proxy label overfits and does not generalize well to the unseen data, since I care about only a specific set of users.

I wonder if this is a valid way of thinking for this specific scenario. Any ideas?

Improve this question
$\endgroup$
3
  • 2
    $\begingroup$ You mean you only want to have a kind of efficient way of memorizing the data (in that case overfitting is indeed good)? Or do you actually want to predict new data from the same users (only a certain type of overfitting is good - this may e.g. change your cross-validation strategy)? $\endgroup$
    – Björn
    Commented Oct 19, 2018 at 12:57
  • 1
    $\begingroup$ See this post for an interesting viewpoint $\endgroup$
    – kjetil b halvorsen ♦
    Commented Oct 19, 2018 at 13:32
  • $\begingroup$ In your example, where does the function that turns the current assignment submission status into the probability of future dropout come from? $\endgroup$
    – dimitriy
    Commented Feb 15, 2023 at 21:39

3 Answers 3

Reset to default
4
$\begingroup$

Actually the labels "generalization" and "overfitting" might be a bit misleading here.

What you want in your example is a good prediction of the dropout status.

So technically: In training you therefore need to have an unbiased sample of dropout and non-dropout-students. It is extremely important to prepare not only the model, but even more the data you are using to evaluate (train, validate, etc.).

There are text-book examples of overfitting, where you can e.g. plot a performance indicator (e.g. mislabelling rate) of the your training data and compare it with validation data. The training performance will always become better, but at some point the validation will become worse. There it is probably very clear that you'd rather stop the learning process before it worsens the performance.

What is meant by "generalization" is actually very specific. You want your trained model to be the best possible once it encounters previously unseen data. You use the validation data, because there you know "the truth". Unlike with real data.

So still as above: you want your model to give a prediction of the students status. - if your model is overfitted, it will give higher valued indicators for the students in your training set; but will perform worse on non-trained data - if you model has good generalization power; it will perform equally well on training data, as well as non-training data

If you talk about "specific data sets", then either these are the basis of your training and validation; or you simply do it wrong. And this has nothing to do with generalization or overfitting in machine learning.

Improve this answer
$\endgroup$
3
$\begingroup$

Does your trained model performs effective on both training and test data? when it's so then its not overfitting. If your model performs well in training data but worse in test data, then it is overfitting which wouldn't make sense for any application. If you mean that your training data and test data are same, then yeah overfitting wouldn't be an issue there. But if you want to work on unseen test data, then overfitting is always a problem.

Improve this answer
$\endgroup$
0
$\begingroup$

another example of this I can think of is when you want to only identify historic patterns repeating themselves(like in the stock market), where I don't care whether new patterns happen or not, as long as I have all previous patterns memorized and if they repeat I get a signal... Specially if my training data is all of the market upto this point

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.