13
$\begingroup$

I understand that using cross validation we can validate our model, but it is also possible that maybe our model is underfitting; hence, providing wrong results. One possibility that I can think of is that if even after making our model less complex results are not right then our model might be underfitting, it is just a guess!
Please write if anything I am missing or any other way to decide whether our model is overfitting or underfitting?

$\endgroup$

2 Answers 2

8
$\begingroup$

You can determine the difference between an underfitting and overfitting experimentally by comparing fitted models to training-data and test-data.

Typical graphs:

a

b

c

d


These plots will show you the accuracy of the model, as function of some parameter (e.g. 'complexity'), for both the

  • training-data (the data use for fitting)
  • and test data (data kept separate from the training phase)

One normally chooses the model that does the best on the test-data. If you deviate then for...

  • For underfitting models, you do worse because they do not capture the true trend sufficiently.

    If you get more underfitting then you get both worse fits for training and testing data.

  • for overfitting models, you do worse because they respond too much to the noise, rather than the true trend.

    If you get more overfitting then you get better fits for training data (capturing the noise, but it is useless or even detrimental), but still worse for testing data.

After performing these steps, you should remain in sight that, while you may have selected the least underfitting/overfitting model out of your options, this does not mean that there would not be some other better model.

In particular: the true model, might be an overfitting model in circumstances of high noise-signal ratios (see https://stats.stackexchange.com/a/299523/164061 )

$\endgroup$
5
  • $\begingroup$ The first plot seems to show error over training time. $\endgroup$
    – Tim
    Jul 12, 2018 at 12:16
  • $\begingroup$ @Tim Training time makes a model (e.g. an artificial neural network) better in fitting the training data (increases complexity) but not necessarily the testing data. Thus it is related to overfitting vs. underfitting. $\endgroup$ Jul 12, 2018 at 12:25
  • $\begingroup$ More direct approach: use a proper accuracy scoring rule (log-likelihood, etc.) to compare the simple model with a model that generalizes the simple model. This assesses adequacy of the simple model. If it's not adequate, it's underfit. $\endgroup$ Jul 12, 2018 at 12:46
  • $\begingroup$ @FrankHarrell log-likelihood may not always be straightforward to express. For GLM and such things with clearly expressible error distributions it is, but beyond that? $\endgroup$ Jul 12, 2018 at 12:54
  • $\begingroup$ Log-likelihood and Bayesian counterparts are available in any fully-specified statistical model. This is related to efficient pseudo R^2 measures and AIC. $\endgroup$ Jul 12, 2018 at 18:14
0
$\begingroup$

A sign of Underfitting could be when the accuracy of the test and training are very similar. The way to "fix" it and through "tuning".

If you are trainning a NN you could increase the number of training cycles, on the other hand, if you are working with traditionalML, for example a Logistic Regression, you coudl try to tune certain parameters(that come by default) of the algorithm that leas to a better perfomance of this, in the case of Logistic Regression it would be changing the differente values of C

i.e

signal of underfitting, then I tune my model

# default
logreg = LogisticRegression(solver='liblinear', random_state=0)

worst performance, discard it

# instantiate the model
logreg001 = LogisticRegression(C=0.01, solver='liblinear', random_state=0)

it improves performance so I keep it.

# instantiate the model
logreg100 = LogisticRegression(C=100, solver='liblinear', random_state=0)
$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.