I understand that using cross validation we can validate our model, but it is also possible that maybe our model is underfitting; hence, providing wrong results. One possibility that I can think of is that if even after making our model less complex results are not right then our model might be underfitting, it is just a guess!
Please write if anything I am missing or any other way to decide whether our model is overfitting or underfitting?
$\begingroup$
$\endgroup$
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
5
You can determine the difference between an underfitting and overfitting experimentally by comparing fitted models to training-data and test-data.
Typical graphs:
These plots will show you the accuracy of the model, as function of some parameter (e.g. 'complexity'), for both the
- training-data (the data use for fitting)
- and test data (data kept separate from the training phase)
One normally chooses the model that does the best on the test-data. If you deviate then for...
For underfitting models, you do worse because they do not capture the true trend sufficiently.
If you get more underfitting then you get both worse fits for training and testing data.
for overfitting models, you do worse because they respond too much to the noise, rather than the true trend.
If you get more overfitting then you get better fits for training data (capturing the noise, but it is useless or even detrimental), but still worse for testing data.
After performing these steps, you should remain in sight that, while you may have selected the least underfitting/overfitting model out of your options, this does not mean that there would not be some other better model.
In particular: the true model, might be an overfitting model in circumstances of high noise-signal ratios (see https://stats.stackexchange.com/a/299523/164061 )
answered Jul 12, 2018 at 12:04
-
$\begingroup$ The first plot seems to show error over training time. $\endgroup$– TimCommented Jul 12, 2018 at 12:16
-
$\begingroup$ @Tim Training time makes a model (e.g. an artificial neural network) better in fitting the training data (increases complexity) but not necessarily the testing data. Thus it is related to overfitting vs. underfitting. $\endgroup$ Commented Jul 12, 2018 at 12:25
-
$\begingroup$ More direct approach: use a proper accuracy scoring rule (log-likelihood, etc.) to compare the simple model with a model that generalizes the simple model. This assesses adequacy of the simple model. If it's not adequate, it's underfit. $\endgroup$ Commented Jul 12, 2018 at 12:46
-
$\begingroup$ @FrankHarrell log-likelihood may not always be straightforward to express. For GLM and such things with clearly expressible error distributions it is, but beyond that? $\endgroup$ Commented Jul 12, 2018 at 12:54
-
$\begingroup$ Log-likelihood and Bayesian counterparts are available in any fully-specified statistical model. This is related to efficient pseudo R^2 measures and AIC. $\endgroup$ Commented Jul 12, 2018 at 18:14
$\begingroup$
$\endgroup$
A sign of Underfitting could be when the accuracy of the test and training are very similar. The way to "fix" it and through "tuning".
If you are trainning a NN you could increase the number of training cycles, on the other hand, if you are working with traditionalML, for example a Logistic Regression, you coudl try to tune certain parameters(that come by default) of the algorithm that leas to a better perfomance of this, in the case of Logistic Regression it would be changing the differente values of C
i.e
signal of underfitting, then I tune my model
# default
logreg = LogisticRegression(solver='liblinear', random_state=0)
worst performance, discard it
# instantiate the model
logreg001 = LogisticRegression(C=0.01, solver='liblinear', random_state=0)
it improves performance so I keep it.
# instantiate the model
logreg100 = LogisticRegression(C=100, solver='liblinear', random_state=0)
