How to judge if a supervised machine learning model is overfitting or not? Can anyone tell me how to judge if a supervised machine learning model is overfitting or not? If I don't have an external validation dataset, I want to know if I can use ROC of 10 fold cross validation to explain overfitting. If I have an external validation dataset, what should I do next?
 A: Here's how you can estimate the extent of overfitting:


*

*Get an internal error estimate. Either resubstitutio (= predict training data), or if you do an inner cross "validation" to optimize hyperparameters, also that measure would be of interest.

*Get an independent test set error estimate. Usually, resampling (iterated cross validation or out-of-bootstrap* is recommended. But you need to be careful that no data leaks occur. I.e. the resampling loop must recalculate all steps that have calculations spanning more than one case. That includes pre-processing steps such as centering, scaling, etc. Also, make sure you split at the highest level if you have a "hierarchical" (also known as "clustered) data structure such as repeated measurements of e.g. the same patient (=> resample patients).

*Then compare how much better the "inner" error estimate looks than the independent one. 


Here's an example:

Trefferrate = hit rate (% correct classified), Variablenzahl = number of variables (= model complexity)
Symbols: . resubstitution, + internal leave-one-out estimate of hyperparameter optimizer, o outer cross validation independent at patient level  
This works with ROC, or performance measures such as Brier's score, sensitivity, specificity, ...
* I don't recommend  .632 or .632+ bootstrap here: they already mix in resubstitution error: you can anyways calculate them later from your resubstitution and out-of-bootstap estimates.
A: In short: by validating your model. The main reason of validation is to assert no overfit occurs and to estimate generalized model performance. 
Overfit
First let us look at what overfitting actually is.
Models are normally trained to fit a dataset by minimizing some loss function on
a training set. There is however a limit where minimizing this training error will
no longer benefit the models true performance, but only minimize the error on the
specific set of data. This essentially means that the model has been too tightly
fitted to the specific data points in the training set, trying to model patterns in the data originating from noise. This concept is called overfit. An example of overfit is displayed below where you see the training set in black and a larger set from the actual population in the background. In this figure you can see that the blue model is too tightly fitted to the training set, modeling the underlying noise. 

In order to judge if a model is overfitted or not, we need to estimate the generalized error (or performance) that the model will have on future data and compare it to our performance on the training set. Estimating this error can be done in several different ways.
Dataset split
The most straightforward approach to estimating the generalized performance is to
partition the dataset into three parts, a training set, a validation set and a test set.
The training set is used for training the model to fit the data, the validation set is
used to measure differences in performance between models in order to select the
best one and the test set to assert that the model selection process does not overfit to the first two sets.
To estimate the amount of overfit simply evaluate your metrics of interest on the test set as a last step and compare it to your performance on the training set. You mention ROC but in my opinion you should also look at other metrics such as for example brier score or a calibration plot to ensure model performance. This is of course depending on your problem. There are a lot of metrics but this is besides the point here.
This method is very common and respected but it puts a big demand on availability of data. If your dataset is too small you will most probably lose a lot of performance and your results will be biased on the split. 
Cross-validation
One way to get around wasting a large part of the data to validation and test is to
use cross-validation (CV) which estimates the generalized performance using the same data as is used to train the model. The idea behind cross-validation is to split the dataset up into a certain number of subsets, and then use each of these subsets as held out test sets in turn while using the rest of the data to train the model. Averaging the metric over all the folds will give you an estimate of the model performance. The final model is then generally trained using all data. 
However, the CV estimate is not unbiased. But the more folds you use the smaller the bias but then you get larger variance instead.
As in the dataset split we get an estimate of the model performance and to estimate the overfit you simply compare the metrics from your CV with the ones acquired from evaluating the metrics on your training set. 
Bootstrap
The idea behind bootstrap is similar to CV but instead of splitting the dataset into parts we introduce randomness in the training by drawing
training sets from the whole dataset repeatedly with replacement and performing
the full training phase on each of these bootstrap samples.
The simplest form of bootstrap validation simply evaluates the metrics on the samples not found in the training set (i.e. the ones left out) and average over all repeats.
This method will give you an estimate of model performance which in most cases are less biased than CV. Again, comparing it with your training set performance and you get the overfit.
There are ways to improve the bootstrap validation. The .632+ method is known to give better, more robust estimates of the generalized model performance, taking overfit into account. (If you're interested the original article is a good read: Improvements on Cross-Validation: The 632+ Bootstrap Method)
I hope this answers your question. If you are interested in model validation I recommend reading the part on validation in the book The elements of statistical learning: data mining, inference and prediction which is freely available online.
A: The overfitting is simply the direct consequence of considering the statistical parameters, and therefore the results obtained, as a useful information without checking that them was not obtained in a random way. Therefore, in order to estimate the presence of overfitting we have to use the algorithm on a database equivalent to the real one but with randomly generated values, repeating this operation many times we can estimate the probability of obtaining equal or better results in a random way. If this probability is high, we are most likely in an overfitting situation. For example, the probability that a fourth-degree polynomial has a correlation of 1 with 5 random points on a plane is 100%, so this correlation is useless and we are in an overfitting situation.
