Can someone explain what area under the curve means for someone with absolutely no stats knowledge? For example, if a model claims an AUC of 0.9, does that mean that it makes an accurate prediction 90% of the time?
AUC is difficult to understand and interpret even with statistical knowledge. Without such knowledge I'd stick to the following stylized facts:
- AUC close to 0.5 means a model performance wasn't better than randomly classifying subjects. It wasn't better than a silly random number generator to mark the samples as positive and negative.
- AUC is used by some to compare models.
- Higher AUC suggests better demonstrated performance in classification.
- AUC is a noisy metric
- Max AUC is 1, for a classification model that is never wrong
- Although technically Min AUC is 0, it makes little sense to have AUC lesser than 0.5. AUC zero means that by a simple switch from positive to negative label you get to a perfect classification
To keep things reasonably simple, an AUC of 0.9 would mean that if you randomly picked one person/thing from each class of outcome (e.g., one person with the disease and one without), there is a 90% chance that the one from the class of interest (the group being modelled, here those with the disease) has the higher value (or this could be a lower value if the thing of interest was associated with the reference or default class).
So if the AUC for predicting "being male" versus "being female" using height was 0.9, this would mean that if you took a random male and a random female, 90% of the time, the male would be taller.
A classifier is a criterion to assign an individual to a category ("positive" or "negative") depending to some of its characteristics.
Some classifiers will provide each individual with a number between $0$ and $1$, with $0$ being "totally sure it's negative" and 1 being "totally sure it's positive". We usually take $0.5$ as the threshold between what we take as "positive" and what we take as "negative", but this is not always the case.
Taking a low threshold will result in more true positives but also more false ones. Taking a higher threshold will reduce the number of false positives, but we'll also leave as negative some of the cases that where actually positive (thus less true postiives as well). So in the end, since no classifier is perfect, it will be a compromise between the two.
Each point in the ROC curve represents the rates of true and false positives for each of the possible thresholds we could choose. The AUC is the area below that curve. A high AUC indicates that the model can get a good FPR (false positive rate) without losing too much TPR (ture positive rate) and vice-versa.(Note that the area below the ROC curve will be big if you get a high TPR already for an FPR close to 0).
SIMPLIFIED EXAMPLE: let's say you want to use a person's height to determine whether they're a man or a woman. Your classifier will choose some height $X$ and predict that everyone above height $X$ is male and everyone below it is female.
If you choose a very high $X$, like $1.90$m, you will hardly ever mislabel a woman as male, but you will also "miss" many men. On the other hand, if you pick a low $X$ like $1.50$m, you will correctly identify almost all men, but you will also classify a lot of women as male. For each $X$ you can choose, you'll get different true and false positive ratios, but it's ultimately a kind of arbitrary choice depending of what type of error worries you the most.
In this context, we could plot the ROC curve with the different TPRs and FPRs, then the AUC would give us an idea of how good of a classifier we can hope to get using height (as opposed to some other classifier we could have thought of using something like weight, age, blood pressure...). (See user215517's answer)
Following up on the comment from @Nuclear Hoagie, the ROC curve for a model is generated by evaluating classifiers using a sequence of thresholds for declaring positive or negative. The AUC represents the area under the curve over the entire range of possible thresholds. Often, only a restricted range of thresholds is really of interest. When this is the case, AUC may not be the best way to compare models.