I'm confused about how scikit-learn's roc_auc_score is working.
As I understand it, an ROC AUC score for a classifier is obtained as follows:
- Sample from the parameter space
- Fit the model
- Make predictions with the model to get $Y_{predicted}$
- Calculate $P(F_P)$ and $P(T_P)$ via $Y_{true}$ and $Y_{predicted}$
The above steps are performed repeatedly until you have enough $(P(F_P), P(T_P))$ points to get a good estimate of the area under the curve.
The sklearn.metrics.roc_auc_score method takes $Y_{true}$ and $Y_{predicted}$ and gives the area under the curve based only on these. How is this possible? It seems you'd need multiple sets of $Y_{true}$ and $Y_{predicted}$ coming from different forms of the model (i.e. trained with different parameters) in order to get multiple $(P(F_P), P(T_P))$ points to estimate the area under the curve.
I'm clearly not understanding something here. What is it?