Suppose a binary outcome $Y=0$ or $Y=1$ where $P(Y=1|X)=f(X)$ is a function of $X$. The goal is to estimate $f$ as closely as possible using a classifier that returns a probability estimate (e.g. random forest or regularized logistic regression) where the tuning parameter is chosen using cross-validation. Which measure of model fit should be applied in the validation sets?
Clearly, $P(Y=1|X)$ is not observed (only $Y$ is) but the objective is estimating $P(Y=1|X)$ properly and not (only) classifying $Y$ accurately. These appear two different tasks. For example, a classifier always predicting 0.51 whenever $P(Y=1|X)>0.5$ has good classification performance with cut-off at 0.5, but does not necessarily predict the probability well.
Which measure of fit should one choose given this objective? Options I know of are accuracy defined as the proportion of validation $Y$ classified to the correct class and $AUC$, area under the receiver operating characteristic curve. But I do not know abou their performance for my objective.