In binary classification, in what specific case should I use accuracy, AUROC, logloss or Brier score to choose model? When selecting models in binary classification, there are a couple of most often used metrics, such as accuracy, AUROC, F1 score, logloss and Brier score.  I understand what each of those metrics means. But in actual model evaluation, I'm often confused about which is the correct and best one to use. 
This is critical in model evaluation. 
Can someone help clarifying this importance topic?  It will be most helpful for me and everyone to truly understand if you can briefly illustrate with some SPECIFIC CASES to correctly use each of those metrics. 
Very much appreciate it in advance. 
 A: To elaborate on some of the comments.  Most models give a probability output--- say 0.78.  If you convert this to a 1, then you have made a "decision".  Accuracy, ROC, precision, recall, are all functions of decisions; log likelihood, brier score are functions of probabilities.  Often, it's best to leave the decision to the end user, but there are notable exceptions (machine translation, speech recognition (the end user is the computer), self driving cars).

accuracy

Better to not.  Perhaps with balanced classes.  

AUROC

This just gives sensitivity and specificity as you vary the threshold. Applications include validation in poorly written papers on risk prediction :).  Seriously, though, use it if you are interested in providing binary decisions (1 or 0) as output, not a probability estimate. This is important for example when you are guiding a self driving car that must make decisions quickly as to what is road and what is not---and the cost of a false positive is identical to the cost of a false negative.

F1 score

A combination of precision and recall.  Application: document retrieval.  Generally document retrieval and NLP tasks like machine translation use these metrics because a decision is made (e.g., in machine translation, you want a sentence as output, not some probability distribution over sentences).  Precision, in contrast to sensitivity and specificity, depends on prevalence.  Therefore it is a good metric for very rare classes, like in document retrieval.

logloss 

If you're interested in probability estimates and have a well-specified predictive model.  This is the loss function when you minimize the negative log likelihood of a collection of independent Bernoulli random variables.  This is related to the probability of your dataset using your estimated parameters.  Application: risk score, prediction. Eg, what's the probability of disease or of defaulting on a loan.

Brier score

If you're interested in probability estimates, this is basically a measure of how close (using mean squared error) your estimates are numerically to the actual probabilities.  This is nice because it's kind of bounded as opposed to log loss, but I have found it to be kind of insensitive.
A: Never use accuracy to evaluate a binary classifier. Here’s an extremely accurate prediction you can make at any party you attend: no one present is an astronaut. Because there are about 8 billion people in the world and only about 600 astronauts, this prediction will almost always be 100% accurate. It will also never correctly predict an astronaut. In any binary classification problem, pick the label that occurs most frequently in the training set and your accuracy will be at least 50%.
The metrics to use for binary classification are precision and recall (and in some applications additional metrics like time-to-detect or computational cost). The relative value placed on these metrics depends on the application and the cost of false positives vs false negatives. If you value them equally, F1 is commonly used; otherwise F0.5 or F2 are common too. But really, look at precision and recall separately to understand what’s going on.
Precision and recall work well in theory, and they work well in practice. They’re easy to communicate to stakeholders. Many data-driven CEOs know what precision and recall are and how to reason about them effectively.
Log-loss (cross entropy) is usually best as the score function for training a classifier (binary or multilabel). It measures the relative uncertainty between the classes your model predicts and the true classes. Brier score is sometimes used for non-ordinal multilabel classification, but is in general inferior to log-loss.
If you’re deeply interested in the theory behind this topic, Some Comparisons among Quadratic, Spherical, and Logarithmic Scoring Rules (2007, Bickel) has a gentle introduction. Tl;dr: use log-loss.
