How to measure test set error with logistic regression In measuring the performance of a model, I divide my data into 2 sets, the training set and the test set, fit my model to the training set and then try to predict the results of the test set. If I'm looking at binary classification, I expect to classify my results into 0's and 1's. However, the output of the logistic regression is probabilities. So if I predict a probability of 51%, but I classify this as a 1 because it is > .5, and I get it wrong 49% of the time, isn't my model right as opposed to being 49% wrong?
Would it be a better measure to check if my error rate is close to the expected error rate for the model? 
 A: There is no standard way to define goodness-of-fit. It depends on your application and what the problem you are going to solve. As in classification, you may define the goodness-of-fit as 0-1 loss. 
For a logistic regression, you can compute the likelihood function. I would use a McFadden pseudo-$R^2$, which is defined as:
$$
R^2 = 1 - \frac{\operatorname{L}(\theta)}{\operatorname{L}(\mathbf{0})}
$$
$\operatorname{L}$ is the log-likelihood function, $\theta$ is the parameter of the model and $\mathbf{0}$ denote a zero vector (i.e. you compare the likelihood ratio of your model against a model with all coefficients 0)
Moreover, given a probability measure $\mu(x) = P(Y = 1|X=x)$, define the loss function of a classifier $g$ as $L(g) = P(g(X) \neq Y)$. 
The Bayes decision rule:
$$
g^*(x) = \begin{cases} 1 & \mbox{if } \mu(x) \geq 0.5 \\ 0 & \mbox{if } \mu(x) < 0.5 \end{cases}
$$
is the rule that minimize $L(g)$. That is nothing wrong to classify as 1 when your logistic regression output probability $\geq 0.5$ as long as you are thinking the loss function as above.
A: (1) You're describing split sample internal validation that has become less popular (in favor of bootstrapping) given the large dataset size you need to produce reliable estimates.
(2) You don't have to choose 0.5 as your classification cut-point. You can choose anything, depending on what suits your objective/utility function
(3) I don't understand your last sentence. You may be trying to distinguish between discrimination and calibration, this is an important distinction.
(4) Good overview: http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3575184/ 
A: Besides data splitting sometimes requiring $n > 10,000$ to be reliable, you are using an "error" measure that is unnatural to probability models.  Besides a generalized $R^2$ as mentioned above, consider the Brier score and the $c$-index (concordance probability) or the related Somers' $D_{xy}$ rank correlation between predicted probability and $Y$.  @wonghang note that the Bayes decision rule requires an external loss (utility) function.
A: Measure the F statistic or proportion of explained variation. Also consider the covariate measurement error as it was performed by this study: S Rabe-Hesketh et al.,Correcting for covariate measurement error in logistic regression using nonparametric maximum likelihood estimation, 2003
A: There isn't a perfect analogy to say R squared for a binary GLM though there are a few approximations using residual deviance and such.
Another way to approach measuring performance is to look at the AUC of the ROC curve which looks more at the ability of the model to separate goods and bads as probabilities increase.
You would expect a good model to score a larger proportion of the actual 1's with higher probabilities and more 0's at the lower probabilities.
AUC is more of a global measure of the power of the model.  It doesn't require you to rely on a strictly greater than or less than .5 label as the measure of quality.
The actual mechanics of it are best explained visually with a ROC chart and confusion matrix.  
More here.
And here
