regression for binary classification Given a binary classification problem, is there any inherent difference (or advantage) to using a classifier (say a logistic regression) and a regression, where the classes are denoted by 0 and 1 (or any two numbers)? Further, say, after running the regression, one can learn to optimize the 'cutoff' point (maybe not 0.5 in this example, but it turns out better to cut at 0.45).
 A: The key question is whether you are likely to need estimates of the probability of class membership, a ranking, or whether you genuinely only interested in a binary classification.  In my experience, you often do want the probabilities as the class frequencies, or the misclassification costs are unknown or variable in operation.  If you have a probabilistic classifier you can compensate for these problems after training, if you have a discrete yes/no classifier you can't.
One of the guiding principles behind the support vector machine was Prof. Vapnik's idea that in solving a particular problem, you should not solve a more general problem and then simplify the answer.  In classification this would mean that if you are only interested in binary classification, then we should not estimate probabilities and then threshold them, because modelling efforts and resources are wasted estimating the changes in probability away from the decision boundary, where they are of no interest.  This is a very reasonable idea, and I fully agree, provided you really are only interested in a discrete yes/no classification.
As it happens, if you perform least-squares regression on 0/1 targets, you will asymptotically end up with estimates of the probabilities anyway.  This is because least-squares results in the output being an estimate of the conditional mean of the target variable.  If this is coded as 0/1 then the conditional mean is just the conditional probability of a 1, given the input vector.
In short, which method you use depends on the needs of the application, if you need the probabilities or a ranking of the test data, use a probabilistic method (or least-squares etc. for ranking).  If you only want the hard classification into discrete classes, use something designed especially for that problem, such as the SVM.
A: You begin with a misunderstanding.  It is important to get the terminology right at the beginning.  Logistic regression is not a classifier.  It is a direct probability model. 
You didn't explain why your problem is an all-or-nothing classification problem vs. a risk estimation problem.
I have to disagree with the answer above.  You will get more efficient/powerful/precise estimates by using maximum likelihood estimation to fit a probability model such as the logistic model, then applying your utilities/cost/loss function to the predicted probabilities to make optimum decisions.  If you cannot come up with a utility/loss function it's hard to argue that you should be doing classification in the first place, but you could make the ridiculous assumption that utilities are the same for every observation and make a classification based on predicted probabilities.  You will quickly see that classification is arbitrary when you proceed that way.
Note that proportion "classified" correct is an improper accuracy scoring rule that is optimized by a bogus model.
A: Intriguing question, I had this question for a while,. Here is my findings
Short Answer
You can create any number of classifier you want, but the point is, you can only prove a few of them to be Bayes/universally-consistent! ( Bayes consistency means that classifier is asymptotically optimal, i.e. with infinite data its risk limits Bayes risk, which is optimal risk) 
The consistency of a classifier, depends on loss function and (inverse)-link function (i.e. mapping from [0 1] probability space to $\mathbb{R}$, and vice versa.)
Long answer
First, according to Tong's great paper all the (consistent) classifiers are equivalent! except in that they are minimizing different loss functions, and almost every difference between classifiers is a consequence of their loss functions. In fact, he showed that minimizing every loss function leads to optimal decision function (technically, inverse-link function), which is completely function of probabilities (even for SVMs!). His result is summarized in this table (by Hamed):

Despite of this unified view over all the classifiers, they are different in their outputs:


*

*Probability-Calibrated: for these class the classifiers (e.g. Logistic Regression), output is DIRECTLY within a probability measure, which this in turn not only answers yes/no question of the classifier, but also gives confidence of the of the decision.

*Not-probability-Calibrated: Other classifiers (e.g. SVM) are real-valued-output classifiers, which you can use some link functions to calibrate the to enforce outputs to be probabilities.


Conclusion
It really depend on loss-function, link-function, calibration. For example, first line of the table says that, least-squares regression and classification are the same,(if your classifier output is calibrated-probabilities $\eta$, and using the corresponding inverse link function)
