Support vector machines (SVMs) are the zero temperature limit of logistic regression? I was had a quick discussion recently with a knowledgeable friend who mentioned that SVMs are the zero temperature limit of logistic regression. The rationale involved marginal polytopes and fenchel duality. I wasn't able to follow.
Is this statement about SVMs being the zero temperature limit of logistic regression true? And if so, can someone describe the argument?  
 A: In the case of hard-margin SVM and linearly separable data, this is true.
An intuitive sketch: The loss for each datapoint in logistic regression dies out almost as an exponential decay curve as you get farther from the decision boundary (in the correct direction of course). This exponential decay means that the points closest to the boundary incur much more loss. As the temperature drops to 0, the points closest to the boundary completely dominate the loss, and the loss is determined by exactly how close the closest points are.
Binary logistic regression has the cross-entropy loss: $- y \log p - (1-y)\log (1-p)$ where $y$ is the label and $p$ is the predicted probability in $(0,1)$. 
Typically, $p = \sigma(w^Tx + b)$ where $\sigma$ is the sigmoid function. Based on the temperature parameter introduced in this paper, I suspect that the temperature refers to a modification of the formulation: $p = \sigma(\frac{w^Tx}{\tau})$, where $\tau$ is the temperature and I've dropped the bias term for simplicity.
Considering only the first term of the loss, $-y\log p = y\log(1+\exp{}(-\frac{w^Tx}{\tau}))$. Assume all $w^Tx > 0$, because anything else would mean that $x$ is on the wrong side of the decision boundary and incur infinite loss as $\tau \rightarrow 0$. Since the exponential term gets very small in the limit, we use the first order taylor expansion for $\log(1+z)$ to write $-y\log p \approx y\exp{(-\frac{w^Tx}{\tau})}$
Up to now, we've been using just the loss for a single datapoint, but the actual loss is $\sum_i y_i \exp{(-\frac{w^Tx_i}{\tau})}$. Consider only positive labels ($y_i = 1$). Then this sum is dominated by the term where $w^Tx_i$ is the smallest (closest to the decision boundary).
This can be seen because the ratio between the $i$ term and the $j$ term is $\frac{\exp (-w^T x_i/\tau)}{\exp (-w^T x_j/\tau)} = \exp(\frac{w^T x_j-w^T x_i}{\tau})$ which goes to infinity or 0 as $\tau \rightarrow 0$, so only the largest $w^T x_i$ term matters. 
A symmetric argument can be used on the second term in the loss. 
Therefore, the loss of the logistic regression problem as the temperature goes to 0 is minimized by maximizing the minimum distance to the decision boundary.
