When does a logistic regression model have a unique solution? Mathematically speaking, for which data does a logistic regression model have a unique solution?
 A: I believe you are looking for the concept of orthogonality of the covariates. As soon as one of the covariates can be written as a linear combination of one of the others, you will not have a unique solution.
As an extreme case: say you have 2 covariates, and one is (in your dataset) always the double of the other, then both
$$logodds(outcome)=\beta_0+\beta_1 X_1$$
and
$$logodds(outcome)=\beta_0+\frac{1}{2}\beta_1 X_2$$
Will yield the same results (regardless of $\beta_1$), and of course there are lots of other solutions.
A: The solution of logistic regression is a solution of maximization of certain function, namely log-likelihood:
$$\sum_{i=1}^ny_i\log p_i+(1-y_i)\log(1-p_i),$$
where 
$$p_i=\frac{\exp(\beta_0+\beta_1x_{1i}+...+\beta_kx_{ik})}{1+\exp(\beta_0+\beta_1x_{1i}+...+\beta_kx_{ik})},$$
and $(y_i,x_{1i},...,x_{ki})$, $i=1,...,n$ is the data.
So mathematically speaking the unique solution of logistic regression exists for given data set if the log-likelihood has a unique maximum. If I am not mistaken full rank of matrix $X=[1,x_{1i},...,x_{ki}]$ is necessary for that. For more mathematical conditions you might look into iterative reweighted least squares, since maximisation of log likelihood function for logistic regression is a special case of IRWLS.
A: I think an interesting point is, that when the data is separable there should be infinite solutions. But if u use GD you converge to the max-margin solution (intuition I have is that GD for linear regression when its overparametrized converges to pseudo-inverse which is min-norm which is max margin since margin is sometimes related to 1/w). So in a way, its like having a unique minimizer. Even though we never truly get there, we do converge to it. You can check this out here:
[1710.10345] The Implicit Bias of Gradient Descent on Separable Data (https://arxiv.org/abs/1710.10345)
intuitively if you look at the gradient:
$$ \nabla_w l(w) = \frac{1}{N} \sum^N_{n=1} \frac{y^{(n)} x^{(n)}}{ 1 + e^{y^{(n)} w^\top x^{(n)}} }$$
since the weights increase so does the score and thus the denominator of the above. But the weights increase “sort of linearly” while the decrease in the size of the of the gradient is exponential (as seen above). So GD stops updating “pretty soon”. Or at least thats the ay I sort of understand it at a high level. For real answers, refer to the paper of course.
Therefore if the data is separable and you use GD you converge (approach) to the max-margin solution for unregularized logistic regression, which is unique.
