4
$\begingroup$

Mathematically speaking, for which data does a logistic regression model have a unique solution?

$\endgroup$
  • $\begingroup$ What do you mean? There's an infinite number of possible data sets that could give rise too any arbitrary logistic regression model. When you get a probability out of a logistic regression model, this is only unique when you have one feature. $\endgroup$ – benhamner Jun 30 '11 at 0:45
  • $\begingroup$ while it may seem annoying to require a more verbose question, in general, being explicit will lead to faster and more accurate responses. $\endgroup$ – M. Tibbits Jun 30 '11 at 1:19
  • $\begingroup$ Though I don't have it handy, I believe that Agresti, Categorical Data Analysis, 2nd ed. provides references to the literature on uniqueness of logistic regression coefficient estimates. As the answers point out, if your design matrix is not full rank, there is no hope. On the other hand, such situations arise often in applications of logistic regression, and often simple reformulation of the model can eliminate these redundancies. $\endgroup$ – cardinal Jun 30 '11 at 13:01
  • $\begingroup$ @M. Tibbits is correct, as the various replies and comments show: several quite different interpretations of your question are possible. Please clarify, because an ambiguous question has little value and would need to be closed. $\endgroup$ – whuber Jun 30 '11 at 13:50
  • $\begingroup$ I don't understand the ambiguity. Are you saying "data" might be independent or dependent (response) variables? To me it is both. A logistic model is a model trying to minimize error between a function of independent variables and the response. All of those are data into the model. Please clarify the different interpretations you are seeing. $\endgroup$ – dfrankow Jun 30 '11 at 16:56
10
$\begingroup$

The solution of logistic regression is a solution of maximization of certain function, namely log-likelihood:

$$\sum_{i=1}^ny_i\log p_i+(1-y_i)\log(1-p_i),$$

where

$$p_i=\frac{\exp(\beta_0+\beta_1x_{1i}+...+\beta_kx_{ik})}{1+\exp(\beta_0+\beta_1x_{1i}+...+\beta_kx_{ik})},$$

and $(y_i,x_{1i},...,x_{ki})$, $i=1,...,n$ is the data.

So mathematically speaking the unique solution of logistic regression exists for given data set if the log-likelihood has a unique maximum. If I am not mistaken full rank of matrix $X=[1,x_{1i},...,x_{ki}]$ is necessary for that. For more mathematical conditions you might look into iterative reweighted least squares, since maximisation of log likelihood function for logistic regression is a special case of IRWLS.

| cite | improve this answer | |
$\endgroup$
  • 1
    $\begingroup$ (+1) I'm not sure I would call maximization of the log-likelihood in logistic regression a special case of IRLS. Certainly, though, IRLS can be used as a method for performing the maximization. Logically, it seems, the maximization of the function and the algorithms for achieving this are logically different concepts. $\endgroup$ – cardinal Jun 30 '11 at 12:58
  • 1
    $\begingroup$ The log likelihood in logistic regression is a concave function of the parameters. A consequence is that the set of $\beta$ that maximizes the likelihood form a convex set. Even if the matrix of covariates has full rank, depending on the values of the responses, I think in some cases it may turn out that the log likelihood is not strictly convex. In that case, unless you have strict convexity at a point of optimality, the solution may not be unique. $\endgroup$ – cardinal Jun 30 '11 at 12:59
  • 1
    $\begingroup$ It seems to me this reply does not answer the question, which is to characterize the data for which the solution is not unique. (I suspect a succinct characterization is not available, but one can say some simple things that characterize certain kinds of data that are sure to have non-unique solutions.) $\endgroup$ – whuber Jun 30 '11 at 13:47
  • $\begingroup$ @whuber: It seems to me this response (and @Nick's) gives at least a partial answer to the question. The structure of the design matrix certainly plays a role in uniqueness or lack thereof of the coefficient estimates. As I think we both allude to, I'm pretty sure it's not a full characterization though, as the observed responses should also play a role. $\endgroup$ – cardinal Jun 30 '11 at 14:19
  • 2
    $\begingroup$ Calculate the hessian of the log likelihood. In order for solution to be a maximal solution, the hessian needs to be negatively defined. Rank condition for the design matrix would translate to rank condition for hessian, i.e. hessian would not have a full rank if the design matrix does not. That is a general idea. $\endgroup$ – mpiktas Feb 1 '18 at 6:58
7
$\begingroup$

I believe you are looking for the concept of orthogonality of the covariates. As soon as one of the covariates can be written as a linear combination of one of the others, you will not have a unique solution.

As an extreme case: say you have 2 covariates, and one is (in your dataset) always the double of the other, then both $$logodds(outcome)=\beta_0+\beta_1 X_1$$ and $$logodds(outcome)=\beta_0+\frac{1}{2}\beta_1 X_2$$ Will yield the same results (regardless of $\beta_1$), and of course there are lots of other solutions.

| cite | improve this answer | |
$\endgroup$
  • 2
    $\begingroup$ Instead of "orthogonality", the concept you are describing is usually referred to as linear (in)dependence. $\endgroup$ – cardinal Jun 30 '11 at 12:50
  • $\begingroup$ (+1) Nice simple demonstration of an example of when uniqueness doesn't hold. $\endgroup$ – cardinal Jun 30 '11 at 13:32
  • $\begingroup$ how is this different from how uniqueness is induced in least-squares or ridge regression? There it seems clear what the conditions are and why, but the log and the sigmoids make it very unclear to me how the rank plays any role at all...can you clarify this? $\endgroup$ – Charlie Parker Jan 31 '18 at 14:32
0
$\begingroup$

I think an interesting point is, that when the data is separable there should be infinite solutions. But if u use GD you converge to the max-margin solution (intuition I have is that GD for linear regression when its overparametrized converges to pseudo-inverse which is min-norm which is max margin since margin is sometimes related to 1/w). So in a way, its like having a unique minimizer. Even though we never truly get there, we do converge to it. You can check this out here:

[1710.10345] The Implicit Bias of Gradient Descent on Separable Data (https://arxiv.org/abs/1710.10345)

intuitively if you look at the gradient:

$$ \nabla_w l(w) = \frac{1}{N} \sum^N_{n=1} \frac{y^{(n)} x^{(n)}}{ 1 + e^{y^{(n)} w^\top x^{(n)}} }$$

since the weights increase so does the score and thus the denominator of the above. But the weights increase “sort of linearly” while the decrease in the size of the of the gradient is exponential (as seen above). So GD stops updating “pretty soon”. Or at least thats the ay I sort of understand it at a high level. For real answers, refer to the paper of course.

Therefore if the data is separable and you use GD you converge (approach) to the max-margin solution for unregularized logistic regression, which is unique.

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.