3
$\begingroup$

Given that $X$ is the $n \times k$ matrix containing n observations of k independent variables, $Y$ is a $n \times 1$ vector that contains n observations of the dependent variable, we know that the OLS estimator can be represented as

$\widehat{\beta}_{OLS} = (X'X)^{-1}X'Y$

But how do we represent the estimator for logit? Should it be something like

$\sigma(X'X)^{-1} \sigma(X'Y) \ $ or $\ \sigma( \ (X'X)^{-1} X'Y \ )$

where the logistic function $\sigma(\cdot)$ is the link function

$\endgroup$
1
  • $\begingroup$ I figured out how to write the Lipovetsky closed-form solution that holds with solely categorical predictors in a way similar to what's in your question, and added that to the answer below. $\endgroup$
    – EdM
    Apr 26, 2021 at 15:28

1 Answer 1

6
$\begingroup$

In general, there is no closed-form solution for a logistic regression that corresponds to the OLS estimator of regression coefficients, as you seem to wish. See this thread for discussion. From that perspective, there isn't an answer to your question.

There is, however, a closed-form solution when all predictors are categorical. If the design matrix $X$ is furthermore coded in a particular way, with orthogonal binary variables, then there is a solution close to the form you seek. This presentation follows Lipovetsky, Journal of Applied Statistics, 42: 37-49 (2015).

First, the design matrix $X$ needs to represent each combination of predictor categories as a distinct binary predictor.* Second, it's simplest to start by treating all such distinct binary predictors as having separate coefficients, omitting an intercept, then transforming back to the usual form. Each row of the design matrix then has a single non-zero entry of 1, in the column representing its particular combination of predictor values.

In that form, $X'Y$ is a vector of counts, for each binary predictor $j$, of cases having $Y=1$, or $N_{j1}$.** The matrix $X'X$ is diagonal, with element $jj$ being the total number of cases having binary predictor $j$ regardless of the value of $Y$, or $(N_{j0} + N_{j1})$. Then the elements of the vector $(X'X)^{-1}X'Y$ are the maximum-likelihood estimated probabilities of $Y=1$ for each binary predictor $j$, or

$$\hat p_j = \frac{N_{j1}}{N_{j0} + N_{j1}},$$

the fraction of cases having $Y=1$ when the predictor is $j$. In this form, $\text{logit}\left((X'X)^{-1}X'Y\right)$ (done element-wise) gives coefficients $c_j$ representing the log-odds of $Y=1$ for each predictor $j$. As Lipovetsky puts it, this:

... model without intercept presents the absolute impact of each category on the binary output, while a related model with intercept presents the relative impact of each category versus one taken as the reference.

To transform to the latter standard representation, choose the last predictor combination $K$ as the reference, set the intercept $b_0 = c_K$, and write the other logistic regression coefficients as $b_j = c_j - c_K$. Lipovetsky then shows how to derive standard errors for coefficients and deviances.


*I'm not sure if this will work if all possible combinations of predictor categories aren't present in the data set.

** Note that $\text{logit}(X'Y)$ as in your first suggestion wouldn't make sense, as those values are counts rather than probabilities.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.