3
$\begingroup$

Question: Are there (known) variance stabilizing transformations for logistic regression?

Backgound: As an M-estimator, logistic regression is asymptotically normal, under suitable regularity conditions. The asymptotic variance, however, depends on the unknown parameter $\theta^*$. Say, $$ \sqrt{n}\left(\hat{\theta}_n-\theta^*\right)\rightarrow \mathcal{N}(0,\Sigma_{\theta^*}). $$ The dependence of the asymptotic variance on the unknown parameter $\theta^*$ is undesirable, for example to find approximated confidence intervals. There are examples, where one can use a variance-stabilizing transformation to free the asymptotic variance of unknown parameters. For estimating a one-dimensional parameter $\theta$ with $T_n$, there may exist a mapping $\phi$, such that: $$ \sqrt{n}\left(\phi(T_n)-\phi(\theta^*)\right)\rightarrow \mathcal{N}(0,\phi'(\theta^*)^2\sigma(\theta^*)^2)=\mathcal{N}(0,c^2). $$ To find $\phi$, one can solve: $$ \phi(\theta)=\int\frac{1}{\sigma(\theta)}d\theta $$ I don't see why this wouldn't work for logistic regression. However, I could not find variance-stabilizing transformations for logistic regression in the literature. I tried to do some calculations myself, but they got messy quite quickly.

Example: (Edit) The commenters pointed out that my question is not clear enough. I'll give an example where the variance stabilizing tarnsformation works. Hopefully this clarifies what I'm looking for.

Suppose that we have $X_1,\ldots,X_n$ following a Poisson distribution with parameter $\lambda>0$. We try to estimate $\lambda$ using the mean $\hat{\lambda}_n:=n^{-1}\sum_{i=1}^nX_i$. Then, by the CLT, $$ \sqrt{n}(\hat{\lambda}_n-\lambda)\rightarrow\mathcal{N}(0,\lambda) $$

It is undesirable that the asymptotic variance depends on the unknown parameter, since this makes the construction of confidence intervals more difficult. To give another example, consider esimating the mean of a normal distribution. If the variance is unknown, we can still perform a T-test. However, if the variance were known, we could perform a Z-test, which has more power.

In the case of the Poisson distribution, we can use a variance stabilizing transformation, namely the square root. By the delta-method: $$ \sqrt{n}\left(2\sqrt{\hat{\lambda}_n}-2\sqrt{\lambda}\right)\rightarrow \mathcal{N}(0,1) $$ With this trick, we can easily construct asymptotic confidence intervals, without further complications. At least this is the case for the given example. My question is, whether such a transformation can be done in logistic regression.

$\endgroup$
9
  • 1
    $\begingroup$ Logistic regression do not assume a constant variance, so I cannot see a need ... $\endgroup$ Commented Oct 13, 2022 at 13:14
  • 1
    $\begingroup$ @kjetilbhalvorsen Apologies, I don't understand your comment. In any case I can fit a logistic regression model, even if $\Sigma$ depends on $\theta$. But to build a confidence interval, this complicates the analysis, at least. $\endgroup$
    – Idontgetit
    Commented Oct 14, 2022 at 7:17
  • 1
    $\begingroup$ @whuber Thank you for your comment. The comparison with OLS regression isn't right I think, since there we don't estimate the noise variance $\sigma^2$. The idea here is that $\Sigma$ depends on $\theta$, which is what we try to estimate. I don't understand your second question. The variance stabilizing transform is applied after estimation, so the variables are not transformed. $\endgroup$
    – Idontgetit
    Commented Oct 14, 2022 at 7:46
  • 1
    $\begingroup$ @whuber I suppose we are not talking about the same thing. That's ok, since my question is not about OLS regression. $\endgroup$
    – Idontgetit
    Commented Oct 14, 2022 at 13:45
  • 1
    $\begingroup$ It is, unfortunately, difficult to determine what your question really is about. Transforming the parameter won't change the model: only transformation of the data could possibly be relevant. $\endgroup$
    – whuber
    Commented Oct 14, 2022 at 16:08

1 Answer 1

4
$\begingroup$

I think in the end your goal is to have inference for binary data resembling logistic regression where the variance of various model quantities (parameter estimates, predictions, etc.) does not depend on the probabilities in the model, hence your search for variance-stabilizing transformations. If that's your desire, you should not transform logistic regression itself. Instead, you should use a different link function other than the logistic link function.

Generalized linear models connect the mean of the response variable to the model factors via: $$g(p)=x^\top \beta,$$ where $x$ is your vector of factors, $\beta$ the linear model coefficients, $p$ is the mean of the response variable (in this case, a probability), and $g$ is a link function. In logistic regression, $g$ is logistic function, but you are not required to use that link function. You could instead choose a link function that is variance stabilizing.

McCullagh and Nelder (1989) mention on pg. 43 that one can make the weights of the Fisher information matrix constant by choosing a link function such that $$g'(p)=V^{-1/2}(p)=(p(1-p))^{-1/2}.$$ The angular link function $g(p)=\arcsin(p^{1/2})$ has this property; notice that this is the variance stabilizing transformation for binomial models, in general. The downside of this link function is that unlike the logistic link function, it is not an injective and surjective mapping from $(0,1)$ to $\mathbb{R}$, but it does eliminate dependence on variances at each point, which can be important in applications.

So in short, if you want variance stabilization, use a variance-stabilizing link function instead of the logit link function used in logistic regression.


McCullagh, P., Nelder, J. (1989). Generalized Linear Models, Second Edition. Chapman & Hall. ISBN: 9780412317606

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.