2
$\begingroup$

Question: Are there (known) variance stabilizing transformations for logistic regression?

Backgound: As an M-estimator, logistic regression is asymptotically normal, under suitable regularity conditions. The asymptotic variance, however, depends on the unknown parameter $\theta^*$. Say, $$ \sqrt{n}\left(\hat{\theta}_n-\theta^*\right)\rightarrow \mathcal{N}(0,\Sigma_{\theta^*}). $$ The dependence of the asymptotic variance on the unknown parameter $\theta^*$ is undesirable, for example to find approximated confidence intervals. There are examples, where one can use a variance-stabilizing transformation to free the asymptotic variance of unknown parameters. For estimating a one-dimensional parameter $\theta$ with $T_n$, there may exist a mapping $\phi$, such that: $$ \sqrt{n}\left(\phi(T_n)-\phi(\theta^*)\right)\rightarrow \mathcal{N}(0,\phi'(\theta^*)^2\sigma(\theta^*)^2)=\mathcal{N}(0,c^2). $$ To find $\phi$, one can solve: $$ \phi(\theta)=\int\frac{1}{\sigma(\theta)}d\theta $$ I don't see why this wouldn't work for logistic regression. However, I could not find variance-stabilizing transformations for logistic regression in the literature. I tried to do some calculations myself, but they got messy quite quickly.

Example: (Edit) The commenters pointed out that my question is not clear enough. I'll give an example where the variance stabilizing tarnsformation works. Hopefully this clarifies what I'm looking for.

Suppose that we have $X_1,\ldots,X_n$ following a Poisson distribution with parameter $\lambda>0$. We try to estimate $\lambda$ using the mean $\hat{\lambda}_n:=n^{-1}\sum_{i=1}^nX_i$. Then, by the CLT, $$ \sqrt{n}(\hat{\lambda}_n-\lambda)\rightarrow\mathcal{N}(0,\lambda) $$

It is undesirable that the asymptotic variance depends on the unknown parameter, since this makes the construction of confidence intervals more difficult. To give another example, consider esimating the mean of a normal distribution. If the variance is unknown, we can still perform a T-test. However, if the variance were known, we could perform a Z-test, which has more power.

In the case of the Poisson distribution, we can use a variance stabilizing transformation, namely the square root. By the delta-method: $$ \sqrt{n}\left(2\sqrt{\hat{\lambda}_n}-2\sqrt{\lambda}\right)\rightarrow \mathcal{N}(0,1) $$ With this trick, we can easily construct asymptotic confidence intervals, without further complications. At least this is the case for the given example. My question is, whether such a transformation can be done in logistic regression.

$\endgroup$
9
  • 1
    $\begingroup$ Logistic regression do not assume a constant variance, so I cannot see a need ... $\endgroup$ Oct 13, 2022 at 13:14
  • $\begingroup$ Even in OLS regression the asymptotic variance of the parameter depends on an unknown: namely, the variance of the conditional error term. And how do you propose to perform logistic regression on any transformed variable in the first place, given that it requires non-negative integral responses? $\endgroup$
    – whuber
    Oct 13, 2022 at 16:44
  • $\begingroup$ @kjetilbhalvorsen Apologies, I don't understand your comment. In any case I can fit a logistic regression model, even if $\Sigma$ depends on $\theta$. But to build a confidence interval, this complicates the analysis, at least. $\endgroup$
    – Idontgetit
    Oct 14, 2022 at 7:17
  • 1
    $\begingroup$ It is, unfortunately, difficult to determine what your question really is about. Transforming the parameter won't change the model: only transformation of the data could possibly be relevant. $\endgroup$
    – whuber
    Oct 14, 2022 at 16:08
  • 1
    $\begingroup$ This post might be helpful: r-bloggers.com/2018/01/delta-method-standard-errors Yes, you can use the DM for LR standard errors. However, it's not clear that you necessarily need to do so. $\endgroup$ Apr 6, 2023 at 2:34

1 Answer 1

4
$\begingroup$

I think in the end your goal is to have inference for binary data resembling logistic regression where the variance of various model quantities (parameter estimates, predictions, etc.) does not depend on the probabilities in the model, hence your search for variance-stabilizing transformations. If that's your desire, you should not transform logistic regression itself. Instead, you should use a different link function other than the logistic link function.

Generalized linear models connect the mean of the response variable to the model factors via: $$g(p)=x^\top \beta,$$ where $x$ is your vector of factors, $\beta$ the linear model coefficients, $p$ is the mean of the response variable (in this case, a probability), and $g$ is a link function. In logistic regression, $g$ is logistic function, but you are not required to use that link function. You could instead choose a link function that is variance stabilizing.

McCullagh and Nelder (1989) mention on pg. 43 that one can make the weights of the Fisher information matrix constant by choosing a link function such that $$g'(p)=V^{-1/2}(p)=(p(1-p))^{-1/2}.$$ The angular link function $g(p)=\arcsin(p^{1/2})$ has this property; notice that this is the variance stabilizing transformation for binomial models, in general. The downside of this link function is that unlike the logistic link function, it is not an injective and surjective mapping from $(0,1)$ to $\mathbb{R}$, but it does eliminate dependence on variances at each point, which can be important in applications.

So in short, if you want variance stabilization, use a variance-stabilizing link function instead of the logit link function used in logistic regression.


McCullagh, P., Nelder, J. (1989). Generalized Linear Models, Second Edition. Chapman & Hall. ISBN: 9780412317606

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.