A comparison of confidence intervals methods on an example from ISL
The book "Introduction to Statistical Learning" by Tibshirani, James, Hastie provides an example on page 267 of confidence intervals for polynomial logistic regression degree 4 on the wage data. Quoting the book:
We model the binary event $wage>250$ using logistic regression with a degree-4 polynomial. The fitted posterior probability of wage exceeding $250,000 is shown in blue, along with an estimated 95 % confidence interval.
Below is a quick recap of two methods for constructing such intervals as well as comments on how to implement them from scratch
Wald / Endpoint transformation intervals
- Compute the upper and lower bounds of the confidence interval for the linear combination $x^T\beta$ (using the Wald CI)
- Apply a monotonic transformation to the endpoints $F(x^T\beta)$ to obtain the probabilities.
Since $Pr(x^T\beta) = F(x^T\beta)$ is a monotonic transformation of $x^T\beta$
$$ [Pr(x^T\beta)_L \leq Pr(x^T\beta) \leq Pr(x^T\beta)_U] = [F(x^T\beta)_L \leq F(x^T\beta) \leq F(x^T\beta)_U] $$
Concretely this means computing $\beta^Tx \pm z^* SE(\beta^Tx)$ and then applying the logit transform to the result to get the lower and upper bounds:
$$[\frac{e^{x^T\beta - z^* SE(x^T\beta)}}{1 + e^{x^T\beta - z^* SE(x^T\beta)}}, \frac{e^{x^T\beta + z^* SE(x^T\beta)}}{1 + e^{x^T\beta + z^* SE(x^T\beta)}},] $$
Computing the standard error
Maximum Likelihood theory tells us that the approximate variance of $x^T\beta$ can be calculated using the covariance matrix $\Sigma$ of the regression coefficients using
$$ Var(x^T\beta) = x^T \Sigma x$$
Define the design matrix $X$ and the matrix $V$ as
$$\textbf{X = }\begin{bmatrix} 1 & x_{1,1} & \ldots & x_{1,p} \\ 1 & x_{2,1} & \ldots & x_{2,p} \\ \vdots & \vdots & \ddots & \vdots \\ 1 & x_{n,1} & \ldots & x_{n,p}
\end{bmatrix} \ \ \ \ \textbf{V = } \begin{bmatrix} \hat{\pi}_{1}(1 - \hat{\pi}_{1}) & 0 & \ldots & 0 \\ 0 & \hat{\pi}_{2}(1 - \hat{\pi}_{2}) & \ldots & 0 \\ \vdots & \vdots & \ddots & \vdots \\ 0 & 0 & \ldots & \hat{\pi}_{n}(1 - \hat{\pi}_{n}) \end{bmatrix}$$
where $x_{i,j}$ is the value of the $j$th variable for the $i$th observations and $\hat{\pi}_{i}$ represents the predicted probability for observation $i$.
The covariance matrix can then be found as: $\Sigma = \textbf{(X}^{T}\textbf{V}\textbf{X)}^{-1}$ and the standard error as $SE(x^T\beta) = \sqrt{Var(x^T\beta)}$
The 95% confidence intervals for the predicted probability can then be plotted as
Delta method confidence intervals
The approach is to compute the variance of a linear approximation of the function $F$ and use this to construct large sample confidence intervals.
$$ \text{Var}[F\mathbf{(x^T \hat \beta)}] \approx \nabla F^T \ \Sigma \ \nabla F $$
Where $\nabla$ is the gradient and $ \Sigma$ the estimated covariance matrix. Note that in one dimension:
$$\frac{\partial F(x\beta)}{\partial \beta} = \frac{\partial F(x\beta)}{\partial x\beta} \frac{\partial x\beta}{\partial \beta} = x f(x\beta)$$
Where $f$ is the derivative of $F$. This generalizes in the multivariate case
$$ \text{Var}[F\mathbf{(x^T \hat \beta)}] \approx f^T \ \mathbf{x^T} \ \Sigma \ \mathbf{x} \ f $$
In our case F is the logistic function (which we will denote $\pi(x^T\beta)$) whose derivative is
$$ \pi'(x^T\beta) = \pi (x^T\beta) (1 - \pi (x^T\beta) ) $$
We can now construct a confidence interval using the variance computed above.
$$ C.I. = [Pr(x\hat \beta) - z^* \sqrt{\text{Var}[ \pi(x \hat \beta) ]} \leq Pr(x\hat \beta) + z^* \sqrt{\text{Var}[ \pi(x \hat \beta) ]} ]$$
In vector form for the multivariate case
$$ C.I. = \mathbf{[\pi(x^T\hat \beta) \pm z^* \sqrt{ \left(\pi(x^T \hat \beta) (1 - \pi(x^T \hat \beta) ) \right)^T x^T \ \ \text{Var}[ \hat \beta] \ \ x \ \ \pi(x^T \hat \beta) (1 - \pi(x^T \hat \beta) ) ]}}$$
- Note that $\mathbf{x}$ represent a single data point in $\mathbb{R}^{p+1}$, i.e. a single row of the design matrix $X$
A open ended conclusion
A look at the Normal QQ plots for both the probabilities and the negative log odds show that neither are normally distributed. Could this explain the difference ?
Source:
