I am reading Elements of Statistical Learning, specifically section 5.2.2, an example on the South African Heart Disease dataset.

The idea is to model the logit of the conditional probability of the output as linear in the basis representation. Specifically, we have

$$\text{logit}(P(Y|X))=\theta_0+\sum_{i=1}^p h_i(X_i)^T\theta_i$$

where there are $p$ features and $h_i(X_i)$ are the basis functions to be used for feature $i$, and $\theta_i$ are the corresponding vectors of coefficients. Then, we can model the whole thing as lienar via


where $\theta$ is the stacked $\theta_i$'s, and $h(X)$ is the stacked $h_i(X_i)$'s. More succinctly, if we take an $N\times df$ matrix $\mathbf{H}$ as the basis values (here $N$ for number of data and $df$ represents the total number of degrees of freedom, i.e. unique basis functions), we can apply linear regression analysis with $\mathbf{H}$ as the design matrix.

The book then states that we estimate the covariance of $\theta$ via


where $\mathbf{W}$ is the "diagonal weight matrix from the logistic regression". Why is the $\mathbf{W}$ here? I'm confused as to how the weigh matrix from the logistic regression comes into play here.


1 Answer 1


If $p=P(Y|X)$ is the vector of fitted values, then $W$ is the diagonal matrix with diagonal $1/(p(1-p))$. This answer explains why it's that rather than $p(1-p)$ as you might expect from thinking about variances

  • $\begingroup$ Could you explain why the covariance of $\hat{\theta}$ is estimated via $\hat{\mathbf{\Sigma}}=(\mathbf{H}^T\mathbf{W}\mathbf{H})^{-1}$? I am thinking it should be $\mathbb{E}[(\hat{\theta}-\mathbb{E}[\hat{\theta}])(\hat{\theta}-\mathbb{E}[\hat{\theta}])^T]$. How does this simplify to $(\mathbf{H}^T\mathbf{W}\mathbf{H})^{-1}$? Or am I misunderstanding something? $\endgroup$
    – Max
    Dec 7, 2022 at 2:53
  • $\begingroup$ If we let $\text{L}$ be the vector of logits (e.g. $\text{logit}(P(Y=1|X=x_1)),\dots,\text{logit}(P(Y=1|X=x_n))$), then with linear regression analysis , we have something like $\text{Var}(\hat{\theta})=(\mathbf{H}^T\mathbf{H})^{-1}\sigma^2$ where $\sigma^2$ is the variance of the "error". In this case, it would be something like $p(1-p)$ as we are estimating Bernoullis, and moving it inside the inverse would be the reason why we need $\frac{1}{p(1-p)}$ in the $\mathbf{W}$ matrix, rather than $p(1-p)$. Does this sound like the correct intuition, and could you explain more details if possible? $\endgroup$
    – Max
    Dec 7, 2022 at 6:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.