Wald test standard error

Question

I want to compute from scratch a Wald test to test significance of one coefficient in a logistic regression model. I've been to so many posts and blogs but couldn't find a clear explanation with sample calculation for a NON-EXPERT like me.

My basic understanding says:

1) $H_0: \theta = \theta_0$
2) $H_1: \theta \neq \theta_0$
3) $W = \frac{(\hat{\theta}-\theta_0)^2}{var(\hat{\theta})}$

1) $var(\hat{\theta})$ relates to the specific coefficient of the parameter that I am testing or it is computed on ALL $\theta$s?
If first case, should I train the model with different samples (e.g bootstrapping?) to gather enough values to compute a $var$?

2) Should I use $\theta_0=0$ or $\theta_0=mean($all $\theta$s$)$?

3) The $var(\hat{\theta})$ should reflect the $L(y|\theta)$ or likelihood of the data given the parameter(s?) $\theta$. Then $var(\theta)$ should be calculated from the performance of the model (otherwise how do I calculate the likelihood of observing that data)?

I apologize for my probable bad use of proper terms here and will greatly appreciate a detailed/clear explanation. If an example calculation of p_values and confidence intervals could be added is a plus!

Answer 1

Start by thinking about $\hat\theta$ and $\theta_0$ as vectors rather than as single values. Similarly, think about $var(\hat\theta)$ as a matrix. That's the variance-covariance matrix for the coefficient estimates, with diagonal elements being the variances of individual coefficients and off-diagonal elements being the coefficient covariances needed to estimate variances of linear combinations of coefficients (as needed for estimating errors in predictions based on the model).

As you recognize in the third part of your question, $var(\hat\theta)$ is related to the performance of the model: it's the inverse of the negative of the matrix of second derivatives of the log-likelihood calculated at the maximum likelihood, as explained for example on this page. A "sharper" log-likelihood function (larger second derivatives) will have smaller variances for the coefficients. Standard statistical software should provide a way to get $var(\hat\theta)$ for a model fit by maximizing log likelihood.*

Once you have the variance-covariance matrix you can use a Wald test for hypotheses about multiple linear combinations of coefficients, as described on the Wikipedia page. Consider an hypothesis test on the entire vector of $p$ estimated coefficients,

$$\hat\theta=(\hat\theta_1,\hat\theta_2,...,\hat\theta_i,...,\hat\theta_p)^T.$$

For a test against a corresponding vector of hypothesized values $\theta_0$, the Wald statistic would be the value $(\hat\theta-\theta_0)^T var(\hat\theta)^{-1} (\hat\theta-\theta_0)$, tested against chi-square with $p$ degrees of freedom.**

If you only want to test a single specific coefficient, the formula simplifies to:

$$\frac{(\hat\theta_i-\theta_{0i})^2}{var(\hat\theta_i)}$$

the Wald statistic for a single coefficient, tested against chi-square with 1 degree of freedom.

If you have a model with nonlinear or interaction terms involving a predictor, a test for its significance including such terms is similar to the above, testing a Wald statistic based on the subset of coefficients that include it.

In general, what components you include in $\hat\theta$ and $\theta_0$ for a Wald test depend on your specific hypothesis about the coefficients.

---

*There can be value in generating a variance-covariance matrix from multiple bootstrap samples, as you suggest in the first part of your question.

**Use the identity matrix of size $p$ as the matrix called $R$ in the formula on the Wikipedia page.