1
$\begingroup$

I want to compute from scratch a Wald test to test significance of one coefficient in a logistic regression model. I've been to so many posts and blogs but couldn't find a clear explanation with sample calculation for a NON-EXPERT like me.

My basic understanding says:

1) $H_0: \theta = \theta_0$
2) $H_1: \theta \neq \theta_0$
3) $W = \frac{(\hat{\theta}-\theta_0)^2}{var(\hat{\theta})}$

1) $var(\hat{\theta})$ relates to the specific coefficient of the parameter that I am testing or it is computed on ALL $\theta$s?
If first case, should I train the model with different samples (e.g bootstrapping?) to gather enough values to compute a $var$?

2) Should I use $\theta_0=0$ or $\theta_0=mean($all $\theta$s$)$?

3) The $var(\hat{\theta})$ should reflect the $L(y|\theta)$ or likelihood of the data given the parameter(s?) $\theta$. Then $var(\theta)$ should be calculated from the performance of the model (otherwise how do I calculate the likelihood of observing that data)?

I apologize for my probable bad use of proper terms here and will greatly appreciate a detailed/clear explanation. If an example calculation of p_values and confidence intervals could be added is a plus!

$\endgroup$
1
$\begingroup$

Start by thinking about $\hat\theta$ and $\theta_0$ as vectors rather than as single values. Similarly, think about $var(\hat\theta)$ as a matrix. That's the variance-covariance matrix for the coefficient estimates, with diagonal elements being the variances of individual coefficients and off-diagonal elements being the coefficient covariances needed to estimate variances of linear combinations of coefficients (as needed for estimating errors in predictions based on the model).

As you recognize in the third part of your question, $var(\hat\theta)$ is related to the performance of the model: it's the inverse of the negative of the matrix of second derivatives of the log-likelihood calculated at the maximum likelihood, as explained for example on this page. A "sharper" log-likelihood function (larger second derivatives) will have smaller variances for the coefficients. Standard statistical software should provide a way to get $var(\hat\theta)$ for a model fit by maximizing log likelihood.*

Once you have the variance-covariance matrix you can use a Wald test for hypotheses about multiple linear combinations of coefficients, as described on the Wikipedia page. Consider an hypothesis test on the entire vector of $p$ estimated coefficients,

$$\hat\theta=(\hat\theta_1,\hat\theta_2,...,\hat\theta_i,...,\hat\theta_p)^T.$$

For a test against a corresponding vector of hypothesized values $\theta_0$, the Wald statistic would be the value $(\hat\theta-\theta_0)^T var(\hat\theta)^{-1} (\hat\theta-\theta_0)$, tested against chi-square with $p$ degrees of freedom.**

If you only want to test a single specific coefficient, the formula simplifies to:

$$\frac{(\hat\theta_i-\theta_{0i})^2}{var(\hat\theta_i)}$$

the Wald statistic for a single coefficient, tested against chi-square with 1 degree of freedom.

If you have a model with nonlinear or interaction terms involving a predictor, a test for its significance including such terms is similar to the above, testing a Wald statistic based on the subset of coefficients that include it.

In general, what components you include in $\hat\theta$ and $\theta_0$ for a Wald test depend on your specific hypothesis about the coefficients.


*There can be value in generating a variance-covariance matrix from multiple bootstrap samples, as you suggest in the first part of your question.

**Use the identity matrix of size $p$ as the matrix called $R$ in the formula on the Wikipedia page.

$\endgroup$
  • $\begingroup$ Thank you for all the details. 1- $var(\hat{\theta})$ is covariance matrix of x and y. Are x and y the coefficients and the performances? 2-How do I generate covariance matrix without bootstrap? $\endgroup$ – aerijman Jun 3 '19 at 4:00
  • $\begingroup$ @aerijman the $var(\hat\theta)$ matrix is the variance-covariance matrix among the regression coefficient estimates, not among the predictors/outcome themselves. Statistical programs typically provide a function to extract that matrix from a model. For example, for a logistic regression performed with the glm() function in R, you use the vcov() function on the object generated by glm(). $\endgroup$ – EdM Jun 3 '19 at 13:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.