3
$\begingroup$

I see that the Wald test can be performed on the model obtained from linear regression as shown here. I understand that it gives information whether a predictor is adding significant value to outcome variable. However, I am not clear why Wald test needs to be done since this information is already available from linear regression. Any clarification will be appreciated.

Edit: I checked with some linear regressions and Wald test for hypothesis (predictor_name = 0) gives almost exactly same P-value as the linear regression fitted model.summary() output (statsmodels package). So where is the need to do Wald test?

My main aim is: To determine "which predictors are independently affecting dependent variable".

$\endgroup$
8
  • $\begingroup$ Could you please specify the null and alternative hypotheses you are interested in? The second link you have provided is about a partial F-test, not about a Wald test. $\endgroup$
    – Michael M
    Jul 19 '20 at 6:46
  • $\begingroup$ I am interested in determining "which predictors are independently affecting response or dependent variable's value". I am not sure how to put this in null and alternate hypothesis, but I think former is that coef of a predictor = 0. I have to then test each predictor separately. $\endgroup$
    – rnso
    Jul 19 '20 at 6:49
  • $\begingroup$ Based on the model equation $y = \beta_o + \beta_1 x + \beta_2 z + \varepsilon$, how would you translate that statement regarding the $\beta_i$? How many hypotheses will need to be tested with how many models? $\endgroup$
    – Michael M
    Jul 19 '20 at 6:54
  • $\begingroup$ The null hypothesis will be Bi = 0. Each predictor will need to be tested separately on same model created with equation dv ~ iv1 + iv2 + iv3 ... . Hence, I can find out if a predictor is contributing to dependent variables value or not. But I get same p-values as from linear regression. So what is the advantage of Wald test? And answer in this link shows how to do it: stackoverflow.com/questions/50117157/… e.g.: hypothesis_1 = '(GNP = 0)' $\endgroup$
    – rnso
    Jul 19 '20 at 7:04
  • $\begingroup$ Its the same then. The usual tests that come along with each parameter in a linear regression are Wald tests and they test the null hypothesis $\beta_i = 0$, magically with a single model fit. In your link they test something different (two parameters simultaneouly). $\endgroup$
    – Michael M
    Jul 19 '20 at 7:35
3
$\begingroup$

You have to be very careful about what you mean when you say that your aim is to determine "which predictors are independently affecting the dependent variable."

Multiple regression adjusts for the values of other predictors when evaluating the association of a predictor with outcome. If you find that predictor $x_1$ isn't significantly associated with outcome when considered individually in a simple regression but it is when other predictors are considered along with it in a multiple regression, do you consider $x_1$ to be "independently affecting the dependent variable"?

This becomes more of an issue when the model includes interactions. With an interaction you can't really talk about whether $x_1$ is "independently affecting the dependent variable," because the model already implies that the association between $x_1$ and outcome depends on the values of the predictors that it interacts with.

In both of those cases, $x_1$ can be closely associated with outcome even if it isn't doing so independently of other predictors. I don't think that in either case you'd want to just ignore $x_1$.

With that warning, let's consider the usual coefficient values and tests reported by statistical software, and what Wald tests add.

The usual output from a multiple regression model contains estimates of the coefficients for each predictor and interaction term individually, along with the associated standard errors and statistical significance tests based on the ratio of the coefficient to the standard error. In ordinary least squares the test is a t-test, appropriate for situations with normally distributed errors in which you are estimating both the mean values and the standard errors from the data. In generalized linear models like logistic regression the t-test isn't valid so a normal approximation is used. The statistical test is then a z-test.

Two things to note. First, as the number of cases gets large, the distinction between the t-test and z-test becomes less and less important and the two tests will provide essentially the same result. Second, a z-test on a single coefficient, as in the usual output from regression software for a generalized linear model, is functionally the same as a Wald test. So with generalized linear models you can even say that the Wald test is the default test on the individual coefficients.

Wald tests are useful when you need to consider the association of multiple predictors with outcome together. An obvious example is when a predictor is involved in interaction terms with other predictors. You might want to know if any of the direct or interaction terms involving it is significantly different from zero. But there are other examples, too.

Consider a multi-level categorical predictor, even with just 3 levels. With standard treatment coding of that predictor, the reported coefficients are for the differences of each of 2 levels from the reference level. The apparent "significance" of one level thus can depend on the choice of the reference level. What you really care about is the association of the entire categorical variable with outcome, including all levels regardless of choice of reference level.

Or say that you have modeled a continuous predictor as a spline, resulting in multiple coefficients associated with it. Is that predictor associated with outcome when combining all those terms? Do the non-linear coefficients add anything?

Wald tests* provide a simple and general way to test such hypotheses. The usual application would be a test of whether all of a set of coefficients is 0. The test takes into account not only the variances of the individual coefficient estimates but also the covariances among them, which is important with correlated predictors that typically are found in practice.

So for considering whether $x_1$ is associated with outcome while considering all of its interaction terms, you do a Wald test on all of those coefficients. For evaluating a multi-level categorical predictor you do a Wald test on the coefficients for all levels of the predictor (necessarily excluding the reference level). For evaluating a spline-modeled continuous predictor, you do a test on all coefficients involving it. For evaluating whether the non-linear spline terms are adding anything, you evaluate all of their coefficients while omitting the linear term.

I don't use statsmodels so I can't speak to whether or under what conditions it performs Wald tests. If it only reports tests on individual coefficients then for ordinary least squares regression it probably is reporting t-tests, and for generalized models you might consider the coefficient tests to be functionally the same as Wald tests.

But such reports of single coefficients don't handle multi-level categorical predictors, polynomial- or spline-modeled continuous predictors, or predictors involved in interactions very well. To determine whether such predictors are "affecting the dependent variable," the Wald test provides a useful tool.


*In ordinary least squares regression, it's possible to use the chi-square statistic from a Wald test together with the error estimate from the regression to do an F-test rather than to depend on the asymptotic normality assumed by the Wald test. For simplicity, I'll include that analysis under "Wald test" here.

$\endgroup$
2
  • $\begingroup$ Thanks for a detailed answer. It seems there are many limitations if situation is complex (e.g. interaction term). It will be very useful, if you could clarify what all can be reliably concluded from results of simple multiple regression of y ~ xnum + xcat2 , where xnum is a numeric variable and xcat2 is a categorical variable with only 2 levels. $\endgroup$
    – rnso
    Jul 19 '20 at 15:52
  • 1
    $\begingroup$ @rnso Even if OLS assumptions are met, omitting from the model any predictors associated both with y and with either of these predictors runs a risk of omitted-variable bias. Such omissions could include non-linearity in xnum, interactions between those 2 predictors, or interactions with other predictors. Those are all limits to "reliability." Don't push "conclusions" too far. "All models are wrong; some models are useful.". $\endgroup$
    – EdM
    Jul 20 '20 at 12:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.