14
$\begingroup$

I am familiar with using multiple linear regressions to create models of various variables. However, I was curious if regression tests are ever used to do any sort of basic hypothesis testing. If so, what would those scenarios/hypotheses look like?

$\endgroup$
  • 1
    $\begingroup$ Can you explain further what you mean? It is very common to test whether the slope parameter for a variable is different from zero. I would call that "hypothesis testing". Are you unaware of that, or do you mean something different? What constitutes a scenario for your purposes? $\endgroup$ – gung Apr 2 '12 at 13:24
  • $\begingroup$ I am unaware of that. I was also unsure if regression-based analysis is used for any other sort of hypothesis testing (perhaps about the significance of one variable over another, etc). $\endgroup$ – cryptic_star Apr 2 '12 at 14:04
25
$\begingroup$

Here is a simple example. I don't know if you are familiar with R, but hopefully the code is sufficiently self-explanatory.

set.seed(9)        # this makes the example reproducible
N = 36
    # the following generates 3 variables:
x1 =     rep(seq(from=11, to=13),           each=12)
x2 = rep(rep(seq(from=90, to=150, by=20),   each=3 ), times=3)
x3 =     rep(seq(from=6,  to=18,  by=6 ),  times=12)
cbind(x1, x2, x3)[1:7,]    # 1st 7 cases, just to see the pattern
      x1  x2 x3
 [1,] 11  90  6
 [2,] 11  90 12
 [3,] 11  90 18
 [4,] 11 110  6
 [5,] 11 110 12
 [6,] 11 110 18
 [7,] 11 130  6 
    # the following is the true data generating process, note that y is a function of
    #   x1 & x2, but not x3, note also that x1 is designed above w/ a restricted range,
    #   & that x2 tends to have less influence on the response variable than x1:
y  = 15 + 2*x1 + .2*x2 + rnorm(N, mean=0, sd=10)

reg.Model = lm(y~x1+x2+x3)    # fits a regression model to these data

Now, lets see what this looks like:

. . . 
Coefficients:
            Estimate Std. Error t value Pr(>|t|)   
(Intercept) -1.76232   27.18170  -0.065  0.94871   
x1           3.11683    2.09795   1.486  0.14716   
x2           0.21214    0.07661   2.769  0.00927 **
x3           0.17748    0.34966   0.508  0.61524   
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 
. . . 
F-statistic: 3.378 on 3 and 32 DF,  p-value: 0.03016 

We can focus on the "Coefficients" section of the output. Each parameter estimated by the model gets its own row. The actual estimate itself is listed in the first column. The second column lists the Standard Errors of the estimates, that is, an estimate of how much estimates would 'bounce around' from sample to sample, if we were to repeat this process over and over and over again. More specifically, it is an estimate of the standard deviation of the sampling distribution of the estimate. If we divide each parameter estimate by its SE, we get a t-score, which is listed in the third column; this is used for hypothesis testing, specifically to test whether the parameter estimate is 'significantly' different from 0. The last column is the p-value associated with that t-score. It is the probability of finding an estimated value that far or further from 0, if the null hypothesis were true. Note that if the null hypothesis is not true, it is not clear that this value is telling us anything meaningful at all.

If we look back and forth between the Coefficients table and the true data generating process above, we can see a few interesting things. The intercept is estimated to be -1.8 and its SE is 27, whereas the true value is 15. Because the associated p-value is .95, it would not be considered 'significantly different' from 0 (a type II error), but it is nonetheless within one SE of the true value. There is thus nothing terribly extreme about this estimate from the perspective of the true value and the amount it ought to fluctuate; we simply have insufficient power to differentiate it from 0. The same story holds, more or less, for x1. Data analysts would typically say that it is not even 'marginally significant' because its p-value is >.10, however, this is another type II error. The estimate for x2 is quite accurate $.21214\approx.2$, and the p-value is 'highly significant', a correct decision. x3 also could not be differentiated from 0, p=.62, another correct decision (x3 does not show up in the true data generating process above). Interestingly, the p-value is greater than that for x1, but less than that for the intercept, both of which are type II errors. Finally, if we look below the Coefficients table we see the F-value for the model, which is a simultaneous test. This test checks to see if the model as a whole predicts the response variable better than chance alone. Another way to say this, is whether or not all the estimates should be considered unable to be differentiated from 0. The results of this test suggests that at least some of the parameter estimates are not equal to 0, anther correct decision. Since there are 4 tests above, we would have no protection from the problem of multiple comparisons without this. (Bear in mind that because p-values are random variables--whether something is significant would vary from experiment to experiment, if the experiment were re-run--it is possible for these to be inconsistent with each other. This is discussed on CV here: Significance of coefficients in multiple regression: significant t-test vs. non-significant F-statistic, and the opposite situation here: How can a regression be significant yet all predictors be non-significant, & here: F and t statistics in a regression.) Perhaps curiously, there are no type I errors in this example. At any rate, all 5 of the tests discussed in this paragraph are hypothesis tests.

From your comment, I gather you may also wonder about how to determine if one explanatory variable is more important than another. This is a very common question, but is quite tricky. Imagine wanting to predict the potential for success in a sport based on an athlete's height and weight, and wondering which is more important. A common strategy is to look to see which estimated coefficient is larger. However, these estimates are specific to the units that were used: for example, the coefficient for weight will change depending on whether pounds or kilograms are used. In addition, it is not remotely clear how to equate / compare pounds and inches, or kilograms and centimeters. One strategy people employ is to standardize (i.e., turn into z-scores) their data first. Then these dimensions are in common units (viz., standard deviations), and the coefficients are similar to r-scores. Moreover, it is possible to test if one r-score is larger than another. Unfortunately, this does not get you out of the woods; unless the true r is exactly 0, the estimated r is driven in large part by the range of covariate values that are used. (I don't know how easy it will be to recognize, but @whuber's excellent answer here: Is $R^2$ useful or dangerous, illustrates this point; to see it, just think about how $r=\sqrt{r^2}$.) Thus, the best that can ever be said is that variability in one explanatory variable within a specified range is more important to determining the level of the response than variability in another explanatory variable within another specified range.

$\endgroup$
2
$\begingroup$

The essential test in regression models is the Full-Reduced test. This is where you are comparing 2 regression models, the Full model has all the terms in it and the Reduced test has a subset of those terms (the Reduced model needs to be nested in the Full model). The test then tests the null hypothesis that the reduced model fits just as well as the full model and any difference is due to chance.

Common printouts from statistical software include an overall F test, this is just the Full-Reduced test where the reduced test is an intercept only model. They also often print a p-value for each individual predictor, this is just a series of Full-Reduced model tests, in each one the reduced model does not include that specific term. There are many ways to use these tests to answer questions of interest. In fact pretty much every test taught in a introductory stats course can be computed using regression models and the Full-Reduced test and the results will be identical in many cases and a very close approximation in the few others.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.