0
$\begingroup$

I was reading about linear regression and what I understood is that once we minimize OLS equation we get the beta parameters. Its just like solving a normal equation to get the unknowns. Then why do we do hypothesis testing to see if the dependent variable is related to independent variable because, according to me,if the beta estimate has a value then it definitely must be related.

$\endgroup$
  • $\begingroup$ You are not testing whether the coefficients are related to the dependent variable. You are assessing the likelihood that having encountered a coefficient different than zero, this finding could possibly been a product of randomness or noise, rather than reflect reality. $\endgroup$ – Antoni Parellada Feb 4 '16 at 4:53
  • $\begingroup$ @Antoni Parellada: 'the likelihood of having encountered... different from zeo'? See stats.stackexchange.com/questions/166323/… $\endgroup$ – user83346 Feb 4 '16 at 6:12
5
$\begingroup$

There's a distinction between population and sample.

If I take a fair coin and toss it 20 times, I don't very often get exactly 10 heads (a little over 1 time in 6).

Say I get 8 or 12 heads instead. Should I conclude that the coin has a 60% chance (or a 40% chance) of coming up heads on the basis of 20 tosses? Or would it make more sense to say "well if it was essentially a fair coin, 12 heads isn't actually very surprising, that might just be random variation"?

Similarly, if you want to show that two variables are related, you'd want more than a sample estimate that wasn't exactly zero -- that could happen with purely random, unrelated data -- just by chance. To assert a relationship, you'd probably want evidence that the relationship was more than could reasonably be explained by chance.

$\endgroup$
3
$\begingroup$

Because, in fact, you can't say it definitely must be related.

Data has noise, and noise can create seemingly non-zero relationships that are spurious.

Here's some completely random data, $X$ and $Y$ are generated from independent distributions, and bear no true relationship

> set.seed(154)
> X <- rnorm(100)
> Y <- rnorm(100)

Let's see what a linear model finds

> M <- lm(Y ~ X)
> summary(M)
Coefficients:
            Estimate Std. Error t value Pr(>|t|)    
(Intercept) -0.23966    0.10417  -2.301   0.0235 *
X            0.07338    0.10255   0.716   0.4760 

the coefficient 0.07338 is definitely non-zero, even though the true relationship is nonexistent. Notice though, that the estimated standard error figures this out, so a hypothesis test for the non-zeroness of the coefficient will indeed be helpful.

$\endgroup$
  • 1
    $\begingroup$ I suspect there actually is a true relationship induced by a preliminary search for a good starting seed! See stats.stackexchange.com/questions/80407. $\endgroup$ – whuber Feb 4 '16 at 14:47
  • $\begingroup$ That's actually one of my favorite threads. I honestly didn't search for a seed here, I've been using the same one for a while (inspired by you in fact), it comes from this album: en.wikipedia.org/wiki/154_(album) $\endgroup$ – Matthew Drury Feb 4 '16 at 15:13
  • 2
    $\begingroup$ Well done then! (+1) $\endgroup$ – whuber Feb 4 '16 at 16:32
1
$\begingroup$

You do the test to evaluate in a probabilistic way how likely you are to have observed a slope as or more extreme as the one you've observed purely by chance. As others have pointed out, this is because real data have noise and spurious correlations happen all the time. Thus, any time we report a conclusion, we do so with the idea that there is some chance that the conclusion is based on chance rather than a true relationship between x and y. This is referred to as the Type-1 error rate (and usually given the symbol α).

We can do a simulation experiment to see the effect. I simulate 40 x-y datasets with no relationship between the two (pure random noise), do regressions with hypothesis tests, and then ask how often I see a "significant" correlation with an error rate of 0.05.

set.seed(1345)
N = 40
x = matrix(rnorm(20*N), ncol=N)
y = matrix(rnorm(20*N), ncol=N)
pvals = sapply(1:N, function(i) (summary(lm(y[,i]~x[,i])))$coefficients[2,4])
sum(pvals <= 0.05)

This gives an answer of 2, which is exactly what is expected (5% error rate with 40 tests). We do the test to give some kind of probabilistic weight to our conclusions, where we define an acceptable proportion of the time where we will draw a false conclusion based on a spurious conclusion.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.