2
$\begingroup$

I'm running a multiple linear regression on a set of sports data. When I run the regression on one season, which has 380 data points and which I thought was a fair amount, I get quite a high p-value on one of my independent variables. However, when I run the regression on all my data points (I have more than 3000 data points in total), the p-value decreases from .97 to .02. As I add more data points, the p-value decreases further. My question is: is my variable really significant or am I just decreasing the p-value by adding more data points?

$\endgroup$
  • 1
    $\begingroup$ The answers are (likely) yes, and yes. This is actually a version of a fairly frequently asked question. One example of the genre is discussed in this answer $\endgroup$ – Glen_b -Reinstate Monica May 10 '14 at 3:12
  • 2
    $\begingroup$ This is what lead John Myles White to comment that "p-values measure effort, not truth". $\endgroup$ – Livius May 10 '14 at 11:25
2
$\begingroup$

Let's say that your independent variable is $x_i$ and its regression coefficient is $\beta_i$. The p-value for $\beta_i$ is $P(t<| t^* |)+P(t>|t^*|)$ where $t^*=\frac{\beta_i}{\sqrt{(X'X)^{-1}_{ii}\frac{RSS}{n-q}}}$. $RSS$ is the residual sum of squares.

The p-value is large when $|t^*|$ is small, small when $|t^*|$ is large. But when $n$ grows, $RSS/(n-q)$ get smaller and $|t^*|$ larger, so the p-value decreases just because $n$ grows.

This is why "in large samples is more appropriate to choose a size of 1% or less rather than the 'traditional' 5%." (M. Verbeek, A Guide to Modern Econometrics, 3rd edition, §2.5.7, p. 32). If you choose 1%, your coefficient is not statitically significant when $p=0.02$.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ quite good way to explain the nuances. $\endgroup$ – Subhash C. Davar May 14 '14 at 9:52
1
$\begingroup$

A t-test of a predictor coefficient in a multiple regression model compares the coefficient's value to the null hypothesis, which generally defaults to zero. The p value for this test represents the probability that, if you were to replicate your study and the null hypothesis is true (i.e., the parameter is exactly zero in the entire population), your multiple regression performed on the new data would produce a regression coefficient that differs from zero at least as much as your current data does. Because sampling error affects smaller samples more strongly than larger samples, this probability decreases as the sample size increases. Your analysis on the larger dataset produces a more precise estimate of the population parameter because it is less likely to be affected by sampling error. Evidently your estimate is not zero, and you can be more confident that the population parameter is not zero by using your larger dataset.

However, be careful not to confuse statistical significance with practical significance. I'm guessing that the difference between your regression coefficient and zero is pretty small if the p = .02 with N > 3K. You might reasonably expect a new sample of 3K to produce a coefficient on the same side of zero as your sample has, but probably not much further from zero. If that difference between your coefficient and zero isn't interesting to you, don't feel obligated to interpret it substantively just because p < .05.

| cite | improve this answer | |
$\endgroup$
1
$\begingroup$

What does the $R^2$ do, what your coefficients?

As Meehl said, everything is correlated with (almost) everything else - to some degree. If your variables are of this kind, if there is any, and be it ever so meagre, effect in the population, your p will decrease as your sample size increases. So maybe your variables are of this normal kind, where some degree of correlation exists. Maybe your variable is statistically significant. However, the more interesting question is probably: are they correlated to an interesting degree? And for that, you want to look at your coefficients and $R^2$s, and preferably some estimate of the reliability of these estimates, like their Confidence Intervals.

Also, you're currently comparing your favourite model to a very boring model; the model that there is nothing going on. And if you believe Meehl, that hypothesis has a very low a priori probability, and it isn't so interesting if your model beats it. Instead, you could construct a non-trivial hypothesis and check if that one is better than your favourite model.

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.