I have a data with 20k samples. It includes 6 categorical and 2 continuous variables. When I performed multiple linear regression and ANOVA to test the variables significance, it shows p-values 1e-16 for all variables. I suppose it is because of data size and has to do something with power. However, I am quite sure that 2 varables are not important for the regression at all. Could I use sum of squares as a quatitative measure of significance instead of p-values or something else? I can also reduce the size of data, but how much? What is an optimal size which I should use?

  • 1
    $\begingroup$ Not a programming question - try stats.stackexchange for statistics questions. I would recommend looking at (standardized) effect sizes to determine non-statistical significance. You could also use regularization techniques such as LASSO. $\endgroup$ – Gregor Thomas Oct 6 '16 at 20:19
  • $\begingroup$ I'm not sure how you would use Sum of Squares as a variable-level metric - maybe fit a model with and without the variable and compare? And come up with some way to decide whether the difference matters or not? I wouldn't recommend it... $\endgroup$ – Gregor Thomas Oct 6 '16 at 20:21
  • $\begingroup$ There are a lot of ways to do variable selection; see Chapter 6 of ISLR. Most methods depend on a quantity like AIC or BIC rather than RSS directly. $\endgroup$ – alistaire Oct 6 '16 at 20:36
  • $\begingroup$ Generally speaking, more data is better. This is the idea behind much of statistics such as LLN, CLT, etc.... There is, therefore, no optimal size of data. That said, you'll want to use an information criterion (AIC, BIC, etc...) for model selection. Try a bunch of different specification (i.e. specification with different co-variates) and pick the one with the smallest information criterion. Using RSS for model selection will not work. RSS will always favor more variables to less, because each additional variable will provide additional, but potentially really small, helpful info. $\endgroup$ – Jacob H Oct 6 '16 at 21:22
  • $\begingroup$ (cont) information criterion, in contrast, penalize the introduction of each additional variable and therefore pick the model which explains the LHS variable the best with the least amount of parameters. Additionally, you can use a LASSO for model selection. Run the LASSO with all your data. Those variables shrunk to zero should be omitted from your OLS regression. However, be wary of shrinkage estimates such as LASSO, if your goal is parameter estimation. Shrinkage estimators generate bias coefficients and therefore should really only be used for forecasting (or model selection). $\endgroup$ – Jacob H Oct 6 '16 at 21:28

If you do an ANOVA, you can see the sum of squared variance explained by each variable. This will help you judge how much explanatory power is added.

Here's some sample R code:

tmp <- data.frame(a = rnorm(100), b = rnorm(100))
tmp$c = tmp$a + 3*tmp$b + rnorm(100)
anova(lm(c ~ a + b, tmp))

Alternatively you can look at the R-squared of regression models with and without the relevant variables, and take the difference. This tells you how much extra explanatory work the variables are doing. I think the two approaches are almost identical.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.