Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

If I have several hundred coefficients generated by running multiple variable regression model (keeping it as broad as possible by not specifying the nature of the predictors and outcome variable), it would appear to me that I have two options for assessing the significance of any one result.

  1. Identify the largest standardized predictors by calculating z-scores for regression coefficients and assessing how many exceed a fixed threshold (e.g. 2 Z-scores from 0).

  2. Use permutation testing to simulate the distribution of covariates under a null hypothesis (of no covariates being associated with the outcome) and calculate a p-value by comparing the actual results, ranked by their order of significance calculated from Z-scores, to the distributions of ranked coefficients in the permutation tests

My question is: What are the advantages and disadvantages of each approach?

share|improve this question

This is by and large a question about multiple testing. I'm assuming based on the problem description that all the covariates in question have been "centered and scaled".

The first approach is equivalent to the plain vanilla Wald test if using a threshold of 1.96 standard errors from 0 for each coefficient. Obviously, this will affect the familywise error rate of inference about the joint distribution of covariates. You can "correct" this threshold using a Bonferroni correction which will give a higher threshold for all covariates. This is a valid, but conservative approach. Your type 1 error rate will then be far under 0.05 for inference about any one of the parameters being a significant predictors.

The second approach takes into account the correlation of the covariates. The distributions of test statistics under the null hypothesis (e.g. distributions of multiply permuted covariates) can be compared directly rank-wise. This can sometimes lead to some contradictory results (or so I have encountered) where the most marginally significant predictor is not significant by quantiles of most-significant-permuted predictor, but the second most significant predictor is significant.

The biggest problem here, especially if $p \gg n$ is that you waste a lot actually estimating values for coefficients you will not ultimately care about. Why not use a LASSO to constrain variables that are very close to zero to actually take 0 as a value? Then you can spend your data estimating a smaller model more reliably. The LASSO is a fantastic method for feature selection and prediction.

share|improve this answer
Great thanks. Is there anything to say about how each approach would cope with a systematic or random error in the results (due to poor predictor or outcome variable data quality for e.g.)? – 2329alan Aug 14 '14 at 23:08
There are specific methods to handle error-in-variables. For linear regression (and most GLMs), the EM algorithm gives you better operating characteristics. – AdamO Aug 14 '14 at 23:52

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.