How to get p-values in high-dimensional settings?

It is easy to get p-values from a linear regression and related methods (t-test, anova, logistic regression), but how can one get p-values in a high dimensional setting (p >> n)? I understand that these problems wouldn't even be solvable, because the necessary collineairity in the data and for the low power. However, it is possible to make very useful models even with high dimensional data, but how to establish any statistical significance?

1. how to establish if the two groups are statistically significantly different, based on a high dimensional feature set?
2. how to establish if an outcome variable, is statistically significantly related to many input variables?
3. How to establish which features are statistically significantly related to an outcome variable, in a model containing many features?

For example, I have a million genes and 200 patients with and without cancer, I want to know if there is an genetic component of this cancer, and if so, which genes are associated.

You might want to read Chapter 18, "High-Dimensional Problems: When $p \gg N$", of Elements Of Statistical Learning.

Regarding your specific points (not in order):

The widely-accepted procedure of the false discovery rate (see Section 18.7 in Elements Of Statistical Learning") addresses your point 3. While there are variations, the basic idea is to calculate the p-value for each feature, then sort them in increasing order. Following that, you conceptually plot a graph whose x axis is $1, ..., N$, and plot the $N$ p-values. You also run a line starting at the origin with slope $\alpha$, and select all features whose p-values are below the line, until the first one that is over the line. It can be shown that under certain conditions, $\alpha$ determines the false-positive rate, irrespective of $N$.

For 2., you can run cross-validation (if you have enough computing power, using the entire 200 folds - note that this is trivially parallel). In each iteration, run some massive dimension reduction technique (e.g., supervised principal components) (see Section 18.6 in Elements Of Statistical Learning") , and check the performance on the holdout data.

1., in its most general form, seems to me unsolvable. With enough (random) features, two sets will differ. If you restrict things, though, to questions such as the values for feature $i$ in set 1 are higher (lower) than those in set $2$, you can again use FDR (false discovery rate). Once again, find the p-value for any such hypothesis, and select those beneath the line.

• In addition, I'd like to mention the work on post-selection inference by Ryan Tibshirani and others (paper here (arxiv.org/abs/1401.3889)) for least angle regression, the lasso and forward stepwise regression. The R package selectiveInference contains the tools. Also see a presentation (youtube.com/watch?v=tImP6KxIUKQ) by Rob Tibshirani. Commented Jul 3, 2017 at 5:32
• Elemnts of Statistical Learning mostly talks about making good predictions in high dimensional settings, and the message basically boils down to use regularization. However, they do not discuss significance testing. E.g my model seems to predict better than chance, with variables I selected with some variable selection method, but what is the chance that this would happen if there is no signal in the data? (what are the p-values?) @ErikL Seems promising, thanks. I will check it out. Commented Jul 3, 2017 at 8:20
• @rep_ho Not sure I understand your question. E.g., your point 3 seems like a classic case for the FDR. The case with the 200 cancer patients and the genetic features, is, IIRC, almost exact the case motivating FDR in "Elements Of Statistical Learning". Commented Jul 3, 2017 at 8:31
• Ah, my bad, I probably didn't scroll down enough. Can you edit your answer to reference 18.7? Commented Jul 3, 2017 at 8:37
• @rep_ho No problem. Added references to 18.7 and 18.6. Commented Jul 3, 2017 at 11:45