Motivation for automated variable selection in case of p>n

I have written the following text as a motivation for using automated variable selection in cases where the number of variables (p) is greater than the number of observations (n). However, I am not completely confident about using the t-test for variable selection when n < p, and then arguing that its unreliability in this context justifies the need for automated variable selection methods.

Could anyone provide suggestions or opinions on this approach? Are there better ways to motivate the use of automated variable selection in the p > n scenario?

Considering a standard linear setup,

$$\underset{(n \times 1)}{y} = \underset{(n \times k)}{x} \ \underset{(k \times 1)}{\beta} + \underset{(n \times 1)}{u} \ \text{with} \ t = 1, \ldots, n$$

the estimates of $$\hat{\beta} = (X'X)^{-1}X'Y$$ can be easily derived using a squared loss function as long as the number of parameters $$k$$ is smaller than the number of observations $$n$$. Under the Gauss-Markov assumptions, $$\hat{\beta}$$ projects $$Y$$ onto $$X$$ while $$X$$ is orthogonal to $$U$$. In this case, the selection of relevant variables would be a trivial problem as a simple t-test could be used to eliminate irrelevant variables that fail to reject the t-test's null hypothesis. However, as soon as $$k \geq n$$, this setup does not work anymore. For $$k = n$$, the estimation results in a perfect fit with the residual variance being 0, rendering the t-test impractical. While for $$k > n , \text{rank}(X) = \text{rank}(X'X) \leq n < k$$, and thus, $$X'X$$ is not invertible, leading to non-identifiable estimates.

• Rather than write a motivation for using automated methods, it would be better to write a motivation for not doing so. Many have been written, but it's always good to spread the word that these automated methods are problematic. Commented Jun 12 at 12:02
• @PeterFlom Yes, agree but need to write a motivation to use it for an applied case, but also trying to cover the problematic parts in the discussion then. Commented Jun 12 at 12:27
• Something worth keeping in mind is that, even when the usual OLS estimator calculation $\hat\beta_{ols} = (X^TX)^{-1}X^Ty$ cannot be solved due to $X$ lacking full rank and lacking an inverse, $\hat\beta_{ols} = \underset{\tilde\beta\in\mathbb R^{p + 1}}{\arg\max}\left\{ \overset{n}{\underset{i = 1}{\sum}}\left( y_i - \left( \tilde \beta_0 + \tilde\beta_1 X_{i, 1} + \dots + + \tilde\beta_1 X_{i, p} \right) \right)^2 \right\}$, what the usual $(X^TX)^{-1}X^Ty$ solves, can exist, with the result being a least squares solution (uniqueness is only guaranteed for $n > p$).
– Dave
Commented Jun 12 at 20:33

What you wrote is false. For instance, no matter how high an order you go for with polynomials, you cannot ever perfectly fit the data when you have multiple distinct $$y$$ values with the same feature values. Such a $$p>n$$ situation cannot achieve a perfect fit.

Your idea of eliminating variables based on the t-test is a form of stepwise regression, the issues with which have been discussed extensively on here, such as at this link.

As an example of a perfect fit being impossible, consider the figure below.

The blue and red points have the same x-value (5) yet different y-values. Therefore, if you try to find some coefficients of a 9th- (p = n) or 10th- (p > n) order polynomial, you can find some coefficients on that polynomial that minimize the square loss, but you will not be able to achieve a perfect fit. No matter what, your fitted polynomial function will not be able to pass through $$(5, y_1)$$ and $$(5, y_2)$$ for $$y_1 \ne y_2$$.

set.seed(2024)
N <- 10
x <- runif(N, 0, 10)
x[1] <- x[2] <- 5
Ey <- x
e <- rnorm(N)
y <- Ey + e
plot(x, y)
points(x[2], y[2], col = 'blue')
points(x[1], y[1], col = 'red')

• Thanks, understood. So how can be argued then that in the p=n case, the t-test is not useable? Commented Jun 12 at 12:29
• @george1994 I think Peter and I are in agreement that the t-test is not particularly useful for your task, even when p<n, so what do you mean?
– Dave
Commented Jun 12 at 13:42
• Yes, I am just trying to find a short formal reasoning that the t-test is not useful in this case to motivate the use for a Elastic-Net. Commented Jun 12 at 14:39
• @george1994 Now you seem to be changing the question. You started out asking how to perform a t-test when you are in a situation that guarantees zero residual variance, and there is no such guarantee. As far as the difficulties of using a t-test to screen out variables (a form of stepwise regression), the link I gave is a good starting point. Do note, however, that even elastic net or LASSO selection would require some delicacy if you want to test hypotheses or calculate confidence intervals. People just seem more willing to ignore that required delicacy when they do stepwise vs LASSO.
– Dave
Commented Jun 12 at 14:48
• @george1994 If you have zero residual variance, then dividing by the standard deviation to get the t-stat doesn't work. However, you can have positive residual variance when p = n or even when p > n, such as if you create your features using polynomials when two outcomes are different despite having the same feature value.
– Dave
Commented Jun 12 at 15:28