2
$\begingroup$

Suppose I have X, k*n, where $M=X'X$. Suppose $n>>k$, and $rank(M) =k-1$. Suppose $\lambda_1, \cdots, \lambda_{k-1}$ are the eigen values of M. Under the assumption that the columns of X are random, what can we say about the distribution of $\lambda_{max}$?

Here is my applied problem. Suppose the columns of X are observed values of n random variables. We only have k observations for each variable. I want to know if there exists an underlying variable that can explain them. If there is none, then $\lambda_i$ should be close to $\frac{1}{k-1}$. I want to know how large $\lambda_{max}$ has to be for us to conclude that at least one significant explanatory variable exists. This is similar to the condition number concept, except for the fact that M is not of full rank.

Edit: take n=2000, k=21.

$\endgroup$
6
  • 1
    $\begingroup$ $M$ isn't full rank? $\endgroup$
    – Dave
    Jan 17 at 0:30
  • $\begingroup$ Yes, it is not. $\endgroup$
    – deb
    Jan 17 at 1:44
  • $\begingroup$ What distributions do the columns of $X$ follow? If all distributions are fair-game, I have written a simulation where two different distributions produce wildly different sets of maximum eigenvalues of $M$. $\endgroup$
    – Dave
    Jan 17 at 2:26
  • $\begingroup$ Assume Gaussian with fat tails $\endgroup$
    – deb
    Jan 17 at 16:17
  • $\begingroup$ What does that mean? Gaussians do not have fat tails (at least not compared to Gaussians). Even different Gaussians are giving me radically different maximum eigenvalue distributions, though. $\endgroup$
    – Dave
    Jan 17 at 16:20

1 Answer 1

0
$\begingroup$

There seems not to be a universal distribution of this maximum eigenvalue. In the simulation below, I draw from $t_5$ and $t_6$ distributions to create the first twenty columns of $X$. Then I create the 21st column by adding some of the first twenty columns. This meets the criteria set out in the question

Even just changing the degrees of freedom in the $t$-distribution results in wildly different distributions of the maximum eigenvalue of $X^TX$.

library(ggplot2)
set.seed(2024)
N <- 2000
k <- 21
R <- 10000
B <- rbinom(k-1, 1, 0.3)
e1max <- e2max <- rep(NA, R)
for (i in 1:R){
  X1 <- matrix(rt(N*(k-1), 5), N, k-1)
  Xk1 <- X1 %*% B
  X1 <- cbind(X1, Xk1)
  M1 <- t(X1) %*% X1
  e1 <- eigen(M1)
  e1max[i] <- max(e1$values)
  
  X2 <- matrix(rt(N*(k-1), 6), N, k-1)
  Xk2 <- X2 %*% B
  X2 <- cbind(X2, Xk2)
  M2 <- t(X2) %*% X2
  e2 <- eigen(M2)
  e2max[i] <- max(e2$values)  
}
d1 <- data.frame(
  Eigenvalue = e1max,
  Distribution = "t5"
)
d2 <- data.frame(
  Eigenvalue = e2max,
  Distribution = "t6"
)
d <- rbind(d1, d2)
ggplot(d, aes(x = Eigenvalue, fill = Distribution)) +
  geom_density(alpha = 0.25)

(This took a few minutes to run, but the time can be decreased by decreasing the number of repetitions R.)

compare distributions of eigenvalues

If you want to say something about the distribution of maximum eigenvalues in your particular situation, your best bet might be to bootstrap the rows of $X$, multiply to give some kind of bootstrapped matrix $M_{\text{boot}}$, and calculate the maximum eigenvalue of that boostrapped matrix. I give an example below.

library(ggplot2)
set.seed(2024)
N <- 2000
k <- 21
R <- 10000
B <- rbinom(k-1, 1, 0.3)

X <- matrix(rt(N*(k-1), 5), N, k-1)
Xk <- X %*% B
X <- cbind(X, Xk)
emax <- rep(NA, R)
for (i in 1:R){
  
  idx <- sample(seq(1, N, 1), N, replace = T)
  X_boot <- X[idx, ]
  M_boot <- t(X_boot) %*% X_boot
  e <- eigen(M_boot)
  emax[i] <- max(e$values)
  
}
d <- data.frame(Eigenvalue = emax, Distribution = "Bootstrap")
ggplot(d, aes(x = Eigenvalue, fill = Distribution)) +
  geom_density(alpha = 0.25)

bootstrap distribution of maximum eigenvalues

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.