I am working with a given factor model of the form $$y=B x +\epsilon$$ where $x$ is a random vector in $R^M$, $B$ is an $N\times M$ matrix of factor loadings, $y$ is a random vector in $R^N$ (with $N \gg M$) and $\epsilon$ is a vector of mean-zero innovations, independent of $x$. For simplicity, the innovations are normally distributed and pairwise independent. My question is relatively simple to state: How can I test whether a factor is significant, in the same sense in which a predictor is significant in linear regression?
Clarification: I should perhaps have emphasized that I am working with a given model, and I have to assess the predictive value of each factor of the model. In other terms, is there a simple way to assess whether dropping a factor from a model would result in a loss of predictive power, without comparing two different models with and without the factor? In the latter case, there is a great deal of literature on model selection, based on ML, GMM etc. (The rotational invariance of factor models doesn't play an essential role, btw).