# Can regression to the mean cause this?

Let's say I have two independent samples (DS and VS). My data is also highly multivariate (e.g. one dependent variable, 300+ potential predictors, N<300). To reduce the dimensionality of this problem I decided to aggregate the effect of my 300+ potential predictors into one index parameter. Therefore, I regressed each predictor on my dependent variable, which provides me with an effect score (beta value) for each predictor. By multiplying these beta values with the predictors, I get the accumulated net effect (risk index) of all 300+ predictors for a particular observation.

Finally, I tested if this risk index is associated with my dependent variable in the Validation Sample (VS). I expected that the beta value for the risk index would fall between 0 (no association) and positive values (predictive power). However, to my surprise I consistently got negative beta values for some sets of predictor variables, especially those that I didn't actually expect to be good predictors.

I'm wondering if regression to the mean might be a sufficient explanation for this?

$$c = \rho(\hat{Y}, Y) \times \sqrt{\frac{\mbox{var}(Y)}{\mbox{var}(\hat{Y})}}.$$
If this concordance is negative in a test sample, it's probably a totally spurious result. I doubt that over simulated resamples using truly independent identically distributed samples you'd find that $Pr(c < 0) = 1$.
Your method in general seems a little bizarre and seems highly prone to overfitting, unless your sample is of size $n = 20 \times p$ or approx. 6000 or bigger. There are other ways of pooling a myriad of predictors of potential interest, such as LASSO or PLS which sound more relevant to your application. In psychometry, for instance, these specialized regression techniques are suited to create "scores" for behavioral diagnoses using large survey tools.
• Quite the opposite in fact. Suppose you have two almost perfectly negatively correlated factors in a sample, such as $E[Y|X,W] = X - W$ and $E[X] = -W$ approximately. Regressing $Y$ upon $X$ and $W$ will give a near perfect fit but $\hat{Y}$ is nearly 0 for all predicted values. This predicted value has low concordance. – AdamO Jun 12 '13 at 19:36
• It would be far better to use an approach based on good statistical principles. lasso and elastic net are two worth trying, or just use ordinary quadratic penalization and keep all 300 variables in the model. Note that your problem is univariate as you have one $Y$ variable. It is a multivariable regression problem. Also note that if you don't use shrinkage, the only kind of dimensionality reduction that is usually reasonable is an approach masked to $Y$, e.g. variable clustering, redundancy analysis, nonlinear principal components. – Frank Harrell Aug 11 '13 at 20:36