I am new to this so please bear with me. I have two models, an original one with many variables, and a new one where I extracted 5 factors. I used both for a cox regression, and the model in which I used the 5 factors as independent variables, performs significantly worse. (according to $R^2$. I am using r-commander). Why is this? I know sometimes that this model turns out better, which makes sense to me because we are kind of pooling together the original variables' power into an optimal number of new variables. Is my original model simply better..and the variability in the original data cannot be captured sufficiently with my new model?
$\begingroup$
$\endgroup$
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
4
I used both for a cox regression
Ok, before I start talking about R squared values, I'll ask you to check some stuff.
Are your sample sizes the same for both regressions?
I suspect that you may have lost some more participants in the factor analysis due to preprocessing while trying to consider censoring. Thats the first thing I'd check.
I have two models, an original one with many variables, and a new one where I extracted 5 factors...the 5 factors ....performs significantly worse.
The R squared value for Cox isn't the same as for OLS regression, but for simplicity I'm going to write as if it is. Cox regressions usually use a likelihood ratio statistic (LRT) based R squared calculation, so keep that in mind.
Here's the takeaway:
You can only lose information when you reduce your variables with factor analysis or PCA.
Try and figure out how much variance you're explaining with the first 5 factors. It's possible that you're only capturing ~65% of your original variance.
If that's the case, then you can simply extract more factors until you're closer to ~80% or ~90%. Or simply until your R squared values looks better.
So, why use factor analysis?
Variable reduction can help you to reduce multicollinearity, model complexity (via degrees of freedom) and even processing power.
However, that probably won't help your R squared values. Remember that R squared is a measurement of how much variance you're explaining in your y (dependent) variable. When you include 100% of your variables, you (usually) have the best chance of explaining that y-variable (ie: have the highest R squared).
If you're reducing your variables and extracting factors that only capture some of that variance, then you're essentially reducing the amount of variance you could explain.
Here's an illustration of what I mean:
100% of x-variables --> explains 35% of variance in y.
5 factors w/ 65% of original variance --> explains 25% of variance in y.
-
$\begingroup$ Reid, thank you for answering. Now let me answer your questions. $\endgroup$– didiCommented Mar 15, 2016 at 20:30
-
$\begingroup$ Yes the sample sizes are the same. What does "preprocessing" mean? You are right I should be looking at the LRT. With my 5 "components" I am capturing around 75% of the variability but the thing is I have tried to add more factors, but it's futile. The R-squared does not increase, in fact, it gets worse. (I was also conducting this in SPSS and there is an option there that allows you to choose the amount of factors you desire, or let SPSS choose it for you based on Eigenvalues--turns out 5 is my optimal). Now out of curiosity, I moved away from Cox regression and tried a logistic,and a linear- $\endgroup$– didiCommented Mar 15, 2016 at 20:31
-
$\begingroup$ model to see if my "new" model (with the factors as predictors) would perform worse. it doesn't! I'm looking at R-squared adjusted now instead of the raw R-squared, because I think it's more appropriate, since the original model has 26 variables, and this 5, and the r-squared is better here. The p-value is also way smaller. could the problem be that I just shouldn't have been hailing R-squared for the cox regression model... $\endgroup$– didiCommented Mar 15, 2016 at 20:40
-
$\begingroup$ Hey didi! Hope I can be of some help! Preprocessing is basically what you do in order to put all your variables in the proper format and that you've removed outliers (etc). If you're unfamiliar with the process, then maybe some of your error is from some preprocessing error. Could you tell me a little more about your project? Are you examining time-to-event data? How are you integrating that into the factor analysis? Maybe something weird is happening there. $\endgroup$– ReidCommented Mar 18, 2016 at 17:23