I have two groups of subjects, A, and B, each with a size of approximately 400, and about 300 predictors. My goal is to build a prediction model for a binary response variable. My customer wants to see the result of applying the model built from A on B. (In his book, "Regression Modelling Strategies", @FrankHarrell mentions that it's better to combine the two datasets and build a model on that, since doing so adds power and precision --- see page 90, External Validation. I tend to agree with him, considering that collecting the type of data that I have is very expensive and time consuming. But I don't have a choice about what the customer wants.) Many of my predictors are highly correlated and also very skewed. I'm using logistic regression to build my predictive model.
My predictors come mainly from mechanics. For example, total time the subject was under a stress higher than threshold $\alpha$ for time period $[t_1, t_2]$, for various values of $\alpha > 0$ and $0 \leq t_1 < t_2$. It is clear that just from their definitions, many of these total times are algebraically related to each other. Many of the predictors that are not algebraically related are related because of their nature: subjects that are under high stress during a time period $[t_1, t_2]$ tend to be under high stress during time period $[t_3,t_4]$, even if $[t_1,t_2] \cap [t_3,t_4] = \emptyset$. To reduce the dimension of the data, I clustered related predictors together (for instance, all the total stress times together) and used principal component analysis to represent each cluster. Since the variables were skewed, I tried two alternative paths:
- Before doing the PCA, I used a logarithmic transformation to reduce the skew in variables.
- I used Mia Hubert's ROBPCA algorithm, as implemented by the package rrcov in R, (PcaHubert), to find the robust principal components.
I am using the overall shape of the ROC curve, the shape of the precision-recall curve, and the area under the ROC curve (AUC) as my performance measures, and I'd like to get similar results for both datasets A and B. I was expecting to get a much better result from using the robust principal components, but to my surprise, the first method did better: better AUC value for both datasets A and B, more similarity between ROC curves, and more similar precision-recall curves.
What is the explanation for this? And how can I use robust principal components, instead of trying to make my data look like normal? Are there any particular robust PCA methods that you'd recommend instead of ROBPCA?