PCA and the train/test/validation split

Question

I have a binary classification problem with 1149 observations and 13,454 predictors. I want to apply the methodology described by cbeleites unhappy with SX in PCA and the train/test split.

In this context, I have two questions:

(i) If i do a grid search for the number of PCs to use, is it incorrect to test a number of PCs to use that is greater than the number of observations in the test set? It seems intuitive to me that the maximal number of PCs that should be tested in the grid search must be equal to or lower than the number of observations in the test set (supposedly to prevent a p>>n situation).

(ii) Is it more correct to use an hold-out set? i.e. first use 10-fold cross-validation to find the optimal number of PCs using 90% of the data as described by cbeleites unhappy with SX, then fit a new estimator using the optimal number of PCs using all the data that was used in the first step predict the classes' probability of the hold-out set?

Just to be more clear, my code looks something like this:

tests=[]
pps=[]
pcs=[]
skf=model_selection.StratifiedKFold(n_splits=10,random_state=61,shuffle=True)

for i in (2,5,10,25,50,100,114):

    tmp_pps=[]
    tmp_tests=[]

    for train_index, test_index in skf.split(X, y):
    
        estimator = SVC(probability=True)
        pca = PCA(i, svd_solver='arpack')
        scaler= StandardScaler()
    
        X_train, X_test = X[train_index], X[test_index]
        y_train, y_test = y[train_index], y[test_index]
    
        fScaler = scaler.fit(X_train)
        X_train_scaled = fScaler.transform(X_train)
        X_test_scaled = fScaler.transform(X_test)
    
        fpca = pca.fit(X_train_scaled)
        X_train_PCA = pca.transform(X_train_scaled)
        X_test_PCA = pca.transform(X_test_scaled)
    
        ft = estimator.fit(X_train_PCA, y_train)
        pp = estimator.predict_proba(X_test_PCA)[:, 1]
    
        tmp_tests.append(y_test)
        tmp_pps.append(pp)
    
    tests.append(tmp_tests)
    pps.append(tmp_pps)
    pcs.append(i)

for i in range(len(pcs)):
    pp = np.concatenate(res[i]).ravel()
    y_test = np.concatenate(tests[i]).ravel()
    score = roc_auc_score(y_test, pp)
    print(pcs[i],score)

Is this an incorrect/biased approach?

Answer 1

i) Yes, number of PCs should be smaller than or equal to the number of observations because the data matrix (assume mean normalised without loss of generality), $X_{n\times p}$, has rank $\leq \min(n,p)=n$, in this case. $X^TX$ will have rank $\leq n$ as well because $\text{rank}(AB)\leq \text{rank}(A)$.

ii) Since you have small number of observations, using cross-validation in choosing best number of PCs might be better. You need to find a criteria (e.g. cover 95% variance) and a proper voting scheme for fusing decisions coming from different folds. In the end, you can use your all training data to find the PCs. The test set should be separate from the beginning, i.e. you shouldn't use it even for selecting number of PCs.