2
$\begingroup$

I have a binary classification problem with 1149 observations and 13,454 predictors. I want to apply the methodology described by cbeleites unhappy with SX in PCA and the train/test split.

In this context, I have two questions:

(i) If i do a grid search for the number of PCs to use, is it incorrect to test a number of PCs to use that is greater than the number of observations in the test set? It seems intuitive to me that the maximal number of PCs that should be tested in the grid search must be equal to or lower than the number of observations in the test set (supposedly to prevent a p>>n situation).

(ii) Is it more correct to use an hold-out set? i.e. first use 10-fold cross-validation to find the optimal number of PCs using 90% of the data as described by cbeleites unhappy with SX, then fit a new estimator using the optimal number of PCs using all the data that was used in the first step predict the classes' probability of the hold-out set?

EDIT Just to be more clear, my code looks something like this:

tests=[]
pps=[]
pcs=[]
skf=model_selection.StratifiedKFold(n_splits=10,random_state=61,shuffle=True)

for i in (2,5,10,25,50,100,114):

    tmp_pps=[]
    tmp_tests=[]

    for train_index, test_index in skf.split(X, y):
    
        estimator = SVC(probability=True)
        pca = PCA(i, svd_solver='arpack')
        scaler= StandardScaler()
    
        X_train, X_test = X[train_index], X[test_index]
        y_train, y_test = y[train_index], y[test_index]
    
        fScaler = scaler.fit(X_train)
        X_train_scaled = fScaler.transform(X_train)
        X_test_scaled = fScaler.transform(X_test)
    
        fpca = pca.fit(X_train_scaled)
        X_train_PCA = pca.transform(X_train_scaled)
        X_test_PCA = pca.transform(X_test_scaled)
    
        ft = estimator.fit(X_train_PCA, y_train)
        pp = estimator.predict_proba(X_test_PCA)[:, 1]
    
        tmp_tests.append(y_test)
        tmp_pps.append(pp)
    
    tests.append(tmp_tests)
    pps.append(tmp_pps)
    pcs.append(i)

for i in range(len(pcs)):
    pp = np.concatenate(res[i]).ravel()
    y_test = np.concatenate(tests[i]).ravel()
    score = roc_auc_score(y_test, pp)
    print(pcs[i],score)

Is this an incorrect/biased approach?

$\endgroup$
0
$\begingroup$

i) Yes, number of PCs should be smaller than or equal to the number of observations because the data matrix (assume mean normalised without loss of generality), $X_{n\times p}$, has rank $\leq \min(n,p)=n$, in this case. $X^TX$ will have rank $\leq n$ as well because $\text{rank}(AB)\leq \text{rank}(A)$.

ii) Since you have small number of observations, using cross-validation in choosing best number of PCs might be better. You need to find a criteria (e.g. cover 95% variance) and a proper voting scheme for fusing decisions coming from different folds. In the end, you can use your all training data to find the PCs. The test set should be separate from the beginning, i.e. you shouldn't use it even for selecting number of PCs.

$\endgroup$
2
  • $\begingroup$ "The test set should be separate from the beginning, i.e. you shouldn't use it even for selecting number of PCs." Is a single test set sufficient for a reliable estimation of the procedure? I am not sure if it is relevant, by my data contain only 66 positive cases. $\endgroup$
    – NatNag
    Aug 18 '20 at 20:07
  • 1
    $\begingroup$ In small datasets, instead of using a single test set, cross validation should be used. I think you should use it as well because the step after PCA will contain only 10-15 positive cases in a single test set. $\endgroup$
    – gunes
    Aug 19 '20 at 3:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.