I have a binary classification problem with 1149 observations and 13,454 predictors. I want to apply the methodology described by cbeleites unhappy with SX in PCA and the train/test split.
In this context, I have two questions:
(i) If i do a grid search for the number of PCs to use, is it incorrect to test a number of PCs to use that is greater than the number of observations in the test set? It seems intuitive to me that the maximal number of PCs that should be tested in the grid search must be equal to or lower than the number of observations in the test set (supposedly to prevent a p>>n situation).
(ii) Is it more correct to use an hold-out set? i.e. first use 10-fold cross-validation to find the optimal number of PCs using 90% of the data as described by cbeleites unhappy with SX, then fit a new estimator using the optimal number of PCs using all the data that was used in the first step predict the classes' probability of the hold-out set?
EDIT Just to be more clear, my code looks something like this:
tests= pps= pcs= skf=model_selection.StratifiedKFold(n_splits=10,random_state=61,shuffle=True) for i in (2,5,10,25,50,100,114): tmp_pps= tmp_tests= for train_index, test_index in skf.split(X, y): estimator = SVC(probability=True) pca = PCA(i, svd_solver='arpack') scaler= StandardScaler() X_train, X_test = X[train_index], X[test_index] y_train, y_test = y[train_index], y[test_index] fScaler = scaler.fit(X_train) X_train_scaled = fScaler.transform(X_train) X_test_scaled = fScaler.transform(X_test) fpca = pca.fit(X_train_scaled) X_train_PCA = pca.transform(X_train_scaled) X_test_PCA = pca.transform(X_test_scaled) ft = estimator.fit(X_train_PCA, y_train) pp = estimator.predict_proba(X_test_PCA)[:, 1] tmp_tests.append(y_test) tmp_pps.append(pp) tests.append(tmp_tests) pps.append(tmp_pps) pcs.append(i) for i in range(len(pcs)): pp = np.concatenate(res[i]).ravel() y_test = np.concatenate(tests[i]).ravel() score = roc_auc_score(y_test, pp) print(pcs[i],score)
Is this an incorrect/biased approach?