I am using two-layer cross-validation.
I want to compare two methods against each other, and in the inner loop, I want to tune the parameters.
I have tried
# K-fold cross validation K = 10 CV = cross_validation.KFold(num_observations, K, shuffle=True) # Tree complexity parameter - constraint on maximum depth tree_complexity = np.arange(2, 21, 1) # tc = [2,3,...,20] # K-Nearest max neighbors num_max_neighbors = 40 # Euclidean distance dist = 2 # Loop through each fold for train_indices, test_indices in CV: # Extract training and test set for current CV fold X_train = X[train_indices] y_train = y[train_indices] X_test = X[test_indices] y_test = y[test_indices] # Decision tree for i, t in enumerate(tree_complexity): # Fit decision tree classifier, Gini split criterion, different pruning levels (=depths?) dtc = tree.DecisionTreeClassifier(criterion='gini', max_depth=t) dtc = dtc.fit(X_train, y_train) # K-nearest neighbors for num_neighbors in range(1, num_max_neighbors + 1): # Fit classifier knclassifier = KNeighborsClassifier(n_neighbors=num_neighbors, p=dist) knclassifier.fit(X_train, y_train)
But how do I
- determine which method was best (outer loop), and
- determine which parameters I should use (inner loop)