I am using two-layer cross-validation.

I want to compare two methods against each other, and in the inner loop, I want to tune the parameters.

I have tried

# K-fold cross validation
K = 10
CV = cross_validation.KFold(num_observations, K, shuffle=True)

# Tree complexity parameter - constraint on maximum depth
tree_complexity = np.arange(2, 21, 1) # tc = [2,3,...,20]

# K-Nearest max neighbors
num_max_neighbors = 40

# Euclidean distance
dist = 2

# Loop through each fold
for train_indices, test_indices in CV:

    # Extract training and test set for current CV fold
    X_train = X[train_indices]
    y_train = y[train_indices]
    X_test = X[test_indices]
    y_test = y[test_indices]

    # Decision tree
    for i, t in enumerate(tree_complexity):
        # Fit decision tree classifier, Gini split criterion, different pruning levels (=depths?)
        dtc = tree.DecisionTreeClassifier(criterion='gini', max_depth=t)
        dtc = dtc.fit(X_train, y_train)

    # K-nearest neighbors
    for num_neighbors in range(1, num_max_neighbors + 1):
        # Fit classifier
        knclassifier = KNeighborsClassifier(n_neighbors=num_neighbors, p=dist)
        knclassifier.fit(X_train, y_train)

But how do I

  1. determine which method was best (outer loop), and
  2. determine which parameters I should use (inner loop)
  • $\begingroup$ I think all you need to do is to compare cross-validation averages for your best performing decision tree and your best k-nn classifier. $\endgroup$ Nov 7, 2016 at 19:32
  • $\begingroup$ I think you are right, but I don't know how to choose which decision tree and k-nn classifier is are performing best $\endgroup$
    – Jamgreen
    Nov 7, 2016 at 20:14


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy