0
$\begingroup$

My goal is to classify samples based on their dynamic time warping distances with k-nearest-neighbor classification.

Therefore I compute a nxn matrix, where n is the total number of samples. This matrix contains the distance from each sample to each sample.

The precomputed metric should be the right choice for the task since I do not want the KNeighborsClassifier to actually compute distances between rows, it should rather just take the distances out of the matrix.

I try to split this matrix in training and test data using k-fold-validation.

However, I get unrealistic poor classification results, and I do not really understand how the precomputed metric is supposed to be used when splitting data into training data and test data. What is wrong with the following code? Maybe the index splicing using iloc?

    X = compute_full_distance_matrix(input_data) # store distances in a pandas data frame
    Y = get_target_labels() # the target labels for each row

    kf = KFold(n_splits=5)
    kf.get_n_splits(X)

    for train_index, test_index in kf.split(X):

        X_train, X_test = X.iloc[train_index, train_index], X.iloc[test_index, train_index]
        Y_train, Y_test = Y[train_index], Y[test_index]

        model = KNeighborsClassifier(n_neighbors=3, metric='precomputed')
        model.fit(X_train, Y_train)
        predictions = model.predict(X_test)

        print(classification_report(Y_test,predictions))
Improve this question
$\endgroup$
6
  • $\begingroup$ Try to split your data first into test and train data and then compute the full distance matrix X which you feed into the KNeighbours function. I do not see any other issue $\endgroup$ – Nikolas Rieble Mar 13 '17 at 14:49
  • 1
    $\begingroup$ I am not sure how this is different to throwing out columns and rows during the folds. It would actually lead to multiple computations of the same distances? $\endgroup$ – Mike76 Mar 13 '17 at 15:08
  • $\begingroup$ Yes, it would. I only wonder whether it is different because you explicitly named the .iloc splitting as a potential source of error. Other than that it is simply possible that KNeighbors with 3 neighbours is not working fine. Try other parameters and other classifiers. Further you would want to optimize your parameter using crossvalidation and then evaluate with another set. $\endgroup$ – Nikolas Rieble Mar 13 '17 at 15:47
  • $\begingroup$ I know from a similar matlab implementation that at least 50% accuracy can be achieved by this method with the given training samples $\endgroup$ – Mike76 Mar 13 '17 at 20:36
  • $\begingroup$ 1) Did you compare whether the distance matrix is exactly the same as in matlab, 2) presumably the test train split is random and therefore not the same as in matlab --> not the same accuracy as a result $\endgroup$ – Nikolas Rieble Mar 13 '17 at 21:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.