0
$\begingroup$

I'm starting on ML with the Iris dataset (with the errors corrected). I've built a typical test harness in Python.

X_train, X_validation, Y_train, Y_validation = train_test_split(X, Y, test_size=0.20, random_state=7)

models = []

models.append(('LR', LogisticRegression(solver='lbfgs', max_iter=1000)))
models.append(('LDA', LinearDiscriminantAnalysis()))
models.append(('KNN', KNeighborsClassifier()))
models.append(('CART', DecisionTreeClassifier()))
models.append(('NB', GaussianNB()))
models.append(('SVM', SVC()))

results = []

for name, model in models:
    kfold = KFold(n_splits=10, random_state=7, shuffle=True)
    cv = cross_val_score(model, X_train, Y_train, cv=kfold, scoring='accuracy')
    results.append((name, cv))

knn = KNeighborsClassifier()
knn.fit(X_train, Y_train)
predictions = knn.predict(X_validation)

I am using the following lib versions:

python: 3.8.5
scipy: 1.5.2
numpy: 1.19.2
matplotlib: 3.3.2
pandas: 1.1.3
sklearn: 0.23.2

All the various tutorials and samples seem to indicate that KNN is the best algorithm for this data set. When I run this with either the original dataset with the 2 incorrect rows, or the fixed one, KNN gets 3 prediction errors. Playing around with the K value, I was able to get it down to 2. Even when I remove the random_state=7, 2 prediction errors was the best I could get it to.

When I tried LDA, out of the box without any param tuning, I only get 1 prediction error.

Am I missing something? Basically, I'm asking why everything I see says KNN should be better, but I'm seeing LDA is better. Or is there something I need to do with KNN?

$\endgroup$
1
  • $\begingroup$ Have you tried different splits? Try doing repeated cross validation for KNN so you can see how the selecting the optimal number of neighbours from the training data changes the accuracy. The results from your experiment may be sensitive to the split you're using. $\endgroup$ May 10, 2021 at 0:46

1 Answer 1

0
$\begingroup$

I wouldn't treat online tutorials as a source of truth about the best practices. The point of the tutorials is to show how to do something. Usually, they do not offer a comprehensive, in-depth review of the available options, but rather show you how to use the available tools.

Are you sure that the author of the tutorial claimed that $k$-NN is the best of all the possible algorithms? It would be a ridiculous claim. First of all, did they check every other possible algorithm and every possible combination of hyperparameters? Likely not. But let's say that $k$-NN achieves perfect accuracy, this still does not prove that there is no other algorithm that can achieve this result.

Last but not least, the results would depend on many technical details: did you use the same versions of the packages as in the tutorial, exactly the same code, same procedures, hyperparameters, random seeds, etc? If not, results can differ because most of the machine learning algorithms are sensitive to such technical details. They can even differ between different implementations, for example, they can use different rules for tie-breaking, resulting in "the same" $k$-NN algorithm giving different results. All this is amplified by the fact that Iris is a small, toy dataset, so even small discrepancies may have a significant impact on the overall result.

Finally, there is a lot of bad and outdated tutorials. I cannot count the cases where the code I found in the online tutorials didn't work. If your results don't match the tutorial, double-check the code, but don't trust at face value something you've read on the Internet.

$\endgroup$
2
  • $\begingroup$ Yeah, definitely. I was working off the Jason Brownlee Intro book and there was some code that no longer worked. His book isn't really that good imho... its mostly a code dump with very little explanation as you said. $\endgroup$ May 10, 2021 at 13:03
  • $\begingroup$ The algo's I tried are above in the code sample. $\endgroup$ May 10, 2021 at 20:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.