I'm reading Introduction to Statistical Learning. It mentions on page 166 for KNN modeling that a success rate of 11.7% is more than double that of random guessing. My question is, first what is the success rate of random guessing and why is the success rate for KNN so low and yet better than other success rates like that of logistic regression which is around 50%?
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
13
Your question is difficult to understand without the context. It is about using the Caravan data set to predict purchasers of caravan insurance
What the book says is that using KNN with $K=1$ gives the following results
> table(knn.pred,test.Y)
test.Y
knn.pred No Yes
No 873 50
Yes 68 9
which means that, after basing the model on $4882$ individuals in the training set and applying that model to the $1000$ other individuals in the test set, it predicts $68+9=77$ people will purchase insurance, and it gets $9$ of these correct, a proportion $\frac9{77}=11.7\%$ correct
If it had been guessing insurance purchasers at random, it would have got an expected proportion $\frac{59}{1000}=5.9\%$ of those predicted to buy correct, about half the KNN figure. So it says KNN did better than random guessing
It goes on to say using KNN with $K=3$ would have led to $\frac{5}{26}=19.2\%$ while using $K=5$ would have led to $\frac{4}{15}=26.7\%$ correct
It does not say that with the same problem logistic regression would have a success rate of around $50\%$. Rather the opposite:
As a comparison, we can also fit a logistic regression model to the data. If we use $0.5$ as the predicted probability cut-off for the classifier, then we have a problem: only seven of the test observations are predicted to purchase insurance. Even worse, we are wrong about all of these!
However, we are not required to use a cut-off of $0.5$. If we instead predict a purchase any time the predicted probability of purchase exceeds $0.25$, we get much better results: we predict that $33$ people will purchase insurance, and we are correct for about $33\%$ of these people. This is over five times better than random guessing!
The moral of the story illustrated by this part of the book is that some questions are difficult, but with suitable methods and good parameters, some machine learning can still produce results which are substantially better than random guessing
-
$\begingroup$ Thank you very much, Henry! Your answer was detailed and helpful. $\endgroup$– watchyCommented Aug 28, 2017 at 0:39
-
$\begingroup$ Hi Henry, I have one more question for you. How exactly do you figure 59 for random guessing? I assume it's from the confusion matrix (50 + 9)... $\endgroup$– watchyCommented Aug 28, 2017 at 9:43
-
$\begingroup$ @user2342434242 Yes. $50+9$ is the number of actual insurance purchases in the $1000$ element test set $\endgroup$– HenryCommented Aug 28, 2017 at 10:33
-
$\begingroup$ Hey Henry, still a little confused... why choose 50, isn't that a false-positive? And by the same token, why choose 68? $\endgroup$– watchyCommented Aug 28, 2017 at 10:52
-
$\begingroup$ @user2342434242: $77$ is the number predicted positive for the test set using KNN and $K=1$: of these $9$ were true positives and $68$ were false positives, giving a precision of $\frac{9}{77}$. Meanwhile $59$ was the number of actual cases: of these $9$ were predicted positive, so true positives, and $50$ were predicted negative so false negatives, giving a sensitivity of $\frac{9}{59}$. There were $873$ true negatives $\endgroup$– HenryCommented Aug 28, 2017 at 11:12