How to plot ROC for knn (and potentially kernel spectral regression) I understand how to plot ROC for logistic classifier (like varies the probability cutoff). For KNN, how can I find the ROC? Also, what about kernel spectral regression?
 A: I am not quite sure about this myself, but I assume the ROC curve for a $k$ nearest neighbour classifier could be computed by means of a test set. 
Concretely, this means you would


*

*split your data in training and testing data

*train your classifier on the training

*increase threshold by adding an example from the testing data, one by one


The main reason I am not quite sure about this approach, is the dependency on the order of adding test samples. Assuming that you use the same order for different curves, I suppose they should be comparable, however.
A: In order to construct the ROC curve, you need to have the target variables and a ranking of your predictions from least likely to be the target class to most likely. ROC does not need to cut base don probabilities, but it does need some ranking of the data.
So for the logistic regression, this is the estimated probability, where you are changing the probability cutoffs to get the curve. For methods that do not produce probability, the ranking can be achieved based on other ways, for example, how close or far away from a decision boundary the specific data point is, where predictions closer to the boundary are considered less likely. In the case of a random forest, you can produce the ranking based on the proportion of trees that are predicting the target class.
If a classifier cannot produce a ranking, for example, because the implementation can just return categorical prediction, then ROC curve cannot be meaningfully constructed, it will just be a point. And of course, if a method is not predicting the target class whatsoever, e.g., if it's an unsupervised method, then ROC curve is not a meaningful concept.
For KNN, datapoints can be ranked for example by what proportion of the neighbors is of the target class, 10/10 is ranked higher than 6/10. But if you have a small k, the curve will not have many points. You can also try to calculate some average distance to create less discrete rankings and break ties, but maybe at this point, it might be better just to use some other method.
A: First note that a ROC curve is only defined for the two class case. Moreover, it is not based on classification decisions, but on probabilities assigned to each class, i.e. on $P(Y=i|\vec{x})$ for both classes $i\in\{1,2\}$, where $\vec{x}$ is the observed feature vector. The ROC curve summarizes the classification results as the decision threshold for the probability varies.
Fortunately, the kNN method also provides an estimate for this probability, which you can derive from Bayes' Theorem as follows:
$$P(Y=i|\vec{x})=\frac{p(\vec{x}|Y=i)\cdot P(Y=i)}{p(\vec{x})}$$
The denominator can be computed as a total probability:
$$p(\vec{x}) = \sum_i p(\vec{x}|Y=i)\cdot P(Y=i)$$
Inserting the class proportions $\hat{P}(Y=i)=n_i/n$ and the kNN estimator
$$\hat{p}(\vec{x}|Y=i) = \frac{k_i}{n_i V(\vec{x})}$$
into the above equations yields
$$\hat{P}(Y=i|\vec{x})=\frac{k_i}{k}$$
where $k_i$ is the number of training samples from class $i$ among the $k$ nearest neighbors.
Then you can enter the vector probs of these values for your test samples with true classes classes into your ROC generating function. The syntac for the R function pROC::roc, e.g., is
pROC::roc(classes, probs, levels=c("class1", "class2"))

