Confidence metric for a single prediction in multiclass classification? Are there any particular methods (Naive Bayes, SVM, RF, etc) that can provide some estimate of 'confidence' along with the class prediction for a test feature set in which the true class is unobserved?  If so, can someone point me towards some readings to learn more about them.
 A: Here are links to to some "essential reading" for any aspiring data scientist. Both are free e-books provided courtesy of some top Stanford statisticians.
Introduction to Statistical Learning (the basic primer) and 
Elements of Statistical Learning (more advanced)
In particular, please read Chaps 2,4,5 in Introduction to Statistical Learning and Chaps 1,2,4,7 from "Elements". They cover the classification context and assessing how reliable a classifier will be using cross validation/resampling methods.

To get an estimate of "out of sample" classifier accuracy, you first need to run cross validation on your classifier. For each fold of the cross validation, you will generate an "out of fold" confusion matrix using the classifier trained on the "in fold" data. 
You will then sum the matrices and normalize them to percentages. These will give you an estimate of the "out of sample" performance of your classifier. 
So, lets say you give your trained classifier a new observation and it says it should go in class $K$. You can estimate how confident you can be that it really is $K$ by looking at the on diagonal value in the out of sample confusion matrix. For example, if 95% of the time, a classification to $K$ will be $K$, then you are approximately 95% confident in this prediction, assuming your new sample is from the same population as your sample (the basic assumption of any learning method).
As an aside, note that the confusion matrix is an estimate of a joint distribution $P_{\hat C,C}(\hat C(\omega), C(\omega))$, where $\hat C$ is the classifier output, and $C$ is the actual class for an observation $\omega$ from some sample space $\Omega$ with an underlying probability measure $P_{\Omega}(\omega)$ on the population being classified. This construction can explain how a classifier that has a high average classification accuracy can have dismal performance for one or more particular classes...basically $P_{\Omega}$ is concentrated around only a few classes, so even "dumb"classifier can be highly accurate in some cases.
