0
$\begingroup$

I have P predictors, C classes

After performing training on a training partition I run the test partition through my classifier.

Say for a particular test instance the predicted class is Cpred with probability Ppred while the original label is Corig and the probability of it being that class is computed to be Porig by the trained classifier.

What type of inferences, insights, uses can I draw from (Ppred - Porig)?

Some specific questions?

  1. Can I use cases where Ppred - Porig >> mean(Ppred - Porig | Corig, Cpred) to look for mislabelled data?
  2. Can I use case where Ppred - Porig >> mean(Ppred - Porig | Corig, Cpred) to further partition the Classes into a new Class Corig,pred -- perhaps the original labels are not expressive enough for the underlying phenomenon?
  3. How could one interpret Pi,pred - Pi,orig == Pj,pred - Pj,orig where i and j are two instances in the test set but where Pi,pred >> Pj,pred

Notes:

  1. How is Porig calculated? Both Porig and Ppred are produced by the classifier. If there are C classes -- the Classifier will output P1, ... , Pc probabilities, one for each class. The predicted class will correspond to the argmax of P1 ... Pc
  2. OK, so how is Porig different from Ppred? Ppred is the max(P1, ... Pc) while Porig is P of the labeled class that we know apriori
  3. See R code below for a simple illustration using the R

Example:

set.seed(1)
str(iris)
train<-sample(2,nrow(iris),replace=TRUE)
rf<-randomForest(Species~.,data=iris,subset=train==1) 
p.resp<-predict(rf,newdata=iris[train==2,],type="prob")
pred<-predict(rf,newdata=iris[train==2,])
table(iris[train==2,]$Species,pred)
virginica_classifiedas_versicolor<-pred=="versicolor" & iris[train==2,]$Species=="virginica"
Porig<-p.resp[virginica_classifiedas_versicolor,"virginica"]
Ppred<-p.resp[virginica_classifiedas_versicolor,"versicolor"]
$\endgroup$
  • 1
    $\begingroup$ "the original label is Corig and the probability of it being that class is computed to be Porig" - How do you compute this probability? $\endgroup$ – Arun Jose May 4 '17 at 4:45
  • $\begingroup$ @ArunJose -- as computed by the classifier in a multinomial case -- let me update the question $\endgroup$ – user1172468 May 4 '17 at 5:02
  • 1
    $\begingroup$ In that case how is Ppred different from Porig? If both are probabilities from the same classifier? $\endgroup$ – Arun Jose May 4 '17 at 5:04
  • $\begingroup$ @ArunJose -- I also updated the question with note 2 to make that distinction $\endgroup$ – user1172468 May 4 '17 at 5:07
  • 1
    $\begingroup$ "Porig is P of the labeled class that we know apriori" - This is still confusing. Are you suggesting (sum of class A/ total N classes) = Porig? $\endgroup$ – Arun Jose May 4 '17 at 5:10
0
$\begingroup$

So after looking at the edits, here's a small clue as to why Porig should not have anything to do with Ppred from the test set.

Porig, is simply the probability of occurrence of a particular record in a given dataset. Some datasets may have more of a particular class, some may have less. This says nothing about why they occur.

Ppred on the other hand is a conditional probability. For example, when predictor 1 takes a value X and predictor 2 takes a value y, in my training data, I see class A 7 times out of 10. (I'm not actually worried about how often class A occurs in isolation)

So Ppred will be exactly the same for any given unique record, irrespective of which dataset it belongs to. If it occurs once or ten times or hundred times. In each of these cases, Porig will be different, but Ppred will be the same.

$\endgroup$
  • $\begingroup$ mmm Arun I think I need to clarify my question ... $\endgroup$ – user1172468 May 4 '17 at 6:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.