I am using a non-parametric classification algorithm to provide probability estimates of class membership for a binary dependent variable (i.e. mutation present or not present) from 5 independent predictors. For example, if I measure the following 5 clinical factors [.1 1.3 2.1 .3 .2] and run the classifier I get an output vector of the probabilities of class membership like [.1 .9].
There are more than 130,000 observations so I applied a chi-square transform to each output vector using the expected distribution as calculated by the training data labels, for example [.05 .95].
My question is, do I need to correct for the unbalanced nature of the expected distribution? It seems to me that the chi-square equation inherently corrects for the unbalanced design by assigning a low value to a probability vector that does not differ much from the expected distribution. To say another way, a naive classifier that guessed a probability vector of [0 1] for all samples would have good accuracy before the chi-square transform, but would perform very poorly when I thresh-hold the chi-square values to the p-value I want.