# Evaluate classifier based on predicted probabilities?

If I had a categorical response $$Y$$ and multiple categorical features $$X$$, and I wanted to fit a model to predict $$Y$$.

If all I cared about was the eventual distribution of $$Y$$ (say in terms of %), I was wondering if I could use that criteria somehow to evaluate my model, rather than use the actual predicted class label.

Originally I thought maybe, if I fitted some model e.g.

library(randomForest)
y <- factor(sample(seq(3), 100, replace = TRUE))
x <- matrix(sample(seq(5), 500, replace = TRUE), ncol = 5)
x <- data.frame(apply(x, 2, as.factor))

rf <- randomForest(y ~ ., data = x)


Then I could get the ratio like

  test_probs <- predict(rf, x, type = "prob")
test_ratio <- colSums(test_probs)
test_ratio <- test_ratio / sum(test_ratio)


and e.g. calculate the MSE on it. But of course inside a CV loop. But even then, of course the distribution throughout the CV folds will always be very similar. Would this be usable if I set the number of folds so high that the variance is very high, or is it always a bad idea?