If I had a categorical response $Y$ and multiple categorical features $X$, and I wanted to fit a model to predict $Y$.
If all I cared about was the eventual distribution of $Y$ (say in terms of %), I was wondering if I could use that criteria somehow to evaluate my model, rather than use the actual predicted class label.
Originally I thought maybe, if I fitted some model e.g.
library(randomForest) y <- factor(sample(seq(3), 100, replace = TRUE)) x <- matrix(sample(seq(5), 500, replace = TRUE), ncol = 5) x <- data.frame(apply(x, 2, as.factor)) rf <- randomForest(y ~ ., data = x)
Then I could get the ratio like
test_probs <- predict(rf, x, type = "prob") test_ratio <- colSums(test_probs) test_ratio <- test_ratio / sum(test_ratio)
and e.g. calculate the MSE on it. But of course inside a CV loop. But even then, of course the distribution throughout the CV folds will always be very similar. Would this be usable if I set the number of folds so high that the variance is very high, or is it always a bad idea?