Accuracy assessment for multinomial probabilities

Question

I am working on a classification problem where the input data are multinational proportions for each point, with $m$>2 groups. And the outcome is likewise a multinomial proportion for that point. For example, if we have $m=4$ groups, then for point $i$, the observed proportions would be something like:

$y_i = (0.1, 0.2, 0.3, 0.4)'$

The result of the model is a new set of predicted proportions for point $i$:

$\hat{y}_i = (0.2, 0.1, 0.4, 0.3)'$

Is there a standard way to calculate the accuracy of this output across all $n$ observations? It's not really a classification problem, since I'm not doing strict classification. It also reminds me of various kappa statistics in land cover classification, but again, I don't really have any strict classification. It's really about comparing how close one vector of proportions is to another. I can come up with a bunch of ad hoc ways to calculate the average error across my $n$ training points, but this seems like a standard question, and I don't want to reinvent the wheel.

EDIT: The best I've found so far is the Bhattacharyya coefficient which is used for various image analysis feature selection and accuracy assessment. It's not super common otherwise, so I'm still hoping to find a better options.

Answer 1

A simplification of this problem is using the features to predict one proportion, and a typical suggestion is beta regression. In your case, you have four proportions to predict, so four beta regressions might make sense, but you know the four marginal distributions to be related in that their sum is always $1$.

When a multivariate distribution has marginal beta distributions and a relationship between the margins where they always add up to $1$, that is a Dirichlet distribution. Yes, it is an assumption, perhaps a bad or even terrible one, that the margins are beta-distributed. However, if you are willing to make such an assumption, then the relationship between the components means that the multivariate distribution is Dirichlet.

Consequently, you might be interested in calculating the Dirichlet likelihood of your prediction, analogous to how the sum of squared residuals is related to the Gaussian likelihood and the crossentropy loss is related to the binomial likelihood (so such an approach is completely common in machine learning). To get something analogous to $R^2$ in linear regression that might be easier to interpret, you might consider comparing to the Dirichlet likelihood of a naïve model as I discuss in that link for the usual $R^2$ (but the idea is the same to compare model performance to some kind of “must beat” level of performance). Then an $R^2$ analogue would be $1-\dfrac{LL(D)}{LL(D_0)}$, where $LL(D)$ is the Dirichlet log-likelihood for your predictions and $LL(D_0)$ is the Dirichlet log-likelihood for the “must beat” model (a “null” model, to explain the subscript).

I confess my lack of confidence in deciding what “must beat” performance would be for a Dirichlet distribution, and I am not so sure that assuming beta margins is reasonable, but I hope this can spark some ideas.