# Estimating joint probability matrix

Motivation I have a classifier which for each input produces a discrete label $$C\in \{0, 1, ..., l\}$$. On the other hand, for each example I also have the true label $$Y\in \{0, 1, ..., l\}$$. I have $$n$$ (hold-out) data points, so I can get exchangeable observations $$\mathcal D = \{(C_1, Y_1), \dotsc, (C_n, Y_n)\}$$.

Now I want to estimate how good this classifier is, that is, I would like to estimate the confusion matrix $$M_{ij} = P(C=i, Y=j)$$.

Approach 1 The most straightforward approach is to make a histogram by counting the occurrences and dividing by $$n$$. This works if $$n$$ is large enough and produces a point estimate $$\hat M$$.

Approach 2 A more Bayesian approach would be to have a prior distribution $$p(M)$$ and assume that $$(C_i, Y_i)$$ are sampled from categorical distribution defined by $$M$$: $$(C_i, Y_i) | M \sim \text{Categorical}(M).$$ I like this approach as it gives a posterior distribution $$p(M| \mathcal D)$$ instead of a single value $$\hat M$$. However, I have to decide on a prior (in particular, I don't know which of them is least informative. A convenient choice would be to use a Dirichlet prior with more mass on the "diagonal" entries) and I am not entirely convinced whether this categorical distribution is the right model.

Hence, I would like to ask you about the proper way to do that. Making the question more precise:

1. Is this problem described in some books or articles? Any references would be very welcome.
2. If it is not, what is the most sensible approach to the problem?
3. Does it make sense to get a distribution using Approach 1 via bootstrapping (or Bayesian bootstrapping) $$\hat M$$?
4. What are other sensible variants of Approach 2? Are there other models I could try?