0
$\begingroup$

Motivation I have a classifier which for each input produces a discrete label $C\in \{0, 1, ..., l\}$. On the other hand, for each example I also have the true label $Y\in \{0, 1, ..., l\}$. I have $n$ (hold-out) data points, so I can get exchangeable observations $\mathcal D = \{(C_1, Y_1), \dotsc, (C_n, Y_n)\}$.

Now I want to estimate how good this classifier is, that is, I would like to estimate the confusion matrix $M_{ij} = P(C=i, Y=j)$.

Approach 1 The most straightforward approach is to make a histogram by counting the occurrences and dividing by $n$. This works if $n$ is large enough and produces a point estimate $\hat M$.

Approach 2 A more Bayesian approach would be to have a prior distribution $p(M)$ and assume that $(C_i, Y_i)$ are sampled from categorical distribution defined by $M$: $$(C_i, Y_i) | M \sim \text{Categorical}(M).$$ I like this approach as it gives a posterior distribution $p(M| \mathcal D)$ instead of a single value $\hat M$. However, I have to decide on a prior (in particular, I don't know which of them is least informative. A convenient choice would be to use a Dirichlet prior with more mass on the "diagonal" entries) and I am not entirely convinced whether this categorical distribution is the right model.

Hence, I would like to ask you about the proper way to do that. Making the question more precise:

  1. Is this problem described in some books or articles? Any references would be very welcome.
  2. If it is not, what is the most sensible approach to the problem?
  3. Does it make sense to get a distribution using Approach 1 via bootstrapping (or Bayesian bootstrapping) $\hat M$?
  4. What are other sensible variants of Approach 2? Are there other models I could try?
$\endgroup$

1 Answer 1

0
$\begingroup$

A natural and simple approach is to use the Dirichlet distribution as a prior for your categorical distribution.

If the probabilities in each bin are you small you can also assume that the counts are Poissonian and estimate them independently.

Sampling based methods seem to me highly unnecessary in this case, as you can easly calculate an exact posterior distribution.

$\endgroup$
2
  • $\begingroup$ Thank you for answering! Indeed, a Dirichlet prior would be very convenient (I forgot to mention it. Edited the post :)) – I'm rather unsure what is the right uninformative prior in this case. Speaking of sampling, I agree – in Approach 2 it's not needed. However, the model may be misspecified, so I'd like to know other sensible approaches. $\endgroup$ Feb 4 at 19:21
  • $\begingroup$ A Dirichlet with all the parameters set to 1 is equivalent to a uniform prior. I don't think you can get much wrong with it $\endgroup$
    – J. Delaney
    Feb 4 at 19:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.