1
$\begingroup$

Let's say I use importance sampling to sample conditional probability distributions for variable X (categorical variable with 3 levels) 50 times, for example,

P(X|Y=1) = { (0.11, 0.21, 0.68), (0.09, 0.22, 0.69), ... }

P(X|Y=0) = { (0.20, 0.25, 0.55), (0.15, 0.23, 0.62), ... }

What sort of a test should I use to test for significant differences between the sampled categorical distributions between P(X|Y=1) and P(X|Y=0)?

EDIT: I can think of a t-test to look for differences between individual levels, e.g. between P(X=1|Y=1) and P(X=1|Y=0), i.e. {0.11, 0.09...} and {0.20, 0.15...} and so on. I think that would be a valid way of comparing means of sampled probabilities.

Is there a way of comparing the categorical distributions themselves? I am not sure how to do this.

Improve this question
$\endgroup$
3
  • $\begingroup$ At first, what’s the purpose of the test? You may consider using MLE or simple tests for difference in means. $\endgroup$
    – Daria
    Commented Jul 15, 2019 at 20:35
  • $\begingroup$ I do not understand the question as it mixes the evaluation of the difference between two distributions, $P_0$ and $P_1$, and the Monte Carlo variability due to an importance sampling, $\hat P_0$ and $\hat P_1$, $\endgroup$
    – Xi'an
    Commented Jul 16, 2019 at 11:26
  • $\begingroup$ @Xi'an So I'm guessing a t-test would be appropriate for testing whether the difference in means between the estimated probability distributions is significant, which would account for the variability in sampling. I guess I'm looking for guidance on how I can check whether the sampled categorical distributions are significantly different, not the individual probabilities for each category. $\endgroup$
    – elbord77
    Commented Jul 16, 2019 at 15:12

1 Answer 1

Reset to default
1
$\begingroup$

This is what the $\chi^2$ test for homogeneity is designed for. For some reason I could not find it in the English language wikipedia, but the German language wikipedia has an article about it (see section 4). Maybe someone can provide an English language reference in the comments.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.