1
$\begingroup$

I am developing sampling procedures for different kinds of discrete objects (rankings, sets, etc.). In most cases I know the theoretical distribution of the objects I'm sampling. My idea is thus to validate my samplers by using some statistical tests.

The most obvious idea would be to use a chi-square test as I'm dealing with discrete categories. However, I am not sure it makes sense to use it since I have synthetic data. The problem being that my sample is arbitrary large, thus the p-value would always be close to 1 even with clearly wrong samplers (see the image below).

enter image description here

I do not really know what the best procedure would be. Any help would be appreciated :)

Simon.

Improve this question
$\endgroup$
4
  • $\begingroup$ Hello Simon. Can you elaborate on what you mean by the word "validate"? Why do you need to validate? $\endgroup$
    – J-J-J
    Commented Dec 5, 2023 at 15:45
  • $\begingroup$ Well my problem is to make sure that my code does what it should. I woud wand a formal way of saying "there is no reason to believe that a sample generated by your sampler is too far off compared to the theoretical distribution". An example would be to have an urn sampler, you could code it in many ways, not all of them would yield a distribution over outcomes that is "correct". So if you were to test the adequation between the observed distribution and the theoretical one, you say that it does not look wrong. I hope this helps? $\endgroup$
    – Siolan
    Commented Dec 5, 2023 at 16:59
  • $\begingroup$ you might not need a test. If you can draw a very large number of observations from your generator, and see that its result is completely discrepant from what you'd expect (as in the plot you show), no need for a test to see there's a problem. Another thing to consider, that won't answer your question but could save you a lot of time: have you checked if your generator has not already been implemented in the programming language you use? For instance scipy has a lot of generators for various discrete distributions, see docs.scipy.org/doc/scipy/reference/stats.html . $\endgroup$
    – J-J-J
    Commented Dec 7, 2023 at 7:13
  • $\begingroup$ Yes I agree with the fact that it may not be needed. I would want something more scientific that "it looks like a good fit". About not re-implementing, yes I am actually building up on the scipy things. Thanks for the reference :) $\endgroup$
    – Siolan
    Commented Dec 7, 2023 at 11:06

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.