0
$\begingroup$

I am asked to check if a categorical distribution with $3$ variables is uniform, which means each variable has $\frac{1}{3}$ probability in the real population. (Required significance level: $0.01$)

Lets say I have a dataset sample of the real population with $1000$ people, and I have a column in my dataframe that represents the monetary status of a person with three categorical variables (poor, moderate, rich).

My Work:
Null Hypothesis: The sample distribution is uniform.
Alternate hypothesis: The sample distribution is not uniform.

Test statistic: The Total Variation distance between the distribution of the sample and the uniform distribution.
In other words, using this formula for TVD: $\frac{|\sum_{i=1}^3p_i-q_i|}{2}$ (where $p_i$, and $q_i$ are probabilities of each categorical variable in each of the two samples).
Here I have $\frac{|\sum_{i=1}^3p_i-\frac{1}{3}|}{2}$. (Because the other sample has uniform distribution, or the Model assuming the null hypothesis is true).

Now, I started sampling from my Model dataset assuming the null hypothesis is true (uniform), decided to take sample size of $500$ (didn't really think of it too much).
For each of those samples, I calculated the TVD from the uniform distribution as described above.

Then I plotted the empirical distribution of the TVDs (the test statistic), which was between $0.00$ to $0.09$ (not exactly but close enough).

Drew a red dot on my graph of the TVD from my dataset and it was at $0.38$.
Calculated my p-value, and of course, none of the samples TVD's were even close to $0.38$, so I got p-value=0 exactly. And based on that I rejected the null hypothesis, and said that the distribution of the monetary status of a person in the population is most likely Not uniform.

Questions:

  1. Is getting a p-value=$0$ exactly weird? should that make me worry that what I did was wrong? (because that's exactly why I'm here).
  2. In the end, I wrote most likely not uniform in my conclusion, is that a formal way to write my results? if not, how would I formalize it more?
  3. Does the idea of what I did and the steps seem logical and alright? (Because alot of my friends didn't reject the null hypothesis, so I'm hesitating about my answer).
  4. When I checked the probabilities in my dataset I got $(0.713,0.179, 0.108)$, that makes me calm down a little, because it seems far away from $(\frac{1}{3},\frac{1}{3},\frac{1}{3})$, am I supposed to feel like that? or it doesn't matter since we're requiring significance level of $0.01$ so it might still be true?

I would appreciate any help or feedback, sorry for making this too long, just wanted to clear everything I did.
Thanks in advance to everyone!

$\endgroup$
9
  • 1
    $\begingroup$ So your dataset has 1000 observations, the "real population" is much bigger? I understand (but am not sure) that you want to test your null hypothesis on the dataset of size 1000. I do not understand why you then draw subsamples of size 500. What do you want to achieve by doing that? I also don't understand how you computed your p-value. $\endgroup$ Commented Jun 15, 2021 at 12:52
  • $\begingroup$ @Lewian First of all thanks alot for your time, to answer the first question, Yes I have a sample that has 1000 observations (let's go with it that the sample does represent the overall real population), I'm not sure about much bigger, in fact I don't know anything about the real population, I just have a sample that I can do my testing with. About the second question, I randomly chose 500, for no reason (I am taking an introduction to data analysis, I haven't yet learnt statistics, and we haven't went into detail about the size of samples we should take, I just know that it should be $\endgroup$
    – Pwaol
    Commented Jun 15, 2021 at 13:19
  • 1
    $\begingroup$ But if you sample from your dataset, you cannot know that the null hypothesis is true. You'd need to sample from a uniform instead, and datasets of the same size as the dataset you analyse. $\endgroup$ Commented Jun 15, 2021 at 13:35
  • 1
    $\begingroup$ By the way, there's a standard test for this that avoids empirical sampling, but it may be that this is not what your exercise is about. $\endgroup$ Commented Jun 15, 2021 at 13:43
  • 1
    $\begingroup$ "Now, I started sampling from my dataset, decided to take sample size of 500" - if sampling from your dataset isn't what you really did, maybe you edit your question so that it is clear what you did instead? $\endgroup$ Commented Jun 15, 2021 at 13:44

1 Answer 1

1
$\begingroup$

What you want to find out is whether your dataset looks like a typical dataset of its size (1000) that is drawn from a uniform distribution. In order to find this out, you need to simulate datasets of the same size from the uniform and compare your dataset to these.

Assuming that results are based on uniform samples of size 1000:

  1. You can have a p-value of 0 in this way; if the distribution in your dataset clearly deviates from a uniform, this is not a too unusual thing to happen.

  2. I think what you wrote is not too bad, but it is up to your instructor to tell you how they want you to write these things. More formally one could say "there's strong evidence that the underlying distribution is not a uniform".

  3. I still don't get the bit about the red dot and you still don't explain how you computed your p-value. It can be done in this way in principle (if you did it correctly that is), however there's a standard test (chi squared test) that can be used for this kind of problem, that does not require you to do sampling. Generally try to understand why you're doing what you're doing, then you can feel better about things (your sample size choice of 500 indicates that you don't really understand it).

  4. Nobody is "supposed to feel" anything, but anyway, if this is the empirical distribution you get among 1000 observations, uniformity should be rejected indeed, and actually it doesn't surprise me at all that your sampling procedure gives you $p=0$.

$\endgroup$
2
  • $\begingroup$ Thanks alot for all the help! I really appreciate it!. I think part of me not understanding is how we are learning the stuff (maybe it's done like this because It's an introduction course), I never heard of chi squared test for example, and what we've learnt about p-value calculation is dividing the number of times the samples we generated from the Model got a test statistic which is "More extreme" of our dataset's statistic. (For example, say that I have got the values $1,2,3,4,5$. and my dataset's number was $4.5$ (my red dot). What I do to calculate the p-value is $\frac{1}{5}$) $\endgroup$
    – Pwaol
    Commented Jun 15, 2021 at 14:10
  • $\begingroup$ *Dividing by the whole number of samples I've generated, sorry I forgot to write that $\endgroup$
    – Pwaol
    Commented Jun 15, 2021 at 14:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.