1
$\begingroup$

Consider a random vector $X \in \mathbb{R}^{d}$ with support $\text{supp}(X) = \{1,2,3,4\}^d$, and let $P_X$ denote its known probability mass function. Note that $\lvert \text{supp}(X) \rvert = 4^d$.

I have $n \ll 4^d$ i.i.d. samples of $X$, and I suspect that these are not distributed according to $P_X$. The following are my questions:

  1. Is it more appropriate to make the null hypothesis that the distribution of the samples is not $P_X$, or should I only consider the alternative to be the null hypothesis? Typically, hypothesis tests seem to make the null hypothesis that the distributions are the same.
  2. What should be my hypothesis test, considering that my sample size $n$ is far less than the support of $X$?
  3. Related to question 2, suppose $d = 4$ and $n = 50$. Then I have $50$ samples but my random vector $X$ can take $256$ possible values. In this setting, it seems difficult to even estimate the distribution from which the $50$ samples are generated with reasonable accuracy. However, I can estimate, e.g., the probability $\mathbb{P}(X_4 = 1)$ from the sample with reasonably high accuracy and compare it to the true value from the marginal of $P_X$. Therefore, it looks like I can potentially use such marginal tests to not reject my hypothesis that the sample distribution is not $P_X$. This feels ad hoc, though - there could be a whole bunch of other tests I can use that are not captured by testing marginals.
$\endgroup$
1
  • 1
    $\begingroup$ Could you add a little information about why you are doing this? Where do your numbers come from? Where does $P_x$ come from? why and in what manner do you think they won't match? This type of background could make the difference between a generic answer that rehashes textbook stat theory and an answer that helps you in your research (not mutually exclusive, I know). $\endgroup$ Oct 21 '20 at 1:22
0
+50
$\begingroup$

TL;DR: Try using likelihood ratio test statistic and evaluate the p-value numerically

Ok, so here's some highly simplified hypothesis testing basics, please correct me if I misremember or oversimplify something.

  1. Let there be some probability distribution $P[X|\theta]$, where $X$ is 1 sample (in your case of dimension $4^d$), and $\theta$ be the parameters of that distribution (in discrete case, the probabilities of each outcome). Then, $P[Y|\theta] = \prod_i P[X_i|\theta]$, where $Y$ refers to all samples in the dataset
  2. Let there be some specific value $\theta_0$ of parameters that we wish to validate. In particular, we doubt whether $\theta_0$ is actually the true parameters, and want to try to refute them given the data we have. Thus, $H_0: \theta = \theta_0$ will be a null hypothesis we will try to refute
  3. By plugging in your data $Y_{data}$ into $P[Y_{data}|\theta_0]$ you will know how likely is it to observe this data. Almost good enough to just use this value, except that we need to be able to compare it to something to see if that probability is actually very high or very low.
  4. The main logic of hypothesis testing is to check if the observed data is in some sense "extreme" based on the null hypothesis probability distribution. If this is the case, we are able to refute the null hypothesis as unlikely to have resulted in the observed data (see above link on exact philosophical implications of this statement, it is quite counter-intuitive). The only way to compare two objects is to convert each object to a number and then compare the numbers. In math its called metric, in hypothesis testing its called test statistic. We can try to use the likelihood $L(Y, \theta) = P[Y|\theta]$ as the test statistic. We can thus evaluate, with what probability the null hypothesis could have generated a data that was as unlikely or more unlikely than the data we have observed. This is commonly called the p-value, and it is the number we will either report directly as the outcome of our analysis, or compare to some pre-defined confidence level (eg 1%), and based on that comparison decide whether to refute $H_0$. Namely, we are interested in the probability $$p = P_Y \bigl [L(Y,\theta_0) \leq L(Y_{data},\theta_0) \bigr]$$ This is a Complementary Cumulative Distribution Function (CCDF) of the test statistic. In some very special cases (like multivariate normal distribution), it has an explicit analytic form, and one can just plug in the data and the null hypothesis parameters and get out the p-value. But in most cases no analytic expression is available. Thus, we have to resort to permutation testing. Basically, we sample random data from null hypothesis many times (e.g. 10000), for each of those we calculate the value of the test statistic, and then see in what fraction of cases the random test statistic was more extreme than the true one. Note that in this case the p-value will be approximate, and will be lower-bounded by 1/10000. The number of required samples depends not so much on the dimension, as on the level to which the observed data violates the null hypothesis if at all. So if the violation is strong, 50 samples may be enough
  5. Finally, not all test statistics are made equal in this world. While likelihood feels like a very natural choice of a test statistic, it is not actually the most powerful, as in, it may be too conservative when refuting the null hypothesis for reasons that are complicated (honestly, I did not fully understand them after multiple attempts of doing so over the years). The literature suggests that a more powerful test statistic is the likelihood-ratio. It's like likelihood, but gets normalized by likelihood for MLE-estimated parameters. Repeating the same procedure as in 4. for this test statistic is probably the best you can do.
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.