0
$\begingroup$

Preamble: When testing goodness of fit with null distribution with atoms using an EDF-based statistics $T_n$, random variable $n\sqrt{T_n}$ converges in law.

Empirically, the law observed with absolutely continuous null is different, than with purely discrete null.

I was assessing whether, given position of atoms (say integers 0 through 10), the distribution is the same for different weight assignments. So I generated samples from different discrete distributions, computed KS statistics, binned it, and was trying to compare counts to see if I can detect statistically significant differences.

Using Mathematica:

KolmogorovSmirnovStatisticFiniteBoundDiscrete[
   vec_ /; VectorQ[vec, IntegerQ], 
   {kmin_Integer, kmax_Integer}, 
   pvec_
] /; Min[pvec] >= 0 && Total[pvec] == 1 :=
Module[{cnt, pdfList, ecdf, ncdf},
  cnt = KeySort[Counts[vec]];
  pdfList = AssociationThread[Range[kmin, kmax] -> pvec];
  ecdf = Accumulate[
        Normalize[Values[KeySort[Merge[{cnt, 0 pdfList}, Total]]], Total]];
  ncdf = Accumulate[Values[pdfList]];
  Max[
     Abs[Most[ecdf] - Most[ncdf]],
     If[First[Keys[cnt]] == kmin, 0, 
         Part[ncdf, First[Keys[cnt]] - kmin]],
     If[Last[Keys[cnt]] == kmax, 0, 
         1 - Part[ncdf, First[Keys[cnt]] - kmin + 1]]
   ]
]

Generate KS statistics for discrete uniform nulls on $[0,5]$

With[{dist = DiscreteUniformDistribution[{0, 5}], n = 1600},
 stat1 = Sqrt[n] Table[
     KolmogorovSmirnovStatisticFiniteBoundDiscrete[
      RandomVariate[dist, n], {0, 5}, 
      PDF[dist][Range[0, 5]]], {100000}];]

Generate KS statistics for binomial null $\mathrm{Bin}\left(5,\frac{2}{3}\right)$

With[{dist = BinomialDistribution[5, 2/3], n = 1600},
 stat2 = Sqrt[n] Table[
     KolmogorovSmirnovStatisticFiniteBoundDiscrete[
      RandomVariate[dist, n], {0, 5}, 
      PDF[dist][Range[0, 5]]], {100000}];]

Plot histograms of statistics under different nulls:

Histogram[{stat1, stat2}, Automatic, "Probability", 
 ChartLegends -> {"DiscreteUniform", "Binomial"}]

enter image description here

These statistics have discrete values, for finite $n$, but in the large $n$ limit their cumulative distribution function should converge to an asymptotic continuous CDF.

I was investigating whether these limiting laws are going to be different for different discrete nulls, given both have the same support, as in the above example.

For this I partitioned the positive semi-axis into disjoint intervals, and recorded counts for statistics values in these experiments.

For example:

In[232]:= HistogramList[stat1, {{0, 0.4, 0.9, 100}}]

Out[232]= {{0, 0.4, 0.9, 100}, {19528, 65074, 15398}}

In[233]:= HistogramList[stat2, {{0, 0.4, 0.9, 100}}]

Out[233]= {{0, 0.4, 0.9, 100}, {32064, 58338, 9598}}

Now, given vectors of integer counts, under the null hypothesis that they arose as samples of the same unknown multinomial distribution, I would like to check if data contain evidence against it.


Hence my question:

I have two independent datasets of equal size, and for each of them I compute bin-counts for the common bins.

I hence get two multinomial samples $\{r_1, \ldots, r_n\}$ and $\{t_1, \ldots, t_n\}$. My null hypothesis is that these vectors are samples from the same population distribution.

What test would be appropriate to assess whether differences between $R$ and $T$ vectors can be attributed to randomness.

I am interested in the math part of the question, rather than in actual test computation.

Thank you.

P.S. Feel free to respond using R, Python or any other software package.

$\endgroup$
  • $\begingroup$ I don't understand what you mean by "the math part". What mathematical thing are you after? $\endgroup$ – Glen_b Mar 27 '16 at 4:56
  • $\begingroup$ I am looking to understand the statistic one would choose to test the hypothesis. Given such statistics, I can work out its large $N$ distribution, where $N$ denotes the parameter in the multinomial population under the null. I was hoping the test would be a generalization of Pearson $\chi^2$, which somehow would not require one to know all parameters of the multinomial population distribution. $\endgroup$ – Sasha Mar 27 '16 at 12:13
  • $\begingroup$ Perhaps I am overthinking this, and an EDF-based test applies, where EDF is built from the multinomial sample and the binning boundaries? $\endgroup$ – Sasha Mar 27 '16 at 12:21
  • 1
    $\begingroup$ No; in particular, if $p_k$ is efficiently estimated under the null (where you only have only been given the counts after the binning), you still have a chi-square. The chi-square test of homogeneity is one case where you don't know (and needn't know) $p_k$. But I'm more concerned about these binning boundaries; what was binned and why? Again, are the bins (categories) ordered? $\endgroup$ – Glen_b Mar 27 '16 at 13:17
  • 1
    $\begingroup$ Sound like you're asking for a two-sample goodness of fit test if that's the case there are any number of these, which have different power against different kinds of alternatives. If you can figure out what kind of differences are likely to arise in these circumstances, you could design your test to have good power against that. In particular, your histogram seems to suggest that something related to smooth tests of low order (but adapted to two samples, more like one based on partitions of a test of homogeneity than partitions of chi-squared goodness of fit) might be one good choice, ... ctd $\endgroup$ – Glen_b Apr 7 '16 at 4:37
0
$\begingroup$

What you're trying to can be thought of as a contingency table test, I think.

Imagine a matrix. Each column is one of your bins (so, $n$ columns). Each row is one of your generating distributions (so, 2 rows). Your null hypothesis is that the rows and columns are independent--that the probability of being in row 2, column 3, is the probability of being in row 2 (1/2, assuming you generated equal numbers form each), times the probability of being in column 3 (which we hypothesize is independent of the row).

We decide beforehand how many are going to be in each row, so we have fixed row marginals. We do not decide how many are going to be in each column, so we don't have fixed column marginals. So what I usually use for contingency tables, Fisher's exact test, is I guess not really applicable--it assumes fixed row and column marginals. If you want ot try it anyway, it's fisher.test in R.

There's also the chi-squared contingency table, chisq.test in R. I don't know what the assumptions of this test are, though.

$\endgroup$
  • $\begingroup$ I think the OP is looking to see if the two distributions look the same and not whether or not they are independent. $\endgroup$ – Michael Chernick Apr 25 '18 at 19:56
  • $\begingroup$ I am testing whether the distributions look the same. Think about what it would mean for the row (source distribution) to be independent of the column (the bin). It would mean that the source distributions are the same (or at least, assign the same probability to each bin). $\endgroup$ – user54038 Apr 25 '18 at 19:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.