1
$\begingroup$

Preamble: When testing goodness of fit with null distribution with atoms using an EDF-based statistics $T_n$, random variable $n\sqrt{T_n}$ converges in law.

Empirically, the law observed with absolutely continuous null is different, than with purely discrete null.

I was assessing whether, given position of atoms (say integers 0 through 10), the distribution is the same for different weight assignments. So I generated samples from different discrete distributions, computed KS statistics, binned it, and was trying to compare counts to see if I can detect statistically significant differences.

Using Mathematica:

KolmogorovSmirnovStatisticFiniteBoundDiscrete[
   vec_ /; VectorQ[vec, IntegerQ], 
   {kmin_Integer, kmax_Integer}, 
   pvec_
] /; Min[pvec] >= 0 && Total[pvec] == 1 :=
Module[{cnt, pdfList, ecdf, ncdf},
  cnt = KeySort[Counts[vec]];
  pdfList = AssociationThread[Range[kmin, kmax] -> pvec];
  ecdf = Accumulate[
        Normalize[Values[KeySort[Merge[{cnt, 0 pdfList}, Total]]], Total]];
  ncdf = Accumulate[Values[pdfList]];
  Max[
     Abs[Most[ecdf] - Most[ncdf]],
     If[First[Keys[cnt]] == kmin, 0, 
         Part[ncdf, First[Keys[cnt]] - kmin]],
     If[Last[Keys[cnt]] == kmax, 0, 
         1 - Part[ncdf, First[Keys[cnt]] - kmin + 1]]
   ]
]

Generate KS statistics for discrete uniform nulls on $[0,5]$

With[{dist = DiscreteUniformDistribution[{0, 5}], n = 1600},
 stat1 = Sqrt[n] Table[
     KolmogorovSmirnovStatisticFiniteBoundDiscrete[
      RandomVariate[dist, n], {0, 5}, 
      PDF[dist][Range[0, 5]]], {100000}];]

Generate KS statistics for binomial null $\mathrm{Bin}\left(5,\frac{2}{3}\right)$

With[{dist = BinomialDistribution[5, 2/3], n = 1600},
 stat2 = Sqrt[n] Table[
     KolmogorovSmirnovStatisticFiniteBoundDiscrete[
      RandomVariate[dist, n], {0, 5}, 
      PDF[dist][Range[0, 5]]], {100000}];]

Plot histograms of statistics under different nulls:

Histogram[{stat1, stat2}, Automatic, "Probability", 
 ChartLegends -> {"DiscreteUniform", "Binomial"}]

enter image description here

These statistics have discrete values, for finite $n$, but in the large $n$ limit their cumulative distribution function should converge to an asymptotic continuous CDF.

I was investigating whether these limiting laws are going to be different for different discrete nulls, given both have the same support, as in the above example.

For this I partitioned the positive semi-axis into disjoint intervals, and recorded counts for statistics values in these experiments.

For example:

In[232]:= HistogramList[stat1, {{0, 0.4, 0.9, 100}}]

Out[232]= {{0, 0.4, 0.9, 100}, {19528, 65074, 15398}}

In[233]:= HistogramList[stat2, {{0, 0.4, 0.9, 100}}]

Out[233]= {{0, 0.4, 0.9, 100}, {32064, 58338, 9598}}

Now, given vectors of integer counts, under the null hypothesis that they arose as samples of the same unknown multinomial distribution, I would like to check if data contain evidence against it.

Hence my question:

I have two independent datasets of equal size, and for each of them I compute bin-counts for the common bins.

I hence get two multinomial samples $\{r_1, \ldots, r_n\}$ and $\{t_1, \ldots, t_n\}$. My null hypothesis is that these vectors are samples from the same population distribution.

What test would be appropriate to assess whether differences between $R$ and $T$ vectors can be attributed to randomness.

I am interested in the math part of the question, rather than in actual test computation.

Thank you.

P.S. Feel free to respond using R, Python or any other software package.

Improve this question
$\endgroup$
12
  • $\begingroup$ I don't understand what you mean by "the math part". What mathematical thing are you after? $\endgroup$
    – Glen_b
    Mar 27, 2016 at 4:56
  • $\begingroup$ I am looking to understand the statistic one would choose to test the hypothesis. Given such statistics, I can work out its large $N$ distribution, where $N$ denotes the parameter in the multinomial population under the null. I was hoping the test would be a generalization of Pearson $\chi^2$, which somehow would not require one to know all parameters of the multinomial population distribution. $\endgroup$
    – Sasha
    Mar 27, 2016 at 12:13
  • $\begingroup$ Perhaps I am overthinking this, and an EDF-based test applies, where EDF is built from the multinomial sample and the binning boundaries? $\endgroup$
    – Sasha
    Mar 27, 2016 at 12:21
  • 1
    $\begingroup$ No; in particular, if $p_k$ is efficiently estimated under the null (where you only have only been given the counts after the binning), you still have a chi-square. The chi-square test of homogeneity is one case where you don't know (and needn't know) $p_k$. But I'm more concerned about these binning boundaries; what was binned and why? Again, are the bins (categories) ordered? $\endgroup$
    – Glen_b
    Mar 27, 2016 at 13:17
  • 1
    $\begingroup$ Sound like you're asking for a two-sample goodness of fit test if that's the case there are any number of these, which have different power against different kinds of alternatives. If you can figure out what kind of differences are likely to arise in these circumstances, you could design your test to have good power against that. In particular, your histogram seems to suggest that something related to smooth tests of low order (but adapted to two samples, more like one based on partitions of a test of homogeneity than partitions of chi-squared goodness of fit) might be one good choice, ... ctd $\endgroup$
    – Glen_b
    Apr 7, 2016 at 4:37

1 Answer 1

Reset to default
1
$\begingroup$

What you're trying to can be thought of as a contingency table test, I think.

Imagine a matrix. Each column is one of your bins (so, $n$ columns). Each row is one of your generating distributions (so, 2 rows). Your null hypothesis is that the rows and columns are independent--that the probability of being in row 2, column 3, is the probability of being in row 2 (1/2, assuming you generated equal numbers form each), times the probability of being in column 3 (which we hypothesize is independent of the row).

We decide beforehand how many are going to be in each row, so we have fixed row marginals. We do not decide how many are going to be in each column, so we don't have fixed column marginals. So what I usually use for contingency tables, Fisher's exact test, is I guess not really applicable--it assumes fixed row and column marginals. If you want ot try it anyway, it's fisher.test in R.

There's also the chi-squared contingency table, chisq.test in R. I don't know what the assumptions of this test are, though.

Improve this answer
$\endgroup$
2
  • $\begingroup$ I think the OP is looking to see if the two distributions look the same and not whether or not they are independent. $\endgroup$
    – Michael R. Chernick
    Apr 25, 2018 at 19:56
  • 1
    $\begingroup$ I am testing whether the distributions look the same. Think about what it would mean for the row (source distribution) to be independent of the column (the bin). It would mean that the source distributions are the same (or at least, assign the same probability to each bin). $\endgroup$
    – user54038
    Apr 25, 2018 at 19:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.