# Test statistic and null rejection of independence in normalized redundancy, as implemented and automated in Python

I currently have an output function that roughly follows a $\chi^2$ distribution for random data with one degree of freedom. As a cursory analysis that has since proven mathematically unrigorous, this appeared to be bounded by $[0,1]$ on the x-axis, with a mode of 0, and a steep falloff such that my PDF offers highly significant results after a value of roughly 0.1 on said x-axis.

My input data set is derived from the normalized redudancy of categorical bins of data. My null hypothesis is that the data coming in for two arbitrarily selected datasets are independent (that is, $p(x,y) = p(x)\ p(y)$ for all $x \in X$ and $y \in Y$ ). My test statistic is the measure from which this test for independence deviates from the observed data. At present, the closed mathematical model for my test statistic remains undevised.

Unsurprisingly because of this, after slogging through the documentation for various tools to provide a statistical measure of significance in the context of this distribution, including SciPy (the framework I am using for other tests, as all of this work is currently in Python), I am unable to come to a good, working definition for my test statistic and significance thereof. Even if I do, I am unsure how to implement it effectively, as tools like SciPy are sparsely and inadequately documented. I understand, for example, how to integrate an arbitrary distribution, but not how to construct one of my devising in line with these requirements.

To this end, I've played with the input parameters for scipy.stats.chi2 and can get a non-robust, 0th level approximation that roughly aligns with my observed output. However, I am unable to see how my model aligns with the constructs provided by SciPy, as I fail to grasp SciPy's object model in the context of the statistical and mathematical definitions of, for example, a $\chi^2$ distribution. I am also not cognizant on the correct terminology to further specify this question, and as evidenced, I have altered its title and content severely because of this.

So, as a further specified question, this becomes two pieces:

• Given these requirements, how might I construct a good, statistically rigorous test statistic and rejection criteria for null? This will be used in an automated manner, so showing the closed form for the approach is greatly appreciated, albeit not required, to solve this problem.

• How might I implement this in Python? Because the documentation is so tersely insufficient, links to further sourcing and crisp definitions of all constants, variables, and non-obvious operators (to a high school graduate with a fundamental course in probabilistic statistics) would be greatly appreciated.

• It is unclear what your real question is. It is obvious that no function can look like $\chi^2(1)$ (which is an exponential distribution) and still essentially be supported on $[0,1]$, so there's a question about your characterization. Do you need to determine the null distribution of a test statistic? If so, we need to know how that statistic is computed and how the data are generated. Otherwise, what does this "function" really represent and how do you intend to use it? – whuber Oct 28 '11 at 14:24
• @whuber Clear, concise, and useful. Thank you. -- My overarching goal, as further defined here (math.stackexchange.com/questions/74675/…), is to discover highly dependent pairwise features from given bags of categorical data, where the data significantly distinguishes itself from random noise. I am roughly trying to devise an alternative to Pearson's or Spearman correlation for discrete data with no obvious natural order, and my methods and ability to phrase this problem are simply failing me. – MrGomez Oct 28 '11 at 18:43
• @whuber The implementation goal, of course, is to support this test in an automated manner in Python. Because I am out of my depth, I do wonder if I have simply sketched out a familiar problem with a familiar solution, to which a terse analogy may be appropriate. For example, my methods sound very much like what is prescribed for Pearson's $\chi^2$ Test for Goodness of Fit (www-users.math.umd.edu/~nstrawn/Lecture25.pdf). – MrGomez Oct 28 '11 at 18:50
• A further specification of the problem is that my null hypothesis should be that my input features are independent -- that is, they satisfy $p(x,y) = p(x)\ p(y)$ for all $x \in X$ and $y \in Y$. My presumption that the null requires randomness is false (as this would be a subset of the featurespace for independence). As such, I will update my question to reflect this. – MrGomez Oct 28 '11 at 19:19
• This situation sounds exactly like the setting for a $\chi^2$ GOF test. Could you elaborate on what special aspects of your problem cause you to think that this standard test of independence is inappropriate for your data? – whuber Oct 28 '11 at 20:05