3
$\begingroup$

I am trying to compare two arrays of frequencies. The second is a subset of the first. I want to know if the second is a representative of the first. The arrays are:

x    All  Subset
a    136   38
5    127   27
b    103   23
1    102   17
6     71   11
2     27    4

I want a test that, where B behaves like A (i.e. can be viewed as A scaled), the test returns a p-value close to 1 (Null hypothesis is that for each point on x axis, percentage frequencies of B are the same of percentage frequencies of A).

I tried to use Chi-squared test but, since I could have missing values, I don't know if the test validity can be compromised.. Data is missing not for technical failures but because in one subset is likely to have only few features (x values) with non-zero frequencies.

Moreover since the size of the second column (the subset) is not fixed, I don't know how to scale the first column in order to obtain a valid p-value (now magnitude order is 10e-80).

Thanks

Improve this question
$\endgroup$
8
  • 1
    $\begingroup$ Chi-square tests generally work well whenever expected frequencies all exceed about 1; the ancient advice about a threshold of 5 still lingers on, despite being rebutted many times. As for your "missing" cell, if there are no relevant items the observed frequency must be entered as 0. Your problem must be converted to a problem about frequencies, not percents. I can't see that Kolmogorov-Smirnov applies to arbitrary categorical variables, as the cumulative distribution is then not uniquely defined. $\endgroup$
    – Nick Cox
    Dec 9, 2013 at 12:55
  • $\begingroup$ KS test assumes continuous distributions. If you apply it to discrete distributions while ignoring that fact it's pretty conservative; you need to simulate the null distribution. "I decided not to use it because it seems that it need at least a value of 5 in each cell of the array." -- did you even read the first answer of the question you pointed to? Or the second-last comment under the question? $\endgroup$
    – Glen_b
    Dec 9, 2013 at 14:10
  • $\begingroup$ Sorry @Glen_b I misread that answer. I'll change the question accordingly if necessary.. $\endgroup$
    – gc5
    Dec 9, 2013 at 14:23
  • $\begingroup$ @NickCox question updated $\endgroup$
    – gc5
    Dec 9, 2013 at 15:19
  • $\begingroup$ Your question is updated, but this point made earlier remains valid if you are considering chi-square: Your problem must be converted to a problem about frequencies, not percents. Also, neither A nor B gives the expected frequencies, or even percents; expected frequencies come from a weighted average of A and B. $\endgroup$
    – Nick Cox
    Dec 9, 2013 at 15:27

3 Answers 3

Reset to default
3
$\begingroup$

Fisher's exact test is useful for cases where you WOULD have used a chi-square test but don't know if you will always meet the cell count conditions (automated testing of survey data for example). It should be noted that Fisher's exact test can be a bit more timid about proclaiming significance (it's more conservative than chi-square).

It's part of the stats package in R... see ?fisher.test

Improve this answer
$\endgroup$
11
  • $\begingroup$ Since I want to use it to assess if between two states the last state has a "better" (in the sense it appears more similar in terms of frequencies) than the first state with respect to an array of expected frequencies, I think this could be a good idea to use it. Thanks $\endgroup$
    – gc5
    Dec 10, 2013 at 12:33
  • $\begingroup$ Suppose I have more row categories (as the example). I decided to run N tests (where the two row categories are category and ~category. How to compute a single pvalue for the N tests? I don't know if it's clear.. $\endgroup$
    – gc5
    Dec 10, 2013 at 12:48
  • 2
    $\begingroup$ An alternative is to do an exact version of the chi-square test (that is, use a chi-square test statistic, but use its exact permutation distribution conditional on the margins). $\endgroup$
    – Glen_b
    Dec 10, 2013 at 20:13
  • 1
    $\begingroup$ The place to begin would be reading about permutation (and randomization) tests. Once you understand permutation tests, the rest is simple application of the principles to a chi-square statistic. See the discussion on the following questions, for example: Q1 ... (ctd) $\endgroup$
    – Glen_b
    Dec 11, 2013 at 15:09
  • 2
    $\begingroup$ ctd... Q2 $\quad $ Q3 $\quad $ Q4 $\quad $ Q5 $\quad $ Q6 $\endgroup$
    – Glen_b
    Dec 11, 2013 at 15:11
3
$\begingroup$

I am trying to compare two arrays of frequencies. The second is a subset of the first.

This makes them dependent. Normally the right thing to do is compare two distinct sets:

x     Subset   Not-in-Subset
a       38        98
5       27       100
b       23        80
1       17        85
6       11        60
2        4        23

If they behave alike, then the subset behaves like the whole. This is fairly simple logic. Call the "not in subset" values "C". If B has the same distribution as C (the null in the test) and B obviously has the same distribution as itself(!), then B has the same distribution as B+C (i.e. A) -- if you require it, I could show it mathematically, but it's rather trivial.

Consider the subset. If the underlying proportions in each category of the subset (the things the sample proportions estimate) were not the same as the remainder, it could not be the as the population as a whole.

I tried to use Chi-squared test but, since I could have missing values, I don't know if the test validity can be compromised.

Can you say more about what is missing and how it arises?

Missingness may be a problem (or may not be a problem) for almost any procedure, depending on its nature.

However, note that categories with ordinary zero counts aren't 'missing values' in the required sense; our data here are counts, if they're just 0's, they aren't missing, you have an observed count of 0.

Moreover since the size of the second column (the subset) is not fixed, I don't know how to scale the first column in order to obtain a valid p-value (now magnitude order is 10e-80).

A chi-square can deal with this in the usual fashion.

Improve this answer
$\endgroup$
3
  • $\begingroup$ 1) Ok, just out of curiosity, can you point me to something that explain "If they behave alike, then the subset behaves like the whole." 2) Updated answer for missing value, I don't know if more informations are needed 3) Ok, so I don't scale anything.. $\endgroup$
    – gc5
    Dec 9, 2013 at 21:24
  • $\begingroup$ 1) It's simple logic. Call the "not in subset" values "C". If B has the same distribution as C (the null in the test) and B obviously has the same distribution as itself(!), then B has the same distribution as B+C (i.e. A). 2) zero frequencies aren't missing in the usual sense; this should cause no problems, though if the expected values were small you might have to modify the test slightly (this presents little difficulty). 3) correct - the chi-square test itself computes appropriate 'scaling' from the table margin. $\endgroup$
    – Glen_b
    Dec 9, 2013 at 21:35
  • $\begingroup$ Ok, one more question. In scipy docs.scipy.org/doc/scipy/reference/generated/… I am able to test whether the subset is dependent on the not-in-subset; is this the same thing of testing if the subset is dependent to the whole, and if the subset behaves like the whole? $\endgroup$
    – gc5
    Dec 10, 2013 at 13:27
1
$\begingroup$

Take the Spearman's Rank Correlation for the two columns. Perform the test of significance either using the Permutation test or the Fisher transform as has been defined in the wiki page. this will establish any monotone dependency between the two sets of numbers and is a non-parametric method so no assumptions required about the data.

Improve this answer
$\endgroup$
1
  • $\begingroup$ I think it cannot be used because it computes the correlation between ranks, so the correlation between [1,2,3] and [2,3,4], and the correlation between [1,2,3] and [2,3,5] is the same (and I don't want it to). Or did I misunderstand? $\endgroup$
    – gc5
    Dec 9, 2013 at 15:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.