7
$\begingroup$

I have two sets of data compiled from different sources. Both sets of data contain words with frequencies of occurrence. I would like to check if a certain word exists in both datasets and if it does, whether it is possible to perform some kind of test of significance statistically to prove the word is significant.

For example:

word = 'apple'
dict1 = {'oranges': 45, 'apple': 34, ..., 'x': y}
dict2 = {'apple': 165, 'orange': 12, ..., 'x': y}

If the word "apple" appears in both datasets (dict1 and dict2), then calculate a test of significance for the word apple.

Edit:

  1. First, I want to check if both words exist in both datasets.

  2. For example, if dict has 1000 words, and I arrange the words according to frequencies, I will get some kind of graph. If the top word has a frequency of 13,000 and "apple" has a frequency of 34, I would like to test whether the gap (13,000 - 34) is too huge and the word "apple" didn't appear enough times compared to the top word to be considered significant. However, if 80% of the words fall within the frequency of 20-50, then it's not a good idea to say "apple" is not significant.

  3. I have 2 datasets where the word "apple" may appear. So, I need to make sure "apple" does not fall too low in the frequency value in either or both datasets.

$\endgroup$
  • 1
    $\begingroup$ The phrase "statistically prove the word is significant" doesn't really mean anything. What null hypothesis do you wish to test, and against which alternative? $\endgroup$ – Glen_b -Reinstate Monica Jan 3 '14 at 3:55
  • $\begingroup$ @Glen_b -That was the problem to begin with. I am not too familiar with statistical analysis and I do not know how to test if the word occurs enough time to be significant. I would like to test using the frequency of the words if possible. $\endgroup$ – Cryssie Jan 3 '14 at 6:32
  • 1
    $\begingroup$ "how to test if the word occurs enough time to be significant" -- I have no idea what this means. Can you avoid using statistical terms (like 'significant') and just explain what you want to find out from your data in ordinary English words? What's the basic research question (imagine you're explaining what you're trying to discover to your grandma or something)? $\endgroup$ – Glen_b -Reinstate Monica Jan 3 '14 at 10:23
  • $\begingroup$ @Glen_b - Please refer to the edit in the original question. Sorry I couldn't explain better because I do not know what statistical tests are available for this kind of data. $\endgroup$ – Cryssie Jan 3 '14 at 11:46
  • 1
    $\begingroup$ @learner Perhaps, but I think you're assuming a lot more than is clear from the question. $\endgroup$ – Glen_b -Reinstate Monica Jan 3 '14 at 17:23
4
$\begingroup$

Note: This answer is now deprecated in light of the new information added by the OP. This answer is of related interest in the context, hence not deleted.

You can do a z-test of equality of proportions if the word appears in both the the dictionaries.

There are two steps to this process -- a combination of Python and statistics:

  1. Efficiently create the dictionary of words that are common, computing their relative counts in each of the samples.
  2. Compute a two sample test of proportions, again efficiently for the entire common dictionary.

Efficient creation of common dictionary

An efficient way to compute the proportions (note that all code is Python 3.3) would be to use dictionary comprehensions:

import math as math
import scipy.stats as sps
from collections import defaultdict

dictA = {'word1': 1, 'word2': 4, 'word7': 99, 'word13': 17}
dictB = {'word71': 1, 'word3': 4, 'word2': 99, 'word7': 17, 'word9': 45}

# compute the sums of the frequencies of occurrence of all the words
#     NOTE this is expensive, but is done only once
sumValuesA = sum(dictA.values())
sumValuesB = sum(dictB.values())

dictAB = {key: (value, dictB.get(key)) for key, 
          value in dictA.items() if key in dictB.keys()}
print(dictAB)

Now you have a dictionary that contains the counts of the words in either dictionary. You can form the test of proportions of your choice using dictAB.

Comparing proportions across samples

It is possible to test, based on the proportion of successes in given numbers of trials, whether the probabilities of success are statistically equal across two given samples.

Just to be clear, in what follows, samples are two documents which contains the words, the trials are the total number of words in either document, and the successes are the total number of a particular word in either document.

That is $H_0: p_1 = p_2$ where $p_1$ is the probability of success in population 1, and $p_2$ is the probability of success in sample 2.

The test statistic is $$ Z = \dfrac{\hat{p}_1 - \hat{p}_2}{\sqrt{\hat{p}(1-\hat{p})\left(\tfrac{1}{N_1}+ \tfrac{1}{N_2}\right)}} $$

where $\hat{p}_j = \tfrac{X_j}{N_j}, \, j = 1, 2$, and $X_j$ is the number of successes in the $j$-th population and $N_j$ is the number of trials in the $j$-th population, and $\hat{p} = \tfrac{X_1 + X_2}{N_1 + N_2}$.

Under the null hypothesis, this statistic is standard normal distributed. Here is the Python code to do this:

#================================================
# compute the two sample difference of proportions
#================================================
def fnDiffProp(x1, x2, n1, n2):
    '''
    inputs:
    x1: the number of successes in the first sample
    x2: the number of successes in the second sample
    n1: the total number of 'trials' in the first sample
    n2: the total number of 'trials' in the second sample
    output:
    the test statistic, and the p-value as a tuple
    '''
    hatP = (x1 + x2)/(n1 + n2)
    hatQ = 1 - hatP
    hatP1 = x1/n1
    hatP2 = x1/n2
    Z = (hatP1 - hatP2)/(math.sqrt(hatP*hatQ*(1/n1 + 1/n2)))
    pVal = 2*(1 - sps.norm.cdf(Z))
    return((Z, pVal))

# apply the function above to each of the common words across the
#     two samples
dictPropTest = {key: fnDiffProp(value[0], value[1],
                                 sumValuesA, sumValuesB) for key, value in dictAB.items() }

To interpret for example, the difference in proportion of 'word7' is very significant across documents, whereas it is not for 'word2'.

$\endgroup$
0
$\begingroup$

You want to know when the difference between counts (a Gap) of the same word in different groups is unexpectedly large. It sounds like you are assuming that the Gap between different groups generally speaking should be small. You also only have 2 groups of counts - you only give 2 datasets - which would make testing for significance difficult.

You could try a Bayesian approach, provided that you are willing to make some assumptions. Bayes' theorem relates the data you have to ideas about the distributions from which your data came. Specifically, the theorem, when applied to the multiple hypotheses of your problem looks like:

$P(H_1|Gap) = \frac{P(Gap|H_1)P(H_1)}{P(Gap|H_1)P(H_1) + P(Gap|H_2)P(H_2)}$

for the first hypothesis and

$P(H_2|Gap) = \frac{P(Gap|H_2)P(H_2)}{P(Gap|H_1)P(H_1) + P(Gap|H_2)P(H_2)}$

for the second. The issue here is the definition of $H_1$ and $H_2$: what are they and how do you specify them? Once you know what they are, what about conditional distributions?

I took a stab at doing this in Python (code/results below) but first, how I answered these questions.

I assumed that you have no idea whether the Gap between 2 counts of a word should be large or small, so I assumed that for each word it could go either way, with 50/50 chance and set $P(H_1) = 0.5$ and $P(H_2) = 0.5$.

Next, what about the conditionals? Well, you want to know about the probability of a Gap given some expected Gap size. As the Gaps will be integers, it made sense to model the distribution of Gaps as a Poisson distribution, with each hypothesis $H$ getting its own $\lambda$ parameter. That is, each $H \sim Pois(\lambda, Gap)$ with a different $\lambda$.

These assumptions are made with no real data, and that is the drawback of this approach. That being said, here is what an implementation could look like:

import math

def pois(l, k):
  return math.exp(-l) * (l**k)/math.factorial(k)

def bayes_test(h1, h2, gap):
  lp1 = pois(h1, gap) * 0.5
  lp2 = pois(h2, gap) * 0.5
  pb = lp1 + lp2

  print ''.join(["P( H1 | gap ): ", str(lp1/pb), "\nP( H2 | gap ): ", str(lp2/pb)])
  return [lp1/pb, lp2/pb]

And here is what possible output would look like:

>>> bayes_test(7, 30, 10)
P( H1 | gap ): 0.999785530317
P( H2 | gap ): 0.000214469683207
[0.9997855303167933, 0.00021446968320674848]

Using the Poisson distribution function I defined, we set $\lambda_1 = 7$ and $\lambda_2 = 30$, meaning that our hypotheses are that the expected Gap size is either 7 for $H_1$ or 30 for $H_2$, and we test with some observed Gap size of 10. We see then that it is much more likely that our data is explained by an expected Gap size of 7 than of 30.

So, the takeaway is that if you can rework your question to be asking about several hypotheses, you can use a Bayesian approach to ask which hypothesis is more probable.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.