# How do I calculate effect size for percentages of totals?

I apologize if this questions is poorly worded or has a very simple answer but I work in a field very different than statistics. I am looking at diction (word use) in books from two different artistic movements over time. I want to compare words that were first used in each century by each set of authors. For example both sets use words from the 18th century at a rate of 15.5 and 15.2 percent of their total word use. How do I find out if this .3 percent means anything? Would I be looking at effect size? And, if so, what measure should I use?

• Well 0.3 percent sounds like a trivial difference to me, but I'd not in your field. Does it sound like a meaningful difference to you? Apr 11, 2015 at 14:07
• To be honest it does not, but I am worried about assuming triviality. And also, I would like to develop the skills to figure out what threshold would not be trivial. Apr 11, 2015 at 14:15
• It would come from subject-area knowledge, numbers don't carry inherent meaningfulness. If I was in high energy physics that 0.3% might be a big change (it might be much larger than effects they look for there), but in some other subject areas it could be so tiny as to be not worth mentioning. Apr 11, 2015 at 14:21
• Thanks glen, I appreciate the comments. I think part of what I am trying to figure out is what constitutes a big change. Im on a fact finding mission to more or less see what areas I should be looking at and if statistics can help. Apr 11, 2015 at 14:29

Look up Cohen's $$h$$. First, you calculate $$h$$, which is pretty straightforward:

$$h = 2\times\arcsin\left(\sqrt{p_1}\right) - 2\times\arcsin\left(\sqrt{p_2}\right)$$

Where $$p_1$$ and $$p_2$$ are the two proportions.

Then, you have to decide on a cutoff. The "rule of thumb" cutoff is that if $$h \ge 0.2$$, then you have something interesting. Though in your particular science, a different cutoff might be more appropriate.

You can test whether two proportions are equal using a test like the one described here.

This is implemented in R in the prop.test function. For example, your data might look like this:

# count of target words for author 1 and author 2
word_counts <- c(27, 46)

# total words for author 1 and author 2
total_words <- c(173, 302)

# here are the proportions for the two authors:
> word_counts/total_words
[1] 0.1560694 0.1523179


This tests the null hypothesis that the rate in both groups is the same:

> prop.test(x=word_counts, n=total_words)

2-sample test for equality of proportions with continuity correction

data:  word_counts out of total_words
X-squared = 2.2201e-30, df = 1, p-value = 1
alternative hypothesis: two.sided
95 percent confidence interval:
-0.06757982  0.07508279
sample estimates:
prop 1    prop 2
0.1560694 0.1523179


Note that if the sample size is large enough, even this small difference in proportions can be significant:

# count of target words for author 1 and author 2
word_counts <- c(27000, 46000)

# total words for author 1 and author 2
total_words <- c(173000, 302000)

> prop.test(x=word_counts, n=total_words)

2-sample test for equality of proportions with continuity correction

data:  word_counts out of total_words
X-squared = 11.873, df = 1, p-value = 0.0005696
alternative hypothesis: two.sided
95 percent confidence interval:
0.001609876 0.005893091
sample estimates:
prop 1    prop 2
0.1560694 0.1523179


There are a few ways to do this, but I think that the way to go there is making a contingency table with the counting and applying a chi-square test to check if counting of the sets differed on the centuries.