Your "test of tests" sounds like a mixed model. In this formulation you aren't concerned with estimating the effect word by word, but rather want to know whether there's an overall effect, so you treat the variations in underlying rates by word as random.
My simulation experiments suggest you have a good chance to detect and quantify relatively subtle effects (such as a log odds ratio of $0.25$ or less) provided you have around 20 or more instances of each basic word on average. The simulation (detailed in code below) recreates all the features you have described: around 100 words, each possibly appearing in uncontracted or contracted form, with greatly varying frequencies.
Let's look at one simulation and what this model does with it. The data are represented in a table showing the base word form (a randomly created nonsense word), the form of the word that was observed, the number of times this form was followed by a pause ("um"), and the number times it was not followed by a pause.
> print(df[, .(base, form, contracted, pause, pause.not)])
base form contracted pause pause.not
1: abhbtf abhb TRUE 9 26
2: abhbtf abhbtf FALSE 1 40
3: agmkkgtrkv agmk TRUE 0 20
192: zprdpoobkstwu zprdpoobkstwu FALSE 0 1
193: zxpvvluon zxpv TRUE 5 35
194: zxpvvluon zxpvvluon FALSE 4 34
Here is a plot of these data. Each dot reflects a single row of the table, located according to whether it is in contracted form (horizontally) and the observed frequency (vertically). Dots are sized in proportion to the total observed frequencies; for instance, the dot for the first data row (word
abhb) has an area proportional to $9+26.$ The colored polygons are "violin plot" estimates of the data distributions (although they do not account for the variation in observation sizes).
There is a slight hint that pauses after contracted word forms (right violin) may be a little more frequent than pauses after base word forms (left violin). It's not perfectly clear that there is any real difference, though, because the many infrequent words (shown with tiny dots) have extremely variable frequencies, as high as 100% in several cases.
The mixed model supposes the true rate of response varies independently and randomly among all forms of all words (contracted or not), but that the mean rate for contractions differs from the mean rate for base forms of the words. This is a parsimonious model: it estimates just three parameters describing the rates for uncontracted word forms, the difference in mean log odds of rates (which is the parameter you wish to test), and the variance of those log odds.
Here is a typical model summary:
Groups Name Variance Std.Dev.
base:contracted (Intercept) 0.3397 0.5829
Estimate Std. Error z value Pr(>|z|)
(Intercept) -2.6114 0.1332 -19.607 < 2e-16 ***
contractedTRUE 0.4949 0.1724 2.871 0.00409 **
The value of $0.4949$ for
contractedTRUE is the estimate of how much the "um" rate for contracted word forms exceeds the rate for uncontracted forms, expressed as a log odds ratio. Its small p-value of $0.00409$ would usually be considered significant evidence of a real underlying difference. Indeed, these data were simulated from a "reality" in which the base log odds is $-2.5$ and the difference is $+0.5.$ In this case, at least, the estimates are highly accurate.
The standard deviation of the random effect, $0.5829,$ indicates how variable the estimated rates are from word to word. The data were drawn from a population in which the true standard deviation is $0.5:$ the estimate is pretty good.
It is of interest to compare model-predicted rates with observed rates. As that SD of $0.58$ indicates, much of the variation in observed rates looks random. The mixed model "discounts" this inherent variation, essentially by "shrinking" the observed rates towards the overall average rate. Furthermore, words seen infrequently naturally experience more shrinkage towards the average. In the plot, this shows up in the small dots strung out to the right: they all had unusually high rates, but the model attributes that almost entirely to chance and thereby predicts their underlying rates to be near the average. The larger the dot, the more times that word was seen, and the closer the prediction comes to reproducing the observed frequencies.
Much more can be said. A good resource for learning about mixed models is Fitzmaurice et al., Applied Longitudinal Analysis (Wiley 2004). I found it informative to vary the parameters of the simulation below to see how well the model works under different circumstances and to understand better how to interpret the model output. Experimentation is fast and easy, because largish datasets (millions of observations) involving hundreds of words can be generated, modeled, summarized, and plotted within a second.
This simulation runs in
R. It uses the
data.table package for some data manipulation,
lme4 for the modeling, and
ggplot2 for plotting.
n.words <- 120 # Total *possible* word forms -- some might not appear
log.odds.base <- -2.5 # Log odds of base form followed by "um"
log.odds.contracted <- -2 # Log odds of contracted form followed by "um"
sigma <- 0.5 # SD of log odds
kappa <- 0.5 # Gamma parameter of relative frequency distribution
n <- 2.5e1 * n.words # Mean observations per word form
#set.seed(17) # Set for reproducible results
#-- Create the base forms of the words
base <- replicate(n.words, paste0(sample(letters, sample.int(8)+5, replace=TRUE), collapse=""))
#-- Describe the actual word behavior
f <- function(y) 1 / (1 + exp(-y)) # Convert from log odds to probability
X <- data.frame(
contracted=c(rep(FALSE, n.words), rep(TRUE, n.words)),
frequency=rep(rgamma(n.words, kappa), 2),
prob.pause=f(c(rnorm(n.words, log.odds.base, sigma),
rnorm(n.words, log.odds.contracted, sigma)))
p <- X$frequency; p <- p / sum(p)
i <- sample.int(nrow(X), n, replace=TRUE, prob=p)
pause <- runif(n) < X$prob.pause[i]
#-- Create a data table to record the raw simulation results
df.raw <- data.table(base = X$base[i],
form = X$form[i],
contracted = X$contracted[i],
pause = pause)
#-- Summarize the raw data in a form suitable for analysis
df <- df.raw[, .(pause=sum(pause), pause.not=sum(!pause)),
keyby=.(base, form, contracted)]
fit <- glmer(cbind(pause, pause.not) ~ contracted + (1 | base:contracted),
fit.base <- glmer(cbind(pause, pause.not) ~ contracted + (1 | base),
# model.matrix(fit, type="random")
# model.matrix(fit.2, type="random")
anova(fit.base, fit) # Compares competing models
#-- Calculate predicted and observed rates for plotting
df[, rate:=predict(fit, type="response")]
df[, observed := pause/(pause + pause.not)]
df[, size := pause + pause.not]
#-- Establish a plotting range comprising all data
prange <- with(df, range(c(range(observed), range(rate))))
#-- Plot predicted vs. observed
ggplot(df, aes(observed, rate, group=contracted)) +
coord_fixed(ratio=1, xlim=prange, ylim=prange) +
geom_point(aes(fill=contracted, size=size), alpha=1/2, shape=21) +
geom_abline(intercept=0, slope=1, size=1) +
xlab("Observed pause frequency") +
ylab("Predicted pause frequency") +
ggtitle("Predicted vs. Observed", "Binomial Mixed Model")
#-- Plot the data
ggplot(df, aes(contracted, observed, fill=contracted)) +
position=position_jitter(height=0.01, width=0.05)) +
scale_size_continuous(range=c(0.1, 3)) +
ggtitle("Observed Frequencies", "Simulated Data")