I am conducting linguistic research to determine whether a property of the subject (animacy) of a sentence has its effect on whether a particular kind of preposition phrase will be mentioned. I suspect that there exists a default tendency to mention, and that the animacy effect might vary according to the verb. The data are not experimental, but drawn from a corpus.

My plan is to use a mixed-effect logistic regression for this. Basically, I wanted to investigate three things: 1) the "default" tendency to mention the PP (I take this to be the fixed intercept), 2) a constant effect (if any) of the animacy of the Subject on this tendency and 3) a verb-specific effect, which may be influenced by Subject animacy as well.

In lme4, I did glmer( has_goal ~ bin_figure_anim + (1 + bin_figure_anim | verb), family="binomial", glmerControl=(optimizer="Nelder_Mead", optCtrl=list(maxfun=2e5))), where bin_figure_anim is the animacy of the subject (0 = inanimate, 1 = animate) and has_goal is whether the PP is realized (0 = no PP, 1 = PP).

Here are my results:

Generalized linear mixed model fit by maximum likelihood (Laplace Approximation) ['glmerMod']
 Family: binomial  ( logit )
Formula: has_goal ~ bin_figure_anim + (1 + bin_figure_anim | verb)
   Data: df
Control: glmerControl(optimizer = "Nelder_Mead", optCtrl = list(maxfun = 2e+05))

      AIC       BIC    logLik  deviance  df.resid 
 617100.9  617156.5 -308545.4  617090.9    504019 

Scaled residuals: 
    Min      1Q  Median      3Q     Max 
-3.2132 -0.8807 -0.4373  1.0559  5.7377 

Random effects:
 Groups Name            Variance Std.Dev. Corr 
 verb   (Intercept)     1.2649   1.1247        
        bin_figure_anim 0.5367   0.7326   -0.31
Number of obs: 504024, groups:  verb, 312

Fixed effects:
                Estimate Std. Error z value Pr(>|z|)    
(Intercept)     -1.22227    0.06048 -20.209  < 2e-16 ***
bin_figure_anim  0.22869    0.04727   4.837 1.32e-06 ***
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Correlation of Fixed Effects:
bin_figr_nm -0.333

Although the effects are significant, the coefficient is very small (0.22869; odds ratio 1.25695232339). I suspect that this is because I have a very large sample size (500,000+ observations).

Is it reasonable to conclude that the low p-values is an artifact of the large sample size, and that the effect is practically insignificant? What else can I do to gain more confidence in making a conclusion?


You do not indicate if you have multiple human subjects in your study who are supposed to mention these preposition phrases. If you do, your model would need to reflect this by also including a random effect for subject.

With your model as it currently is formulated, the odds ratio of 1.26 refers just to a "typical" verb (i.e., one for which the random intercept and the random slope of bin_figure_anim are equal to 0). The odds ratios corresponding to other verbs will vary around 1.25, with the extent of variation governed by the variance of the random slopes of bin_figure_anim. In your case, it probably would be informative to describe the extent of this variation. So the first takeaway would be to not overreact when seeing the odds ratio for the "typical" verb - the odds ratios for other verbs could be higher/smaller than it.

Focusing on the "typical" verb, you can construct a confidence interval for the true odds ratio associated with it. Some people would choose a higher confidence level (e.g., 99%) to offset the fact that the sample size is quite large. (For the same reason, they would use a significance level alpha = 0.01 for their tests of significance.). The confint() function will help you get this interval on the log odds scale - you can exponentiate its endpoints to get the interval on the odds ratio scale. Let's say this interval comes out to be (1.11, 1.35) on the odds ratio scale. While your best guess from the data is that the true odds ratio for the "typical" verb is 1.26, in actuality this true ratio can be as low as 1.11 and as high as 1.35. This type of statement will more aptly describe the uncertainty involved in estimating the true odds ratio for the "typical" verb. It will also help you focus on describing the "effect size", as recommended in @RobertLong's excellent answer.

The size of the sample should play in your favour in terms of helping you produce a better value for your best guess as to what the true odds ratio is for the "typical" verb. When constructing a confidence interval for this true value, you can guard against the large sample size by choosing a higher confidence level (e.g., 99%). Similarly, you can choose a smaller significance level for your significance tests involving this true value (e.g., alpha = 0.05).

P.S. It's helpful to define your variables and their values more explicitly when posting here. For example, what does bin_figure_anim mean? Is it a binary variable? When does it take the value 0 and when does it take the value 1? Describing your study design (briefly) is also recommended - this way, people can tell whether your proposed model adequately reflects your study design or not.

  • 1
    $\begingroup$ I forgot to mention that this is a corpus study, hence a very large sample. I extracted this data from a parsed corpus. bin_figure_anim is the animacy of the sentence subject, coded as 0 if the subj is inanimate; 1 if animate. has_goal is 1 when the goal PP is mentioned, 0 when it is not. I had fit a model where there is another random intercept that varies according to the sentence ID to account for any sentence-specific effect but the result is that it is almost singular (variance very close to 0). $\endgroup$ – algnot Dec 13 '19 at 16:56
  • 1
    $\begingroup$ I planned on computing and reporting CIs as well, but haven't done it yet because it takes a very long time to compute. Thank you very much for your suggestions. $\endgroup$ – algnot Dec 13 '19 at 17:01
  • 2
    $\begingroup$ You can also look at fitting your model with the mixed_model() function from the GLMMadaptive package (drizopoulos.com/vignettes/glmmadaptive_basics). This package has a marginal_coeffs() function which you can use to get a marginal odds ratio rather than verb-specific ones (rdrr.io/cran/GLMMadaptive/man/marginal_coefs.html). $\endgroup$ – Isabella Ghement Dec 13 '19 at 17:10

As you say, statistical significance is due to the sample size. You should focus on the effect size. 1.26 is not a particularly low odds ratio in many disciplines but in others it is, so it really does depend on what you consider to be low, for your study, in your domain.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.