I am conducting linguistic research to determine whether a property of the subject (animacy) of a sentence has its effect on whether a particular kind of preposition phrase will be mentioned. I suspect that there exists a default tendency to mention, and that the animacy effect might vary according to the verb. The data are not experimental, but drawn from a corpus.
My plan is to use a mixed-effect logistic regression for this. Basically, I wanted to investigate three things: 1) the "default" tendency to mention the PP (I take this to be the fixed intercept), 2) a constant effect (if any) of the animacy of the Subject on this tendency and 3) a verb-specific effect, which may be influenced by Subject animacy as well.
lme4, I did
glmer( has_goal ~ bin_figure_anim + (1 + bin_figure_anim | verb), family="binomial", glmerControl=(optimizer="Nelder_Mead", optCtrl=list(maxfun=2e5))), where
bin_figure_anim is the animacy of the subject (0 = inanimate, 1 = animate) and
has_goal is whether the PP is realized (0 = no PP, 1 = PP).
Here are my results:
Generalized linear mixed model fit by maximum likelihood (Laplace Approximation) ['glmerMod'] Family: binomial ( logit ) Formula: has_goal ~ bin_figure_anim + (1 + bin_figure_anim | verb) Data: df Control: glmerControl(optimizer = "Nelder_Mead", optCtrl = list(maxfun = 2e+05)) AIC BIC logLik deviance df.resid 617100.9 617156.5 -308545.4 617090.9 504019 Scaled residuals: Min 1Q Median 3Q Max -3.2132 -0.8807 -0.4373 1.0559 5.7377 Random effects: Groups Name Variance Std.Dev. Corr verb (Intercept) 1.2649 1.1247 bin_figure_anim 0.5367 0.7326 -0.31 Number of obs: 504024, groups: verb, 312 Fixed effects: Estimate Std. Error z value Pr(>|z|) (Intercept) -1.22227 0.06048 -20.209 < 2e-16 *** bin_figure_anim 0.22869 0.04727 4.837 1.32e-06 *** --- Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 Correlation of Fixed Effects: (Intr) bin_figr_nm -0.333
Although the effects are significant, the coefficient is very small (0.22869; odds ratio 1.25695232339). I suspect that this is because I have a very large sample size (500,000+ observations).
Is it reasonable to conclude that the low p-values is an artifact of the large sample size, and that the effect is practically insignificant? What else can I do to gain more confidence in making a conclusion?