4
$\begingroup$

I'm looking to see if income level affects the number of pets a person has, controlling for number of children. I have run a GLM (Poisson family) model, and found that if a person is categorised as 'Low' income, their number of expected pets decreases by 21.3%. I also found that for every child a person has, their number of expected pets increases by 4.03%.

However, the 21.3% decrease is not significant, and the 4.03% increase is significant. I'm wondering if this is incorrect/whether I have done something wrong? I am aware I may be confused about how statistical significance works, so just want to check whether the below makes sense!

The R outputs for estimates and p-values look like this:

                             Estimate   Pr(>|z|)
Level_of_IncomeLow:         -0.239043   0.0830 .      
Num_of_children              0.039527   7.54e-10 ***

My calculations:

Exponent of -0.239043: 0.787381
1-0.787381=0.212619
= 21.3% decrease

Exponent of 0.039527: 1.040319
= 4.03% increase

Essentially, is it okay that the larger effect is not statistically significant, but the smaller effect is? It just feels wrong!

Improve this question
$\endgroup$
7
  • $\begingroup$ Significance of a test depends on the ratio of the coefficient to its standard error (which you should also report in your analyses). The standard error is determined (i) the sample size, (ii) the variance of the residuals of the model, and (iii) the variation in the independent variables. Components (i) and (ii) affect the tests for both coefficients. However, it's possible that for (iii) there is more "relative" variation in the number of children in your sample than the level of income. $\endgroup$
    – ecnmetrician
    Jan 16, 2022 at 21:51
  • $\begingroup$ This means that it is easier to "detect" an effect. There could still be an income effect but the sample size is too low to detect it. $\endgroup$
    – ecnmetrician
    Jan 16, 2022 at 21:51
  • $\begingroup$ The fact that it depends on a ratio is extremely important. For example, if you rescale the variable, it affects both the numerator and denominator, and doesn't alter the significance. This makes significance a measure that doesn't depend on the units of measurement. $\endgroup$
    – ecnmetrician
    Jan 16, 2022 at 21:54
  • $\begingroup$ Thank you for all the helpful answers! I will also add standard error into my report $\endgroup$
    – James L
    Jan 16, 2022 at 23:32
  • 5
    $\begingroup$ You can't compare the size of coefficients that are in different units. There's no basis on which to call one coefficient "larger" or "smaller" when you're comparing incommensurable units. $\endgroup$
    – Glen_b
    Jan 17, 2022 at 0:46

2 Answers 2

Reset to default
5
$\begingroup$

low (vs. high) has a restricted range (i.e., only 2 values). Number of children presumably has a much wider range. The narrower the range of an $X$ variable, the less power there is to test it. That's a general principle. Specific to this example, it sounds like the underlying variable (income) was subjected to a median split, or something similar. If so, you should be aware that categorizing a continuous variable has long been considered poor statistical practice.

Improve this answer
$\endgroup$
4
$\begingroup$

Significance is not purely about the magnitude of an effect. For instance, if the parameter/effect is $1$ light-year or $9.46×10^{12}$ km, then it is different figures but the same distance.

(You might argue that this example is bad because I changed the units, but how are you gonna make sure that the units match when you compare 'pets per unit of income' with 'pets per number of kids'. When you change the currency then you change the effect size.)

Significance is an expression of the magnitude of an observed effect size in terms of how likely or unlikely it is to have an at least as strong deviation from the value predicted by some null hypothesis, given that the null hypothesis is true.

When this probability is low then you might consider the effect as an anomaly from the point of view of the null hypothesis. If you detect an effect with a small magnitude to be an anomaly, but an effect with a large magnitude is not, then it means that the experiment has different sensitivity for the two effects.

is it okay that the larger effect is not statistically significant, but the smaller effect is? It just feels wrong!

It depends on the relative effect size. Relative to the error of the estimate and this might be different for the two coefficients.

Below you see an example where the estimated intercept of 1.724 is different from 0 but it is not more significant than the estimated slope of 0.095 being different from 0.

The slope is a smaller effect size if you compare the plain figures like 0.095 < 1.724. But the slope has a much smaller variation from sample to sample, and such a small effect is significant.

example

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.