1
$\begingroup$

I am looking for example scientific papers in which association works (p-value < 0.05) in a univariate setting but is not significant in multivariable analysis due to low sample size (p-value > 0.05). Moreover, the researchers should have (for example) a biological explanation of why they think it would be independently associated with a greater sample size.

$\endgroup$
1
  • 3
    $\begingroup$ It looks like this could reasonably be made community wiki. $\endgroup$ Jan 24 at 8:14

2 Answers 2

3
$\begingroup$

You are comparing two different things: Univariate analysis assesses marginal distribution, whereas multivariate analysis assesses conditional distribution. They answer different questions. Both analyses only give association estimates and do not necessarily reveal causal relationship. To understand common techniques in causal inference, see my answer to Seeking Assistance in Evaluating My Research Plan for Regression Analysis. For best practice related to p value, see my answer to Justification for reporting non-significant effect sizes.

The p value is the conditional probability of observing more extreme outcomes conditional on that the null hypothesis (e.g., coefficient is zero) is true. This condition or hypothesis is different in univariate (direct effects of other covariates are not separated) and multivariate (direct effects of other covariates are controlled for) analyses. The size of a p value is affected by several factors, and sample size is only one of them. For specific p values in different models, it is impossible to tell apart if the significance disparity is due to the sample size or something else (e.g. effect size, power of a statistical test, conditions, model specification, etc.). If the sample size keeps increasing, any given magnitude of difference or association can eventually become significant. Therefore, in a well-designed experiment, increasing the planned sample size may undermine the study by reducing the p value and inflating the Type-I error. There is also a matter of multiple testing when many variables are tested one after another.

I disagree with the statement that Mikolaj Buchwald gave.

Usually in practical settings, the more variables you have the more probable it is that you will find the relationship between observed change and (quasi)experimental intervention

It is possible to observe nonsignificant relationship in univariate analysis but significant ones in multivariate analysis, as they answer different questions. Therefore, controlling more covariates may not reduce the p value of the treatment indicator. There are some covariates that should not be controlled given an experimental design. In practice, picking between univariate and multivariate analysis is in essence variable selection and model selection. For confirmatory studies, theory and biological mechanisms should guide this process. For exploratory studies, many techniques are useful.

More materials than listed in my other to answers:

$\endgroup$
1
  • 3
    $\begingroup$ In general, and especially with regression models that are not regular linear models, only one of uni variable and multivariable analyses can be correct. The study design will dictate which one should be used (it’s usually the multivariable one). Comparisons of the two are less useful than they seem. $\endgroup$ Feb 3 at 12:41
1
$\begingroup$

It would be interesting to see a simulation meeting your criteria. Usually in practical settings, the more variables you have the more probable it is that you will find the relationship between observed change and (quasi)experimental intervention -- it is due to the fact that in biology (and in the physical world in general) usually the observed (and measured) phenomenon is influenced by multiple factors, not a single one. (I mean: weakly/moderately by multiple factors, not strongly by a single one.)

There are some reports in which for univariate analysis more variables turned out to be statistically significant than in the multivariate analysis:

https://onlinelibrary.wiley.com/doi/abs/10.1002/jso.2930510208

https://www.sciencedirect.com/science/article/pii/S0302283802004931

https://www.sciencedirect.com/science/article/abs/pii/S0959804905802928

However, you would have to check if in any of these papers this effect was attributed to the insufficient sample size for the multivariate analysis. In other words, what would happen with the significance of the univariate outcomes if the sample size was bigger? Would it not increase too? As I was saying, in my area of research usually even for the smaller sample sizes multivariate analysis just works better.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.