6
$\begingroup$

I ran a multivariate logistic regression with glm in R with some continuous and some categorical variables. Only continuous variable $A$ showed a p-value of < 0.05 and a confidence interval which did not stradle 1.

Running a Wilcoxon test (actually a Mann-Whitney test because the samples are not paired) with $A$ divided into the two outcomes groups returns a p-value of 0.15. This indicates that there is no difference between the means of $A$ in the two groups.

How do I reconcile these two results? The logistic regression result indicates that $A$ is a predictor of the outcome, but the Wilcoxon/Mann-Whitney indicates that there is no difference between the two groups.

$\endgroup$
4
  • $\begingroup$ What's the sample size? It's hard to base a decision of significance only based on p-value. $\endgroup$ May 9, 2011 at 21:43
  • $\begingroup$ Another concern here is about the Wilcoxon test (or the Mann-Whitney). I'm no experte on these tests, but I think they assume you are comparing two independent samples and test if their mean differ. Is that assumption valid in your case? $\endgroup$ May 9, 2011 at 21:45
  • $\begingroup$ @Manoel: Outcome 1 has 209 data points; outcome 2 has 72 data points. Yes, the samples are independent as they are either predictors for outcome 1 or outcome 2 (mutually exclusive). $\endgroup$
    – SabreWolfy
    May 10, 2011 at 6:22
  • $\begingroup$ You have a good sample, so your p-value seems ok. Take a look at my answer below to see if it helps you. $\endgroup$ May 10, 2011 at 16:13

5 Answers 5

6
$\begingroup$

Did you just fit one big glm model then look at the individual p-values?

Remember that those p-values are measuring the effect of that variable above and beyond all other variables in the model. It is possible that more of your covariates are really contributing, but there is redundant information, so they don't show significance. It could be the combination of A along with another covariate or 2 that shows the real difference, but A by itself is not meaningful.

Also look at the effect sizes and standard errors for all terms. There is a paradox associated with the Wald tests/estimates (the individual p-values in the standard summary) that you can have a very important variable appear to be non-significant because the standard error is over estimated. Likelihood ratio tests are much better because of this.

You can assess both the above by fitting a reduced model using only A, then use the anova function to compare the 2 models, if that is significant then it indicates that there is something important in your model beyond the A variable.

$\endgroup$
1
  • $\begingroup$ Yes, I included all variables in the model and looked at the p-values of each variable. $\endgroup$
    – SabreWolfy
    May 10, 2011 at 8:37
3
$\begingroup$

The tests make different assumptions, and so do not give exactly the same result. The bigger problem is the (incorrect) assumption that failure to reject the null "indicates that there is no difference". It does not. It just means that you don't have enough evidence to reject the null of no difference.

$\endgroup$
1
  • 1
    $\begingroup$ Thanks. Although either way I am not able to report, based on the Wilcoxon/Mann-Whitney, that there is a difference between the means of the two groups. $\endgroup$
    – SabreWolfy
    May 9, 2011 at 17:43
2
$\begingroup$

The more appropriate comparator is the proportional odds model, which contains the Wilcoxon test as a special case. And note that 'multivariate' refers to the simultaneous analysis of more than one dependent variable. I think you meant to say 'multivariable'.

$\endgroup$
1
$\begingroup$

If I understand things correctly, you're testing if the mean value of predictor $A$ associated with outcome $1$ differ from the mean value of predictor A associated with outcome $2$. Even if they don't differ, this result says nothing to your research question. What it says is only that in your sample, the average value of the predictor $A$ is'n different among succes and failures of the dependent variable. I'll give an example so you can see why.

Imagine that i'm interesting to know if education affects the probability of an employee being manager in a given firm. I collect a random sample of employees, with $200$ employes that are not managers and $70$ which are actually manager. I ran a logistic regression and the effect of education is significant (usual caveats about p-value apply here as well).

Now, you may wonder if the sample is balanced, which is, if there is the same variation in education for managers and non-managers. One first test to see this is to test if their means differ. If their means don't differ, than you know that on average on your sample, managers and non-managers have the same level of education. You could check if there is roughly the same variation of educational level among managers and non-managers. But I think that it's pretty clear that the Wilcoxon test (or the Mann-Whitney) has nothing to say about the effect of your predicton on the probability of success (in my example, success means to be a manager).

$\endgroup$
3
  • $\begingroup$ Thanks for the explanation (typo: p-valor). How can education be a good predictor if "their means don't differ" and "on average [...] managers and non-managers have the same level of education". Surely this means that the distribution of education is the same in the two groups. If that is the case, how can education be a good predictor if it is the same in the two groups? $\endgroup$
    – SabreWolfy
    May 10, 2011 at 17:34
  • $\begingroup$ One possibility is because of outliers (skewed distribution). In my example, imagine that there are a few outliers, with a lot of education, but that aren't managers (they are CEO, whatever). This would push up the average educational level of non-managers, but most of non-managers would have low educational level. $\endgroup$ May 10, 2011 at 20:42
  • $\begingroup$ By the way, I guess that in the context of one single predictor, logistic regression and ANOVA would produce the same result. $\endgroup$ May 10, 2011 at 20:43
0
$\begingroup$

The p-value that you got for the parametric test (from glm) depends on the effect size, i.e., by how much can the predicted values change depending on the value of A. Now if a small proportion (that proportion may be dependent on other predictors) of predicted values change by a large quantity due to the variable A, you have a large effect size and significant correlation and significant p-value. The rank-sum test, however only cares about the effect size in as much as it changes the ranking. If you were to push all of the overall highest 10 values (regardless of their class) by a large number in the absolute scale so that the rankings remain the same, the rank sum statistic will not change while parametric tests will. If you have reason to believe the effect size or the distribution you assumed (probably normal), then use the p-value from the parametric test, otherwise use rank-sum.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.