6
$\begingroup$

I have a multinomial logistic regression with dependent variable valued in {-1,0,1} (reference category is 0) and a number of continuous and discrete predictors. After running the regression a continuous predictor of interest ('size') has a Type 3 analysis of effects p-value of 0.0683, and the two coefficients (corresponding to outcomes of -1 and 1) have p-values of 0.8786 and 0.0220 respectively.

I read somewhere that one should only look at the significance of the coefficients if the predictor itself is significant at the chosen level. Is this right? My naive sense is that the predictor is borderline (taking alpha=0.05 for argument's sake), and that 'size' has a significant relationship to outcome=1 but not to outcome = -1. I would say that the significance of the relationship to outcome=1 is not terribly strong, but that is ok for the application in mind (or at least, with the indirect data I am forced to use)

Improve this question
$\endgroup$
1
  • 2
    $\begingroup$ You might more detail about what your goals are. In general, like @rolando2 I encourage emphasizing predictions and confidence intervals rather than hypothesis tests. $\endgroup$
    – Michael Bishop
    Commented Mar 27, 2011 at 4:44

1 Answer 1

Reset to default
5
$\begingroup$

The p-value itself cannot tell you how strong the relationship is, because the p-value is so influenced by sample size, among other things. But assuming your N is something on the order of 100-150, I'd say there's a reasonably strong effect involving Size whereby as Size increases, the log of the odds of Y being 1 is notably different from the log of the odds of Y being 0. As you indicate, the same cannot be said of the comparison of Y values of -1 and 0.

You are right in viewing all of this as somewhat invalidated by the overall nonsignificance of Size (depending on your alpha, or criterion for significance). You wouldn't get too many arguments if you simply declared Size a nonfactor due to its high p-value. But then again, if your N is sufficiently small--perhaps below 80 or 100--then your design affords low power for detecting effects, and you might make a case for taking seriously the specific effect that managed to show up anyway.

A way around the problem of relying on p-values involves two steps. First, decide what range of odds ratios would constitute an effect worth bothering with, or worth calling substantial. (The trick there is in being facile enough with odds to recognize what they mean for the more intuitive metric of probability.) Then construct a confidence interval for the odds ratio associated with each coefficient and consider it in light of your hypothetical range. Regardless of statistical significance, does the effect have practical significance?

Improve this answer
$\endgroup$
3
  • 1
    $\begingroup$ In that last paragraph aren't you effectively arguing against any use of p-values? Why, then, use a hypothesis test at all? Why not just estimate the effect sizes and be done with it? $\endgroup$
    – whuber ♦
    Commented Feb 25, 2011 at 6:18
  • 1
    $\begingroup$ I stop short of calling p-values completely useless. I do advocate for the use of other methods, esp. when I hear people struggling with hypothesis testing and when they seem to have been taught such testing in the flawed way that I and many of us seem to have been taught. I've commented on this elsewhere on the site so I won't repeat too much of my views here. $\endgroup$
    – rolando2
    Commented Feb 25, 2011 at 16:04
  • 1
    $\begingroup$ @rolando2 - p-values are not "useless", but its more that this uncritical use of 5% as some sort of global cut-off is what is wrong. If used properly (i.e. based on sufficient statistics, varying the significance level with the sample size and with the relative costs of false rejection and false acceptance) p-values are quite an efficient way to communicate your results. $\endgroup$
    – probabilityislogic
    Commented Jul 18, 2011 at 16:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.