# How can I compare the relative effects between values of a categorical IV?

I am a social science phd student trying to figure out a statistical test for small conference presentation I am working on. However, I realized I don't know how to run the model I want. Some background on my question:

DV: Change in a person's weight (continuous variable)

IV: Which fruit someone ate for breakfast (categorical variable with four categories: Guava, raspberries, Pears, and strawberries)

Statistical Method: Simple OLS Regression

According to my theory, eating guavas should have the greatest effect on someone's weight compared to the other fruit, then raspberries, then pears, and with strawberries having the least effect on someone's weight.

To test my theory, my original plan was to run a simple OLS regression with 3 binary variables (one for each fruit, with strawberries being the baseline). However, I just realized that that sort of model would only tell me if each fruit had a greater effect than strawberries, but it would not tell me that the effect of guava was greater than raspberries, or if raspberries had a greater effect than pears, etc...

Is there a simple method in an OLS regression (in STATA) that would allow me to test if guava's had the greatest effect, raspberries had a second greatest effect, pears had the third greatest effect, and strawberries had the least effect?

Thanks!

FYI: No my paper is not really about fruit and weight loss. Those variables were made up for the purposes of this question.

• IV typically refers to instrumental variables so it would have been better to use a different wording. – Alex Aug 23 '11 at 12:59
• @Alex, it may econometrics. It does not in sociology. The term "independent variable" is terrible by itself, though, as nothing is ever independent in social sciences. I always say, "explanatory variables". – StasK Aug 23 '11 at 13:57

You are in a rather special situation of one-sided testing. Assuming your dummy coding is such that guava is the reference category, you have $H_0: \beta_1 \le 0, \ldots, \beta_k \le 0$ vs. $H_1:$ at least one of $\beta_1, \ldots, \beta_k >0$. You can use the regular one-sided test, but it will be extremely conservative (more so with more dimensions). The proper distributions are non-standard: they are mixtures of $\chi^2$ or $F$-distributions with varying degrees of freedom. Stata mentions the conservativess of this test in a context where it most frequently arises: testing variance components in mixed models, see help j_xtmixedlr.

Even though the problem was first addressed more than fifty years ago (Chernoff (1954), Bartholomew (1961)), it is still a relatively esoteric piece of statistical theory. If you can handle Econometrica articles, Andrews (2001) will be very helpful. The textbook length treatment is given by Silvapulle & Sen (2004). I am not aware of a less technical discussion, though. Vika Savalei and I tried to retell the story for psychologists (Psych Methods 2008), you might find it somewhat more accessible. In particular, we discuss conditional inference that is easier to understand and apply.

You don't need to do regression here. You can equivalently do an ANOVA to test fruit, overall, as a predictor then you can do a series of $t$-tests for the individual levels of the predictor (adjusted appropriate for the multiple testing). This will tell you a) whether fruit is important, overall and b) which particular pairs of fruits are significantly different from each other in terms of their effect on weight loss.

If you insist on using regression (for example, if you're controlling for a number of other variables) then you can take the coefficient estimates for two particular levels, call them $\hat{\beta}_{1},\hat{\beta}_{2}$ and their standard errors $s_{1}, s_{2}$ and conduct a test of the hypothesis $\beta_{1} = \beta_{2}$. If the two samples from which the levels are collected are independent (i.e. that each person ate and only one of the fruits and the individuals are independent of each other) then the test statistic is

$$T = \frac{ \hat{\beta}_{1} - \hat{\beta}_{2} }{ \sqrt{ s_{1}^{2} + s_{2}^{2} } }$$

where the usual inference for the two-sample $t$-test applies. You could also do the test by collapsing the two groups into one and re-fitting the model, comparing the fits (i.e. looking at how much worse the fit gets when you collapse the groups, which you should be able to do under the null hypothesis) using the $F$-test or the likelihood ratio test (assuming normal errors) and repeating this for each pair of groups. The $t$-test is probably simpler.

Note 1: Since comparing the groups is usually called a "post-hoc" test that is done after it is determined that the grouping variable itself is significant, you'll definitely want to test the overall importance of "fruit" before you go ahead comparing levels. Also, since there are multiple pairs of levels, you should probably correct for multiple testing.

Note 2: If there is dependence between the samples (the groups that took each fruit) then you would need to know ${\rm cov}(\hat{\beta}_{1}, \hat{\beta}_{2})$, and the denominator would have to be corrected by subtracting $2{\rm cov}(\hat{\beta}_{1}, \hat{\beta}_{2})$ twice under the square root. If there is positive correlation between the samples, then the value $T$ above will actually be underestimated (since the denominator will be over-estimated), meaning that the independent test statistic $T$ will be conservative (i.e. a 'positive' result can be trusted, but the power will be lower than it needs to be).

• @Marco isn't ANOVA a (very) special case of regression? – suncoolsu Aug 22 '11 at 21:16
• Yes, it is. The ANOVA $p$-value corresponds to the $F$-test $p$-value comparing the model with the categorical predictor vs. the model without it (i.e. the means are the same across groups). – Macro Aug 22 '11 at 21:49