# Comparing effects of different methods when each method has multiple levels

I'm working in retail and we are trying to determine the effect (on units sold) of reducing an expense for a set of products within a product group.

We have two methods of reaching this goal. One is to reduce the number of products bearing this expense, the other is to reduce the amount of expense on each product. We do not expect to use both methods at the same time.

We are also looking to do each method at different levels, an overall control group, a 50% reduction, and a 100% reduction.

So we have:

Level        Method1 Method2
Control        0%      0%
1             50%     50%
2            100%    100%


We also have a number of covariates we are trying to control for, but thankfully few have interactons with the treatment. (The overall test would be conducted all at once, with one set of control stores used to compare against both treatments).

Importantly we want to know if M1,50% is different than M2,50% and M1,100% is different than M2,100% as well as M1,50% vs M2,100%.

I've thought about using some dummy variables and creating one model:

 y = Intercept + Method + Treatment + Method * Treatment + Covariates


However, I'm unsure how to interpret the results at the mid-level area between treatments. Especially, whether the difference in coefficients are statistically different from each other.

At this point I'm wanting to create two identical models for each method

 y(m1)= Intercept + Treatment(m1) + Covariates
y(m2)= Intercept + Treatment(m2) + Covariates


The test will contain the same number of stores at each level, and the covariates would not change. Comparing the coefficients using the formula:

(b1-b1)/sqrt(seb1^2 + seb2^2)


I'm not positive which is the better route to go, and which are the pitfalls with each one.

Any help or guidance would be greatly appreciated.

## 1 Answer

I would go with the first choice you have, where you are modeling them together. You can think about the dosage as a separate variable that is interacted, or you can think about them as 4 different treatments as such:

$Q=\beta_0+\beta_1 M1_{50} + \beta_2 M1_{100} + \beta_3 M2_{50} + \beta_4 M2_{100} + \Sigma \gamma X$

Where X is your covariates. Then you can compare the coefficients to see which has the greatest effect on your sample. Even though you have a linear model and different variances of the two samples shouldn't affect the results (in nonlinear models this is a problem, see here), conceptually it is more satisfying to me to model them all together. And the extent to which you have sample size limitations, pooling the samples could help with estimation and hypothesis testing.

• Thanks for the reply! So, my question is then how do I determine that B1(M150) is significantly different than B2(M150), which is where I've gotten hung up before in thinking on this problem. The examples I've seen on this problem basically are testing a binary set and if the beta is significant, then the difference between treatments is significant. Since I have multiple Treatments, I'm not sure how to check the statistical significance of their difference. – Robert Richardson Dec 10 '14 at 19:31
• Use a Wald test to compare the coefficients. In Stata, for example, after your reg command type: test m250=m150 – robin.datadrivers Dec 10 '14 at 20:53