Design of experiment in multiple regression

Question

Suppose we have the following model of our environment:

$\hat{y}_t = e^{dayofweekeffect} * x_{1, t}^{\beta_0} * x_{2, t}^{\beta_1}$

which we can linearize into: $log(\hat{y}_t)= dayofweekeffect + \beta_0 * log(x_{1, t}) + \beta_1 * log(x_{2, t})$

To imagine this, lets assume that $y$ consists of daily sales and $x_{1, t}, x_{2, t}$ different daily investments, lets assume promotional activitiy expenditure. Our investments are not the sole driver behind sales, the baseline sales is modelled by $dayofweekeffect$. We can imagine that the $dayofweekeffect$ is modelled by fourier terms or dummies. Notice that we dont "control" for the dayofweekeffect in the sense that we have an proxy for dayofweekeffect but instead we use dummies or fourier terms and models it "simultaneously". We could also imagine some structural time series model trying to decompose the effects.

---

The dataset is problematic since the previous analyst, did not consider the baseline sales but instead used an model $log(\hat{y}_t) = \beta_{dayofweekeffect_0} * log(x_{1, t}) + \beta_{dayofweekeffect_1} * log(x_{2, t})$ and used this model to optimize expenditure.

This means that he essentially created datapoints for $x_{1, t}$ in which he spent less when the $dayofweekeffect$ was lower and more when the $dayofweekeffect$ was higher.

---

My goal is to design an experiment(DOE) such that we can efficiently disentangle the $dayofweekeffect$ from the effect of $x_{1, t}$ but i need some guidance on which literature/design criterions i should further investigate.

Question What would be an appropiate design criteria for this scenario?

---

Also notice, that we are not sure how well he optimized the previous expenditure, there might not be an perfect collinearity between the unknown $dayofweek$-demand and $x_{1, t}$ thus it could be useful to also factor in the previous dataset in some way.

Answer 1

I would randomly split your stores (or groups of stores if they are close enough to worry about spillovers) into three groups: marketing spend as usual, reduce spend, and increase spend. If you have two marketing channels, then you would have 9 groups. This can keep your marketing spend roughly constant while allowing you to learn about the average effectiveness of the change in spend. You can also take one group to zero if you want to estimate a zero-spend baseline. You can also do a two-cell experiment if that more closely matches your goal. You can also vary spend more continuously, but that is often tough to do from a setup perspective and may not work well if there are carry-over effects from day to day. The advantage is that it lets you trace out a revenue-spend curve/surface instead of just 3 points on it.

Then run a regression like:

$$sales_{it}=\alpha + \gamma_{Mon} + \gamma_{Tue} + ... \gamma_{Sat} + \delta_{Up} + \delta_{Down} + \eta_{Mon \cdot Up} + \eta_{Mon \cdot Down} +... + \eta_{Sat \cdot Up} + \eta_{Sat \cdot Down} + \varepsilon_{it} $$

This data is at store and day level. The omitted group is Sunday and normal spend, so all the effects are relative to that. You can even add a store fixed effect to pick up differences between store trade areas that are constant over time.

If you had two channels, you would have more terms and can also include interactions between channels.

You can log sales or just use a Poisson model with robust standard errors. If you randomize over groups of stores, you should cluster your SEs by group.

You can do minimum reliably detectable effects using simulation on historical data. This should work well unless you have strong seasonality and are going into a slow period. Larger changes should be easier to detect.