0
$\begingroup$

Let's say I'm trying to measure the effect of a marketing campaign on dwell time (length of stay at a location).

Why not just use a simple year over year calculation to measure the effect? E.g. 20% increase in time from last year. Assuming other main factors such as weather/seasonality/day of the month affect dwell time, wouldn't this calculation take this into account since it roughly maintains these variables constant?

What drawbacks does this approach have? Why should I use a more complicated method such as modeling with casual inference? In essence, why should I go down the road of time-series analysis instead of just comparing original data from the same period each year?

$\endgroup$

closed as unclear what you're asking by Nick Cox, John, usεr11852, mdewey, gung Oct 29 '16 at 21:45

Please clarify your specific problem or add additional details to highlight exactly what you need. As it's currently written, it’s hard to tell exactly what you're asking. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.

  • $\begingroup$ Well, you can. Moreover, no one approach in statistics is always best, try several and see which works best. $\endgroup$ – Carl Oct 24 '16 at 22:37
  • 1
    $\begingroup$ Casual $\ne$ causal. Affect $\ne$ effect. A more serious problem is that it's far from clear precisely what your alternatives mean. $\endgroup$ – Nick Cox Oct 28 '16 at 20:39
  • $\begingroup$ Sorry, I made an attempt to change casual to causal but after reading the question I am not sure what it's asking, I reverted my edit. $\endgroup$ – Penguin_Knight Oct 28 '16 at 20:41
1
$\begingroup$

Sometimes simple A/B baselining works just fine (also called pre-post analysis), but there are very specific conditions that need to be met. If there are other confounding variables (long-term trends, economic indicators, seasonality, etc.), you can draw bad conclusions from single-variable analysis of this sort.

As long as you have some reasonable assumptions that you can hold all other variables constant, year-over-year can be a perfectly valid comparison. If, however, you tried to compare summer of this year to winter of previous, you have the confounding variable of weather/seasonality which may influence your results deleteriously. Likewise, if you ALSO changed prices or had a new competitor come in to the market, or a bed bug infestation, or what have you, "all else equal" doesn't hold anymore. The need to control for the influence of multiple variables is what causes people to turn to more complicated modeling approaches.

Most "AB Test" or "Control Focus" methodologies attempt to measure the influence of a single variable in the fashion you describe (although there are multivariate AB tests too - this just involves using multivariate regression across multiple trial variables). You just need to satisfy yourself that you really can hold all other significant factors "reasonably" constant over the pre-post intervals (or be able to account for the expected variability due to the inability to control for all potential confounding factors).

Control-focus tests typically designate specially-selected control groups that are held constant over the variable in question specifically to allow differential analysis versus the test group with the assumption (usually supported by historical trending analysis) that non-test factors impact the sets equally (or at least within well-defined statistical tolerance). This approach makes impact measurement more robust against external influencing factors not directly tied to the variable being tested.

$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.