So I have recently been debating two possibilities for determining the effectiveness of, let us say for simplicity, a single marketing strategy for getting return customers.

Let us assume that we have years of historical data where a company has, at certain uncontrolled times and at uncontrolled store locations, marketed to in-store customers in an attempt to get them to return (let's say they give out pamphlets at the check-out register, for example).

My question is: What's a more effective strategy.

  1. Using the historical data from times in the past when the store has marketed to try to estimate the effectiveness of the marketing to get customers to return. (e.g. marketed customers are 50% more likely to return in a week)

  2. Using a designed experiment to give some randomly selected customers pamphlets and do not give them to the rest (control). Perhaps do this for a period of time (maybe a week) and track them for a while (maybe a month) to estimate the effectiveness of the marketing to get customers to return.

The argument for option 1 is that you have much more data; however, you have confounding variables and no obvious control group to compare what would have happened if no marketing was done.

The argument for option 2 is that you essentially have a Randomized Controlled Experiment; however, you have less data which takes time (money) to gather in a controlled fashion.

I understand there may not be a concrete answer, but is there any theory or practical statistical reasoning that suggests when one or the other option is better.

Just as a note: assume customers can be tracked for return visits and try to avoid arguments based solely on logistical issues (unless you feel they are very important).

  • 2
    $\begingroup$ You bring up the same considerations/criteria that I would. These already reflect the "theory or practical statistical reasoning" that I would introduce. Now for a definitive decision I would need to know just how great is the difference in sample size, as well as some of the logistical issues that you would rather avoid discussing :-) At any rate I think your question boils down to "please help me decide" rather than "please tell me what is true" or "please tell me how things work." So to get more productive answers you may want to change it around. $\endgroup$
    – rolando2
    Commented May 2, 2014 at 11:10
  • 1
    $\begingroup$ Perhaps practical experience or knowledge of a situation in which the later or former is used for example would be beneficial to me. Perhaps even what should go into the justification process. $\endgroup$
    – Underminer
    Commented May 2, 2014 at 16:14

1 Answer 1


"More data" is good only if the data come from the same Data Generating Process, (DGP). Can the customers from years back be considered to be part of the same DGP as recent customers, at least approximately? This would mean for example asking the question, is the composition of customers the same (say in terms of age, sex, year, education, socioeconomic status)? Or also, can you say with some confidence that even if the composition is roughly the same, consumer preferences regarding the company's products have remained relatively unchanged through the years? You may not have the appropriate data to test the above in a statistically proper manner -but at least, you should give the issue some thought, and use whatever experience and memory resides in the company about how its customer base has evolved over the years. Sometimes changes make the past irrelevant -and worse, misleading.

But assume that the questions above are answered in the affirmative, so yes you have a lot of data from the same DGP, and so you are still facing your dilemma.

In such a case, the issue about the absence of a "control group" is rather watered down: we just accepted that "customers yesterday" are the same (statistically) as "customers today". So if we performed a marketing act on "customers yesterday", then our control group is "customers today" -on whom we did not perform a marketing act.

As for confounding variables, you can disaggregate your (large) data sample, and see whether the responses to marketing activities per sub-sample are roughly the same or not (the reasoning is obvious).

If you can accept that the big historical data are comparable, use them first since it will be cheaper (and of course costs factor in, in a business activity). When humans are involved, I am biased in favor of using the larger amount of data (subject to the initial qualification). It is too difficult to properly design and successfully execute a Randomized Controlled Experiment with human subjects, without having just tricked yourself in believing that all shorts of biases and unwanted influences have not crept in. Large data samples dehumanize the situation relatively more, which is as it should be for good quality statistical inference.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.