2
$\begingroup$

A company wants to run a promotional campaign to encourage usage of a credit card that the customer already has. The company wants to send this promotion to all credit card customers. It wants to split the group into a Control and Treatment. The Control group will not receive the promotion and the Treatment group will.
What the company wants to know is whether the spending behaviour of the Control group is different from Treatment group.

My question is if I am working with the entire customer base of this company, do the significance tests (t-test, ANOVA (if have have more groups)) actually make sense to perform?
I tried searching for an answer to this, but so far I don't think I found anything. I found a few things on superpopulation but I'm not sure if it applies to this scenario.
I also cannot really think of what would be a larger population in this case.
Could the larger population be those who are not our customers?
If anyone has any references regarding how to make this distinction between a sample and a population, that would be great.

$\endgroup$
3
  • 4
    $\begingroup$ Do you never hope to acquire new customers? Is your aim only to learn about existing customers or will you apply your findings to future customers? $\endgroup$
    – whuber
    Feb 22, 2018 at 15:09
  • $\begingroup$ Yes we do aim to get new customers and apply this knowledge to new customers. Is that how how I identify the population and therefore refer to the entire dataset as a sample? $\endgroup$
    – jmich738
    Feb 22, 2018 at 21:38
  • $\begingroup$ The sample consists of data you use to make inferences. The "population" refers to whatever might be the object of those inferences. This ought to make it apparent that the sample does not determine what the population might be: it is the questions you ask of the sample that determine the population. Going further, the choice of hypothesis tests depends on how the sample might be related to that population (which is described by a statistical model), as well as what you want to learn and what kinds of mistakes you can afford to make. $\endgroup$
    – whuber
    Feb 22, 2018 at 21:45

2 Answers 2

2
$\begingroup$

Standard errors are usually motivated by adjusting for sampling variation, based on viewing the data as a random sample drawn from a large population of interest, even in applications where it is difficult to articulate what that population of interest is and how it differs from the sample. This is the idea of a super-population that you brought up.

I don't think you actually have the full (super-)population for the following three reasons:

  1. You will have new customers (hopefully), as @whuber pointed out. Geographic expansion is another potential channel.
  2. You will have additional marketing opportunities in the future that you want to extrapolate to using results from today's experiment (even for the same set of customers).
  3. You observe treated outcomes for some percent of the users who are treated and untreated outcomes for the control users. Each user is missing potential outcomes for the treatment levels the unit was not exposed to. This means your data is incomplete since you are doing the experiments to learn about the mean of the random variable that is the treated outcome less the untreated outcome. This comes up even without new users or other time periods. In some sense, the fundamental problem of causal inference is that you have 1/2 of the data and there is nothing you can do about that.

SEs attempt to adjust for all that under some set of environment stability assumptions (future users are similar to current, the market conditions will be the same later, treatment has no permanent effect, the experiment was set up correctly, etc). There are some complications about whether the usual SEs accomplish this, or you need some special ones, that are beyond the scope of this answer, but worth keeping in mind.

$\endgroup$
0
$\begingroup$

This is a question I have struggled with over the years. Sadly, few people tend to answer these kinds of questions, perhaps because they don't want to jeopardize their careers or appear stupid.

While those who support the "population as a sample of a super-population" position would likely make what they see as internally-coherent arguments about why you could treat the entire population as a sample, I find it intellectually dishonest to define population data as equivalent to data acquired from random sampling or random assignment, which in both cases do not enumerate the entire population.

My position on this is rigidly axiomatic. If we don't have a random sample, we can't use inferential statistics. Inferential techniques assume that we draw a random sample to make inferences about the larger population from which the random sample is drawn. We then estimate the underlying population parameter from the random sample, to get a sample statistic. It is an estimate of the true population parameter. If we have enumerated the population, then we have population level data and don't need to rely on inferential statistics. We can calculate the population mean, true value of a regression coefficient, or the exact difference in means. I expect none of this is new to you, given you've asked your questions.

With that said, I'm happy to learn more. I'm not so old that I can't have my mind changed. I think @whuber makes an interesting point in respone to your question about using your current data as a way to understand the characteristics of potential customers in the future. If you have reason to believe that your market share is not 100%, and potential customers are similar to current customers, what he says makes intuitive sense.

Finally, did you consider using permutation approaches to generate pseudo p-values? If your employers/clients are hung up on the need for p-values, this might be a solution.

[Caveat 1: Unless we have so much data that it exceeds the computational power of our computers; this is the land of big data, where random samples are taken from the population, and these parameter estimates stand in for the true population mean. But I don't think that is the situation you are describing].

[Caveat 2: I am assuming there is no measurement error in any population data, which is problematic. I suspect that if the measurement error could be assumed to be random, one might make an argument that the enumerated population serves as a random sample of the "real" population. However, the underlying logic for that is beyond my patience and/or ken.]

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.