For many years I have been conducting t-tests on response to mailing activity. Recently I was challenged that we should infact be conducting tests on profit rather than response.

So, let me put this in context. If you have two groups of customers of sample size 10,000 each that you were mailing two different offers. One receives free delivery (group a) on goods and one receives 25% off their next purchase (group b). And assume the two groups responded as follows.

  • Group a - 10% of the 10,000 customers mailed responded
  • Group b - 15% of the 10,000 customers mailed responded

I would conduct a t-test on the response rates (10% and 15%), which is correct.

I have been asked to do a test on the profitability, which i am reluctant to do. Again to put this into context:

  • Group a - generated a profit of £1 per customer mailed (i.e. £10,000)
  • Group b - generated a profit of £1.50 per customer mailed (i.e. £15,000)

I have been asked to do a test on the £1 and £1.50 profit, but my concern is that there are many variable factors that contribute to the final profit figures, eg, margins, cost of the mailing the customer received etc, that are not really considered when setting the groups up.

Am I correct in thinking that for this particular scenario that testing on profit is not the right thing to do, instead test on response as we are currently doing and then factor in all the other variable costs when making a final commercial decision?

  • $\begingroup$ I've changed the title so that statistical problem is immediately apparent. If you (or anyone else) think that the original meaning was lost, please change it back. $\endgroup$
    – mpiktas
    Apr 20, 2011 at 10:23
  • $\begingroup$ Do you mean profit or revenue ? If you mean profit there should be no additional costs. If you mean revenue, I see no problem on reporting back to a decision layer whether one group outperforms the other one significantly (regarding revenue). Calculating the costs (which includes you performing the analysis ;)) is not your job, isn't it ? $\endgroup$
    – steffen
    Apr 20, 2011 at 12:05
  • $\begingroup$ Thanks for your response. I would traditionally do a test on either variance on reponse generated or revenue (which does not include additional costs) but have been asked to do tests on profit, which does include all the additional costs, as these costs are variable depending on what each group receives i find that the tests can already be biased before we start. Hope that makes sense? $\endgroup$
    – user4248
    Apr 20, 2011 at 12:26
  • $\begingroup$ No, why should they. If the experiment is randomized (i.e. customers have selected randomly for mailing A or B and not dependent on costs), then the costs should have no effect. $\endgroup$
    – steffen
    Apr 20, 2011 at 14:35

1 Answer 1


The reason why you are conducting this test is to determine which policy is more valuable, and if value is measured in profitability, then it makes no sense to do statistical testing on any other variable. A properly conducted test on profitability gives you all the information needed for your companies' decision: once you have the results of this test, results about other variables (e.g. response rate) provide no decision-relevant information.

  • $\begingroup$ The selection is completely randomised, but the costs associated to each group may be different. So even if there was not any difference in response or spend we would report a significant difference which is purely becasue one group had higher costs associated to it, which doesn't help us identify if there was any change in behaviour. $\endgroup$
    – user4248
    Apr 20, 2011 at 16:33
  • 1
    $\begingroup$ The start of this reply looks good but I got lost in the second half, perhaps due to some lapses of grammar and the long sentence. Could you clarify this part? $\endgroup$
    – whuber
    Apr 20, 2011 at 16:39
  • $\begingroup$ @Nil : exactly! Wouldn't you want to know that you can achieve the same result at less cost? $\endgroup$
    – Aniko
    Apr 20, 2011 at 16:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.