# Given two responses for two groups, how to decide what to test on response or profit?

For many years I have been conducting t-tests on response to mailing activity. Recently I was challenged that we should infact be conducting tests on profit rather than response.

So, let me put this in context. If you have two groups of customers of sample size 10,000 each that you were mailing two different offers. One receives free delivery (group a) on goods and one receives 25% off their next purchase (group b). And assume the two groups responded as follows.

• Group a - 10% of the 10,000 customers mailed responded
• Group b - 15% of the 10,000 customers mailed responded

I would conduct a t-test on the response rates (10% and 15%), which is correct.

I have been asked to do a test on the profitability, which i am reluctant to do. Again to put this into context:

• Group a - generated a profit of £1 per customer mailed (i.e. £10,000)
• Group b - generated a profit of £1.50 per customer mailed (i.e. £15,000)

I have been asked to do a test on the £1 and £1.50 profit, but my concern is that there are many variable factors that contribute to the final profit figures, eg, margins, cost of the mailing the customer received etc, that are not really considered when setting the groups up.

Am I correct in thinking that for this particular scenario that testing on profit is not the right thing to do, instead test on response as we are currently doing and then factor in all the other variable costs when making a final commercial decision?

• I've changed the title so that statistical problem is immediately apparent. If you (or anyone else) think that the original meaning was lost, please change it back. Apr 20, 2011 at 10:23
• Do you mean profit or revenue ? If you mean profit there should be no additional costs. If you mean revenue, I see no problem on reporting back to a decision layer whether one group outperforms the other one significantly (regarding revenue). Calculating the costs (which includes you performing the analysis ;)) is not your job, isn't it ? Apr 20, 2011 at 12:05
• Thanks for your response. I would traditionally do a test on either variance on reponse generated or revenue (which does not include additional costs) but have been asked to do tests on profit, which does include all the additional costs, as these costs are variable depending on what each group receives i find that the tests can already be biased before we start. Hope that makes sense?
– user4248
Apr 20, 2011 at 12:26
• No, why should they. If the experiment is randomized (i.e. customers have selected randomly for mailing A or B and not dependent on costs), then the costs should have no effect. Apr 20, 2011 at 14:35