For many years I have been conducting t-tests on response to mailing activity. Recently I was challenged that we should infact be conducting tests on profit rather than response.
So, let me put this in context. If you have two groups of customers of sample size 10,000 each that you were mailing two different offers. One receives free delivery (group a) on goods and one receives 25% off their next purchase (group b). And assume the two groups responded as follows.
- Group a - 10% of the 10,000 customers mailed responded
- Group b - 15% of the 10,000 customers mailed responded
I would conduct a t-test on the response rates (10% and 15%), which is correct.
I have been asked to do a test on the profitability, which i am reluctant to do. Again to put this into context:
- Group a - generated a profit of £1 per customer mailed (i.e. £10,000)
- Group b - generated a profit of £1.50 per customer mailed (i.e. £15,000)
I have been asked to do a test on the £1 and £1.50 profit, but my concern is that there are many variable factors that contribute to the final profit figures, eg, margins, cost of the mailing the customer received etc, that are not really considered when setting the groups up.
Am I correct in thinking that for this particular scenario that testing on profit is not the right thing to do, instead test on response as we are currently doing and then factor in all the other variable costs when making a final commercial decision?