4
$\begingroup$

I am hoping someone can either validate something I think is wrong or explain to me why it is not.

Here is the situation

  1. On a marketing test a given selection criteria for customers was tested, and a random control group was kept (of all customers who qualified, a random was sample was the control). The test group was called and the control was not.
  2. a given response was measures (purchase or not) and statistics aggregated
  3. an hypothesis test was performed to see if calling increase the purchase rate (2 proportion test); however
  4. The person who did this multiplied the size of the control group and the number of responders for the control by a factor - the factor was roughly the one required so that the test group was the same size as the control group. Then they performed the test and reported the result

My position is that this is incorrect, you can't adjust the size of the control after the test. They could have kept a larger control of course. I think the problem was the Ns were too small and hence the result were not significant and they really wanted to show them to be significant

Is there a logical statistical explanation that makes this valid? I do not believe so as if multiplying by say 5 does not work you can multiply by 10 or 100. My best guess is they were trying to do something like oversampling the control group

Thanks in advance

$\endgroup$
4
  • 1
    $\begingroup$ When you say "multiplied the size" do you mean they simply used a different $n$ in the calculations to the $n$ that was actually sampled? Or do you mean they went back and sampled some more to increase $n$? (as ghonke appears to think). Both are wrong, but one is - in some sense at least - a substantially greater level of fakery and flim-flam than the other. $\endgroup$
    – Glen_b
    Feb 4 '15 at 0:03
  • $\begingroup$ Yes say the control group was N=100 and had 5 responders, they multiplied by say 5 and use N=500 and 25 responders and they they did the test with whatever the test had $\endgroup$ Feb 4 '15 at 0:41
  • $\begingroup$ If they genuinely think it's a good idea to do that, a simulation might be the best way to convince them otherwise. $\endgroup$ Feb 4 '15 at 19:27
  • $\begingroup$ agree, i actually built something quick for using coin tosses to make sure I was no missing anything $\endgroup$ Feb 5 '15 at 1:22
2
$\begingroup$

I agree with your conclusion. This is akin to p-hacking, adding additional data points until the difference comes out and is often brought up as QRP - questionable research practice

The primary reason not to do this is that as a sample grows larger it will (likely) regress to the mean. This is not the case if you are simply multiplying an existing sample thus keeping the mean constant

This happens way too often unfortunately, check out point 2 in table 1 from the paper below for an idea of how often this is done. Keep in mind in this case the researchers are gathering more unique data and it is STILL inappropriate https://www.cmu.edu/dietrich/sds/docs/loewenstein/MeasPrevalQuestTruthTelling.pdf

$\endgroup$
2
  • $\begingroup$ Thanks - this is just marketing, they are not doing anything other than making them believe it works when it may not and maybe waisting their budget $\endgroup$ Feb 3 '15 at 21:48
  • $\begingroup$ yea, they might be interested in an approach like multiple imputation or expectation maximization to generate more data based on the data you have but with added noise $\endgroup$
    – ghonke
    Feb 3 '15 at 22:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.