2
$\begingroup$

Weird situation:

We've been conducting a study for a stakeholder, and have collected data on all willing volunteers. They aren't satisfied with the sample size, and want us to collect more data - even on people who've already participated. They don't want us to turn it into a repeated measure design, they want us to treat restudied participants as an addition to the sample.

Is there a statistically valid way to do this? It flies in the face of what I was taught to do, but our stakeholder is adamant that we need to increase the n of the sample.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
2
$\begingroup$

They don't want us to turn it into a repeated measure design, they want us to treat restudied participants as an addition to the sample.

Uh, what?

If that isn't a design, then what is?

Honestly, this is repeated measures, and disregarding the repeated measurements just because a stakeholder doesn't like it is statistical misconduct, nothing else. (Sorry, I'm not putting the onus on you - I'm putting it on the stakeholder.)

I'm sure that we had a question before that essentially asked why it was not OK to just duplicate or repeat observations to artificially increase our $n$, but I can't find it now. It would be very relevant to your question.

Of course, you can learn a lot from a experiment. Maybe this is enough. Maybe it isn't. If the latter, then you and the stakeholder would best think about why you have already

collected data on all willing volunteers.

That is: why there are no additional willing volunteers. Maybe you need to search more, or offer additional incentives, to get more willing participants. (I am not advocating using unwilling ones.)

That said, if you change your recruitment parameters, I'd recommend modeling this. People who participate in your experiment for 10 USD might be systematically different from people who will only do so for 500 USD. It's a good idea to spend one degree of freedom here, even if this will increase your required sample size even more.

In addition, just why does the stakeholder want to increase the sample size? Is it because the results so far are not to her or his liking? Would she or he be happy and not require additional measurements if the results so far had been "better"? If so, she or he is requesting what is sometimes called "optional stopping". And continuing to collect data until they show some preconceived result is of course another instance of statistical misconduct.

(Sorry for appearing smug about this from the safety of my computer. You have a bad situation on your hands. I'd advise avoiding this stakeholder in the future if you can reasonably do so.)

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thanks for the info! Honestly, I wish I knew why they want more data. They called the data "stale" and requested more. We're already paying participants close to 1000 USD so I'm not sure what we can do to get new people. $\endgroup$
    – dmacfour
    Commented Nov 3, 2017 at 0:26
  • $\begingroup$ Honestly this smells of p-hacking and other bad practices. $\endgroup$
    – Adam B
    Commented Nov 3, 2017 at 8:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.