I've seen this kind of problem in survey-like and psychological experiments.

In one case, the entire experiment had to be chalked up to a learning experience. There were problems at multiple levels that resulted in a jumble of results, but results that seemed to give some support for the hypothesis. In the end, I was able to help plan a more rigorous experiment, which essentially had enough power to reject the hypothesis.

In the other case, I was handed a survey that had already been designed and executed, and there were multiple problems that resulted in several areas of interest being affected. In one key area, for example, they asked how many times the customers were turned away from an event due to it being full when they arrived. The problem is that there's no time range on the question so you couldn't tell the difference between someone who had tried to attend 4 times and been turned away 4 times and someone who had tried to attend 40 times and only been turned away 4 times.

I'm not a trained, capital-s Statistician, but if they'd come to me beforehand, I would have been able to help them fix these issues and get better results. In the first case, it still would have been a disappointing, "Sorry, your hypothesis seems extremely unlikely", but it could have saved them a second experiment. In the second case, it would have given them answers to some important questions and would have made the results sharper. (Another problem they had is that they surveyed multiple locations over time and at least some people were thus surveyed multiple times, with no question like "Have you taken this survey elsewhere?")

Does this apply to your question?

I've seen this kind of problem in survey-like and psychological experiments.

In one case, the entire experiment had to be chalked up to a learning experience. There were problems at multiple levels that resulted in a jumble of results, but results that seemed to give some support for the hypothesis. In the end, I was able to help plan a more rigorous experiment, which essentially had enough power to reject the hypothesis.

In the other case, I was handed a survey that had already been designed and executed, and there were multiple problems that resulted in several areas of interest being affected. In one key area, for example, they asked how many times the customers were turned away from an event due to it being full when they arrived. The problem is that there's no time range on the question so you couldn't tell the difference between someone who had tried to attend 4 times and been turned away 4 times and someone who had tried to attend 40 times and only been turned away 4 times.

I'm not a trained, capital-s Statistician, but if they'd come to me beforehand, I would have been able to help them fix these issues and get better results. In the first case, it still would have been a disappointing, "Sorry, your hypothesis seems extremely unlikely", but it could have saved them a second experiment. In the second case, it would have given them answers to some important questions and would have made the results sharper. (Another problem they had is that they surveyed multiple locations over time and at least some people were thus surveyed multiple times, with no question like "Have you taken this survey elsewhere?")

Perhaps not statistical issues per se, but in both of these cases smart, well-educated domain experts created instruments that were flawed, and the results were one dead experiment and one experiment with limbs amputated.

I've seen this kind of problem in survey-like and psychological experiments.

In one case, the entire experiment had to be chalked up to a learning experience. There were problems at multiple levels that resulted in a jumble of results, but results that seemed to give some support for the hypothesis. In the end, I was able to help plan a more rigorous experiment, which essentially had enough power to reject the hypothesis.

In the other case, I was handed a survey that had already been designed and executed, and there were multiple problems that resulted in several areas of interest being affected. In one key area, for example, they asked how many times the customers were turned away from an event due to it being full when they arrived. The problem is that there's no time range on the question so you couldn't tell the difference between someone who had tried to attend 4 times and been turned away 4 times and someone who had tried to attend 40 times and only been turned away 4 times.

I'm not a trained, capital-s Statistician, but if they'd come to me beforehand, I would have been able to help them fix these issues and get better results. In the first case, it still would have been a disappointing, "Sorry, your hypothesis seems extremely unlikely", but it could have saved them a second experiment. In the second case, it would have given them answers to some important questions and would have made the results sharper. (Another problem they had is that they surveyed multiple locations over time and at least some people were thus surveyed multiple times, with no question like "Have you taken this survey elsewhere?"

Does this apply to your question?

I've seen this kind of problem in survey-like and psychological experiments.

In one case, the entire experiment had to be chalked up to a learning experience. There were problems at multiple levels that resulted in a jumble of results, but results that seemed to give some support for the hypothesis. In the end, I was able to help plan a more rigorous experiment, which essentially had enough power to reject the hypothesis.

In the other case, I was handed a survey that had already been designed and executed, and there were multiple problems that resulted in several areas of interest being affected. In one key area, for example, they asked how many times the customers were turned away from an event due to it being full when they arrived. The problem is that there's no time range on the question so you couldn't tell the difference between someone who had tried to attend 4 times and been turned away 4 times and someone who had tried to attend 40 times and only been turned away 4 times.

I'm not a trained, capital-s Statistician, but if they'd come to me beforehand, I would have been able to help them fix these issues and get better results. In the first case, it still would have been a disappointing, "Sorry, your hypothesis seems extremely unlikely", but it could have saved them a second experiment. In the second case, it would have given them answers to some important questions and would have made the results sharper. (Another problem they had is that they surveyed multiple locations over time and at least some people were thus surveyed multiple times, with no question like "Have you taken this survey elsewhere?")

Does this apply to your question?