0
$\begingroup$

Thanks in advance for any responses or resources you can link me to.

I've had more of an Econ/Stats training and recently I've been working with some people who have a Psych background at work.

They do what they call "experiments" which, to be honest, I don't think they are experiments and whenever I ask questions about it, I don't get satisfactory answers. A satisfactory answer to me is more is a statistical explanation and not a practical one like "we've done this and it works" or "we need smaller sample size".

They have survey participants look at one (or more) control conditions and answer a question, then multiple treatment conditions and answer a question. The treatment conditions are all different treatments, not versions of the same treatment. The questions are all randomized in order.

Another issue is that, sometimes, they have so many controls/treatments, that not everyone answers all of them.

I think this is just a survey. It is not an experiment/survey experiment. I'm familiar with conjoint experiments and other types of within subject experimental designs, but this is not that.

How can this work when participants are seeing all treatments? Their answers would be affected by the order in which they saw them. Even if the order is randomized, 1/2 would see one first and the other 1/2 would see one second.

The results are not ATE (or related), so isn't this just observational data?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
3
$\begingroup$

Talking about the definition of the term "experiment", I think it is fulfilled if they first have a research hypothesis, and then what they run is planned in such a way that they can control what happens so that it can be replicated (obviously on different test persons) by somebody else. Also whatever analysis they do obviously needs to address the research hypothesis, and optimally they plan the analysis together with the actual running of the experiment in advance.

As far as I read your text, this is fulfilled here. Calling something "experiment" doesn't imply that it's a good experiment and results will be reliable. If you run this kind of experiment five times and get strikingly different results each time, the experiment isn't designed very well, but it's still an experiment.

But the relevant discussion is rather whether what they are interested in can be well identified from such data, and what features of the experiment may threaten the reliability. Carryover effects are probably a central issue here. I don't think the relevant discussion is whether it's an experiment or not.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Yes, that’s a good point. I guess when I think of experiment, I think of obtaining causal effects of x on y. They claim their results are causal effects of each treatment compared to the control and other treatments , but the treatments are not independent of one another. Your point helps me frame the questions I have in a better way and I could actually try to measure this from the results. $\endgroup$
    – DirichletC
    Commented Apr 12 at 15:21
  • $\begingroup$ My answer at stats.stackexchange.com/questions/116324/… may spark ideas of areas you would want to look into further, e.g., vignette analysis. $\endgroup$
    – rolando2
    Commented Apr 12 at 15:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.