How to deal with participants who fail comprehension checks non-randomly Let's say I'm running a study evaluating the effects of time pressure on some decision making task.  I add a comprehension check to ensure participants understand the task they are about to complete, and the comprehension check involves some element of time pressure.   The control condition gets no such comprehension check because their task does not involve time pressure.
Now, let's say 40% of the treatment fails the comprehension check.  I can see two approaches to dealing with this:


*

*I remove from the dataset all individuals who fail the comprehension check

*I include all the observations, regardless of whether or not they pass the comprehension check, and add a dummy variable for passing the comprehension check.


Which of these two approaches is preferable?  Is there a better approach than either of these?
More importantly, it strikes me that both of these approaches suffer from the problem that the comprehension check itself covaries with the treatment (in a somewhat unavoidable way).  Is it possible to find a treatment effect, even after doing option #2, that is somehow "caused" by comprehension check?
 A: I found a really good paper on post-treatment variables that mentions:

If passage rates on an attention check are rejected by the treatment for any
  reason, then dropping respondents who fail it would be the equivalent of dropping cases on a post-treatment covariate and would again risk bias.

So, yes, your concern is valid. I believe they talk about possible techniques to handle this case.
I'm not an expert, but my gut is that you should: 1) document what percentage of the control group failed the check. 2) Analyze all of the data, ignoring the check question, as if the question weren't asked.
By ignoring the check question, you won't commit post-treatment bias -- which in your case could be severe I think -- but you will add some noise because of participants who were not paying attention.
But you do at least have a baseline percentage of check failure in the control group. Hopefully it's low. I think we could say that a similar proportion of the treatment group would also fail the check if stress were not part of the treatment, so that proportion of the whole would give a rough idea of how much noise -- meaningless surveys -- we introduced into our study by "not doing" (not using, actually) an attention check.
(For the record, your original posting seems to imply that the check is a question about the task ahead, that's administered before the survey. But I think it's a question slipped into the survey to check comprehension/attention.)
