0
$\begingroup$

Suppose I'm sampling two groups for an experimental research where I'm measuring how people respond to a certain type of question under the interference of certain factors in one of the groups, and without the interference in another.

Aiming to have a 0.95 confidence within a 0.05 interval, the sample size is almost a prohibitive target to seek twice (one time for each group) due to certain project constraints.

So suppose we have i.e one variable x1 to measure. Rather than measuring it once for each participant (by asking one question), what if we measure it twice for each participant through asking him/her two equivalent questions instead (i.e. same difficult level, structure, etc, but different numbers)?

Example: Suppose we ask two questions q1 and q2 to measure x1 variable, and got the following: - Participant 1 answered q1 wrong and q2 right. - Participant 2 answered both q1 and q2 right. - Participant 3 answered q1 right and q2 wrong.

So I attribute the following to x1: n=6; 4 right answers, p=0.67

Would it be a valid procedure for both drawing an inference for x1? What if we, later, compare the measurements of x1 in the two groups for a Z test?

Is there any literature discussing such approach?

Also, is there anything to do to check whether the questions were really equivalent (i.e. like a MacNemar test)? Would it be necessary?

$\endgroup$
1
$\begingroup$

This is repeated measures. It tends to give you more information than a single answer from each person and less than having the same number of independent answers from different people. You cannot treat answers from the same person as independent (and sum up proportions across and within people), because you would assume answers from the same person to be independent. Repeated measures models/hierarchical models/random effects models are the typical approach for taking this into account.

You would gain the most by observing the same subject not just repeatedly, but also under both conditions of interest (cross-over trial), if that is possible.

Whether different questions have the same difficulty can be investigated or taken into account in analyses using item response theory. You essentially assume a subject skill level (a fixed or more likely random effect) and a question difficulty (likely a fixed effect; or when it is more than y/n, difficulty to get a certain score) and let the (log it of the) probability of a correct answer depend on that in a way that is more or less steep in the skill (very steep: below a certain skill level you never get the question right, above always; less steep: much more gradual rise in probability of correct answer with skill).

Whether they measure totally different concepts would be a lot harder to look at.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.