up vote 1 down vote favorite

I'm rephrasing a question I asked before - any input would be hugely appreciated, as I haven't found similar examples online.

Consider a design where random effect 'a' is fully crossed with random effect 'b' and random effect 'c' is fully crossed with random effect 'd'. Random effect 'c' is nested within a x b combinations.

Design structure

'a' are language editors, 'b' are sentences to be edited, 'c' are the edited versions of these sentences (i.e. edited by 'a'), and 'd' are judges who provided quality scores on 'c', the edited versions. Each instance of 'c' received a different ID in the data, though they may turn out to be identical (i.e. in case editors edit the same instance of 'b' in the same way). The DV is quality scores related to 'c' (provided by 'd') and predictors relate to 'a' and 'b'.

My questions are: - Do I need 'c' as a random effect in the model, when each instance of 'c' is already represented by a x b combinations? - Is it OK to give different IDs to each 'c' even when there's the possibility of these being identical (i.e. if the same 'b' happens to be edited in the same way by different editors)?

share|cite|improve this question
  • Could you please add a link to the previous question? – Xi'an Oct 18 '15 at 13:41
  • Here: stats.stackexchange.com/questions/177313/… – user3744206 Oct 18 '15 at 13:42
  • 1
    Is there any evidence that a judge would give a different score to the identical edited sentence if he or she saw it more than once? Do you care about such intra-judge discrepancies? – EdM Oct 18 '15 at 14:40
  • I did measure this and intra-judge discrepancies are quite low, i.e. they do give the same scores on most occasions. Ideally, these discrepancies shouldn't exist, so if there's a way of controlling for them or taking them into account, that would be better, but I guess they're unavoidable, so it's not a huge problem. – user3744206 Oct 18 '15 at 14:49
  • I guess only now I understood the question: yes, if the same judge provides different scores to identical sentences, this is not assumed as a discrepancy in the model, since each instance of 'c' is labelled as a different entry - I'm not too worried about this because intra-judge discrepancies were rare, as I said. – user3744206 Oct 18 '15 at 16:24
up vote 1 down vote accepted

Adding a label to each distinguishable edited sentence in set 'c' might provide a way to evaluate intra-judge variability in scoring, based on each judge's different ratings of the quality of the same sentence. Note that this labeling of distinguishable sentences doesn't add anything to evaluating the upstream issues of the input sentence-editor-output sentence process, your primary interest.

You could incorporate this intra-judge variability in your model, but you don't have much evidence for such intra-judge variability, and it doesn't seem to be of much concern to you. In that case there really is no need to add this extra complexity to what seems otherwise to be a nicely balanced crossed design, and would probably cost you unnecessary degrees of freedom.

If you want the best estimate of inter-judge variability, you will have to code things in a way that you compare responses of all judges to each combination of input sentence and editor. If the inter-judge variability is not of concern, and you simply want to average scores among the judges and perhaps correct for differences in mean scores among judges, then you don't have to code the individual output sentences. So the answer to your question depends on how much you want to know about inter-judge variability.

share|cite|improve this answer
  • This helps a lot. Am I right then that for the purpose of answering my main question (input sentence-editor-output sentence process) it's irrelevant if I add codes for the output sentences to the study as a random effect? This changes coefficients very slightly (though it doesn't alter my findings). And inter-judge variability is not an interest in itself, I just need to correct for that in answering the main question. – user3744206 Oct 18 '15 at 22:08
  • Couldn't add @EdM to the previous comment for some reason. – user3744206 Oct 18 '15 at 22:16
  • Yes, there is no need to include the output sentences as a random effect, although you could choose to. There is always a trade-off between the extra precision you might get by adding terms to a model and the associated loss of degrees of freedom for your significance calculations. Use your judgment based on your knowledge of the subject matter and of the intended audience for your results. And be clear in your description of the model. – EdM Oct 18 '15 at 22:17
  • @user3744206 : the author of an answer automatically gets notified of a comment on the answer, just as the author of a question gets notified of comments on the question (and maybe comments on the answers, I'm not sure). – EdM Oct 18 '15 at 22:19

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.