How can reliability and validity of content analysis be quantified when there is only one person coding the data?
$\begingroup$
$\endgroup$
2
-
1$\begingroup$ Are you asking if there's an analog of (say) Cohen's kappa for interrater reliability when you have only one rater? $\endgroup$– gung - Reinstate MonicaJul 16, 2013 at 2:28
-
$\begingroup$ Well not exactly. I am asking for a reliable and valid way to do a content analysis from one researcher (which also serves as one coder) Thank you very much for asking. $\endgroup$– user22478Jul 16, 2013 at 13:18
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
3
Usual reliability indices (Cronbach $\alpha$, Cohen $\kappa$, etc.) only ever quantify the influence of a single source of error. For $\alpha$ the relevant source of error is item-specific variance, for $\kappa$ and other measures of inter-rater argument, it is rater-specific error. In any measurement situation, there are other sources of error that might or might not matter for your purposes.
Generalizability theory provides a systematic framework to approach these multiple sources of error. In generalizability theory, a study has a number of “facets” like items, time or raters. Their influence is first quantified in a “G study” and these estimates can be used to adjust the measurement. This has a number of practical applications but the important thing with respect to your question is that it is only possible to quantify error related to the facets that vary in your study and you must design it to include all the facets you care or worry about.
For example, if you only have one point of measurement, you don't know how stable the measure is over time and if you have only one rater, you can't possibly know how much raters would differ from each other. Stated that way, it might sound obvious but talk of “estimating the reliability” often obscures this basic point.
In practice, it means that in your case you have no way to check if your results would generalize to other raters or if different people would categorize the content in the same way (one rater might in fact be enough, but you can't check that with your data). If you have concerns about inter-rater agreement (as is often the case), you really need to have another rater code at least part of the content. You can however quantify other sources of error (say temporal stability, different coding schemes or software, etc.) with a single rater and call that “reliability assessment” but that still would not tell you about inter-rater agreement or eliminate potential issues with rater-specific error.
-
1$\begingroup$ Although the answer did not keep me happy which was highly anticipated, I am VERY VERY VERY grateful for your detailed and justified answer. What a great site! What a great man! Thank you! I really appreciate it. $\endgroup$ Jul 16, 2013 at 13:15
-
$\begingroup$ In that case, @user22478, you might want to upvote this answer by clicking on the upwards facing normal distribution to its left, & accept it by clicking the check mark below the vote total. $\endgroup$ Jul 16, 2013 at 15:44
-
1$\begingroup$ Alright I happily did it! Any other additional or even controversial inputs are welcomed! $\endgroup$ Jul 16, 2013 at 18:30