3
$\begingroup$

I have performed a study where I tested 2 groups, before and after two different types of manipulations. The main outcome of my study were the group differences at time point 2.

Now someone asks what the reliability of my outcome score is. Could any observed changes be an artifact of the measurement method?

I was wondering if this is something I could statistically test using my existing dataset. I tried rearching online for some validity/reliability tests. Here I found for example test-retest reliability checks. I however wonder if this is appropriate, since this seems to test the consistency of the results over time, whereas I am actually interested in seeing a change in the results, given my manipulations.

My main question therefore is, can I check the realibility of my results after having collected the data, or should a seperate study be conducted on the test itself?

$\endgroup$
1
  • 1
    $\begingroup$ What are these measurements that you took and why would someone suspect that there is an issue with the measurement method? $\endgroup$
    – dipetkov
    Commented Oct 29, 2022 at 14:15

1 Answer 1

6
$\begingroup$

There are various reasons why a test outcome may be unreliable. Some of these can be diagnosed from the data such as certain model assumption violations. You are also encouraged to look at a confidence interval of the difference, as this will tell you something about the precision of results (assuming the model). But it seems you are concerned here in the first place about problems with your measurements.

Now this is an issue that the data themselves cannot address, as it refers to potential deviations of what you actually measured from what you want to measure. The latter is not observed in your data, so the data themselves will not serve to find out about this. Generally measurement validity and reliability are hard to address, as very often the measurements are chosen because better alternatives don't seem to be available. What you need is a theory regarding the relation of measurements to their actual target. Test-retest reliability is useful if you can run retests under conditions under which your theory suggests that the "true" results (your measurement target) should be the same. It depends on the specific problem/study whether this makes sense. From how I read your question, this may apply to measuring twice without the "manipulation" the effect of which you're interested in, but I can't tell from the given information.

You can also use alternative measurements that are supposed to measure the same concept or to be in a specific relation to the concept in order to check validity if such measurements exist, however the basic difficulty is always that the ultimate question is how your measurements relate to an assumed underlying true value that cannot directly be observed. Apart from empirical investigation, properly thinking through the measurement, how it is related to the concept of interest, and potential issues that may invalidate the measurement, is also important, plus considering the related literature for thoughts and experiences.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.