My stats knowledge is average but not great, so I apologise if this is something I just haven't come across before. Basically, I've just read a paper which seems to state that it has conducted correlational analyses on a set of variables, where the IV scores are not matched with the DV scores, and I simply don't see how this is possible? To give more detail:

The study is looking at the possible association between:

a) anti-social personality traits, and social climate on a therapeutic prison unit

b) suicidal ideation, and social climate on a therapeutic prison unit

The measures were operationalised and measured in the following way:

The independent variables (anti-social personality and suicidal ideation) were measured via a personality inventory given to everybody entering the therapeutic unit.

The dependent variable (social climate) was measured using an established questionnaire of social climate, comprising three sub scales. The questionnaire was given out to everyone who had completed the IV measures, BUT responses were returned anonymously. The final sample size was 125.

Therefore, there was no way of matching the social climate scores to the scores for the IV measures.

My issue is that I don't see how any tests of correlation can possibly work, if you don't have an IV score AND a matched DV score for each participant?

The description of the analytical procedures from the article might help. It says (paraphrased for ease of reading):

"As the [social climate measure] was completed anonymously, the data from the two questionnaires were unmatched. Therefore, data was grouped by [the five different groups within the therapeutic unit], producing five sets of data."

"Kendall’s tau-b rank correlation coefficient analysis was administered to explore correlations between variables. This was appropriate as the data was not normally distributed, non-linear, and the sample size was relatively small [125 residents]. Kendall’s correlation is a robust and efficient non-parametric test (Croux & Dehon, 2010). Mean scores for each measure were used in the analysis because the raw data was unmatched."

A table of correlations between the IVs and all the DV sub scales is then presented, which looks just like a regular correlation table.

The first comment (about splitting the data into five groups) I just don't understand why that makes a difference? The second comment about Kendall's tau I can understand because of other characteristics of the data. But the last part about using mean scores, I don't understand either. What does that look like and how does it help?

The only way I can make any sense of it (and this isn't detailed in the article, this is purely a guess), is that they took the mean IV and DV score for each of the five therapy groups within the bigger therapy unit, which would produce five pairs of matched scores.

But even with those "matched" scores, that would be effectively a sample size of 5, right? Which is way too small for any correlational analyses to be reliable. Plus by using the means, wouldn't that obscure information on variance, which is needed for correlational analyses? I'm not even sure if this is what they did anyway.

Basically my main question is: is this bad statistics, or is this me just not understanding what on earth is going on??

Would really appreciate some answers in not too complicated language :)

  • $\begingroup$ Can you provide a citation & link to the paper? $\endgroup$ – gung - Reinstate Monica Dec 16 '20 at 20:08
  • 1
    $\begingroup$ It sounds like they averaged over the responses w/i each unit, & correlated the unit means. If so, that's an inefficient use of the data, but not necessarily invalid. $\endgroup$ – gung - Reinstate Monica Dec 16 '20 at 20:09
  • $\begingroup$ Thanks! Sorry I can't provide a link because its in a practitioner newsletter thats only available via member login. So.....do you mean they just gave each individual person the average score for their unit? $\endgroup$ – ConfusedPsychologist Dec 17 '20 at 10:06
  • $\begingroup$ No, I mean they have 1 data point per unit, but I can't tell without the paper. $\endgroup$ – gung - Reinstate Monica Dec 17 '20 at 12:43

What you describe seems to be a growing problem, due to privacy concerns one gets unmatched data where matching is needed. There must be a better solution to this privacy problems ... but that is not the topic now (too late for you).

So yes, seems that by taking means within the five groups, you have a data set of $n=5$ matched pairs of means. Apart from that being very few for a meaningful correlation, it will also overestimate the correlation. Such an analysis is not invalid, but probably not very helpful. They need to find a better solution to the privacy concerns!


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.