0
$\begingroup$

I want to compare the outcomes of individual subjects. This outcome changes systematically depending on the subject's location (e.g. it will always be lower in a certain location). I want to be able to compare this metric across subjects in different locations (i.e. to be able to see if a subject in one location is better/worse than a subject in another location).

The only thing I've come up with is to group the subjects by location, standardise their outcomes as Z-scores in these subgroups, then use those scores as the "outcome" to compare. Is this sound?

Edit for more detail: The 'outcome' I'm looking at is a performance measure (think a KPI) that is calculated over a period of time (number of detections/items searched). It's not a change from baseline measures. This differs between locations because of more/less opportunity (fewer things to detect in some locations).

$\endgroup$
6
  • $\begingroup$ Do you measure the subjects multiple times - for example at the start and end, in order to determine outcome ? Please provide more info about the study design and your research questions. $\endgroup$ Commented Aug 17, 2021 at 13:03
  • $\begingroup$ Do you mean something like "Subject 1 in location A is one standard deviation about the mean of A, and subject 2 in location B is two standard deviations above the mean of location B, so subject 2 is doing better, despite the lower measurement"? (Think of having \$2 million now vs \$1 million a hundred years ago. The number of today's dollar is larger, but who is wealthier?) $\endgroup$
    – Dave
    Commented Aug 17, 2021 at 13:33
  • $\begingroup$ @Dave - yes, pretty much. The metric might be smaller for an individual in a certain location, but they are actually performing better than an individual at another location. $\endgroup$
    – Ella
    Commented Aug 18, 2021 at 2:15
  • $\begingroup$ @RobertLong - sorry, I've edited my post to be more clear. The 'outcome' I'm looking at is a performance measure (think a KPI) that is calculated over a period of time (number of detections/items searched). It's not a change from baseline measures. This differs between locations because of more/less opportunity (fewer things to detect in some locations). $\endgroup$
    – Ella
    Commented Aug 18, 2021 at 2:15
  • $\begingroup$ Then I repeat my question about the millionaires: who is wealthier? Why? $\endgroup$
    – Dave
    Commented Aug 18, 2021 at 2:39

1 Answer 1

0
$\begingroup$

To address the "is this sound?" question: It depends. It's not generally a bad idea. Using location-wise Z-scores here would be based on the following considerations:

  1. You have enough data to estimate mean and SD precisely enough for all locations.

  2. The implied idea is that performance is measured appropriately relative to the location mean scaled by the location SD. This may or may not be the case. In particular it has to do with how individuals are assigned to locations. It needs to assume that it can be expected that the "quality distribution" of individuals in the different locations is the same. If for some reason one location has generally weaker performing individuals, its mean will reflect that, and therefore Z-scores will not be properly comparable to those of other individuals elsewhere. Also, are there reasons to expect the different locations not only to have different means but also different SDs? Generally, if some values compare well over locations without Z-scoring, Z-scoring may well destroy that.

  3. Z-scoring operates with a symmetric concept of distribution (the SD doesn't differentiate between values that are above and below the mean). This may be problematic if you have very skew distributions or different skewness in different locations. E.g., measures may be bounded from below by zero, and distributions may concentrate close to zero at the lower performing locations. The nature of measurements and known possible value ranges also play a role assessing whether mean and SD may be badly affected by potential outliers.

  4. Other issues will play a role, for example if dependence between observations occurs. One instance is that several observations stem from the same individual, which should not be treated as independent. There may be learning effects from one task to the next, and potentially also interaction between individuals and locations (which can only be detected if you have several observations from the same individual at the same location).

  5. Knowing the tasks, are there any other possibilities for reasonable standardisation? For example, is there a realistic maximum that changes between locations, in which case one could use a percentage of this as score? (Obviously knowing all the details can reveal problems with that as well; ultimately the issue of whether this is appropriately comparable will always rely on subject matter knowledge as well as statistics.)

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.