1
$\begingroup$

I have a variable test score that is measured for an organization consisting of 2050 people. The goal is to test if men outperform women (or vice versa). Unfortunately, only 50 of the 2050 people are women. I was told that in order to run inference when there is such a disproportion of one group within a sample that I should pursue some sort of matching strategy.

First, is this true, that if I match on a few characteristics that I can run inference (e.g., ordinary least squares, linear probability models) and be less worried about the fact that there are so few females--since many males will drop out?

Second, is there anything I can do to compare the groups before the match even as an anecdote?

This is the data:

           Test Score    Test Score
     N        Mean        Variance    Coef. Var.     Gender    
   2000      26.12         10.89         0.13         Male        
     50      56.10         25.01         0.09        Female
$\endgroup$
  • $\begingroup$ Were you given a reason why you should match rather than directly compare the means? (Oh and um, what does 'drop up' mean?) $\endgroup$ – Glen_b -Reinstate Monica Nov 4 '13 at 4:07
  • $\begingroup$ @Glen_b Typo, drop 'out', thanks. Basically I was told that when you are comparing groups and one dominates the other in terms of N then nothing but the strongest of effects will be significant. I am open to other options, but I basically want to show that females perform better (can I do this rigorously?) and then see if it is gender that drives the effect or if something else, such as females are just more educated on average. I guess that is where matching would come in if I match on education I could run an OLS and see if gender still matters. $\endgroup$ – CJ12 Nov 4 '13 at 4:38
  • $\begingroup$ An unpaired comparison with group sizes 50 and about 2000 will have almost equal power than a one sample test with 50 observations. To beat this with a paired comparison, an intrapair correlation of at least 0.5 is required as long as the mean difference remains the same. Thus, even based on power considerations only, there is no general rule which way to go. $\endgroup$ – Michael M Nov 4 '13 at 6:44
  • $\begingroup$ @CJ12 "one dominates the other in terms of N then nothing but the strongest of effects will be significant" -- I'm not sure what the basis of this claim might be. Having one sample size 40 times as large as the other doesn't add much over having it 5 times larger, but there's nothing to gain by reducing the sample size. Matching, if you're able to do it, may be quite helpful, or it may do little, depending on the level of correlation. $\endgroup$ – Glen_b -Reinstate Monica Nov 4 '13 at 11:36
  • $\begingroup$ @Glen_b This is helpful, how do you recommend comparing the Test Score between males and females in this sample to see if the difference is statistically significant? $\endgroup$ – CJ12 Nov 4 '13 at 13:58
1
$\begingroup$

This answer will not be very complete.

1) If the performance scores are reliable (in the sense that they do not change considerably when measured repeatedly within worker), then there is absolutely no point in doing statistical inference (i.e. estimates, tests, confidence intervals) about population parameters, because your sample is equivalent to the corresponding population. Your descriptive presentation of means and variances could be complemented by adding medians, quartiles and boxplots etc.

2) If the performance scores are not very reliable, then you could think of a very simple model of the form $$ \text{Performance} = \alpha + \beta \cdot \text{Gender} + \varepsilon $$ with Gender = 1 if female and 0 if male. The random error $\varepsilon$ accounts for the fact that performance varies within person (depending on day, mood etc.) and that the true "gender-effect" $\beta$, i.e. the true average overperformance of women, is not exactly $\hat\beta =56.10 - 26.12 = 29.98$ but might change from day to day. With a two-sample t-test (with Welch-correction for unequal variance) you might test whether $\beta \ne 0$. If the p-value is below 5%, you can assert with 95% confidence that women systematically outperform men. (We could now discuss about whether we should pick an other two-sample test or log transform the data etc.)

3) As quite always when men are compared to women, the question of confounding raises: Is the difference really due to gender or can it be explained at least partially by other factors such as age? It's possible that in your company, women are typically older than men, which could already partially explain the difference in performance. One simple way to try to adjust for confounding is to modify the simple model from 2) to $$ \text{Performance} = \alpha + \beta' \cdot \text{Gender} + \gamma \cdot \text{Age} + \varepsilon $$ and infer about the age-corrected gender effect $\beta'$ by linear regression.

An alternative to the modelling approach is matching by age followed by a paired t-test (or similar). This also adjusts for potential age-confounding but leads to another estimated "gender-effect" $\hat \beta''$. Depending on the variability of the performance difference between pairs (which is directly linked to the intra-pair correlation) and the difference between $\hat \beta$ and $\hat \beta''$, the power of such test is higher or lower than the two-sample test in 2).

Summa summarum: Matching leads to inference about a confounder-adjusted effect of gender, whereas the unmatched comparison infers about the "crude" gender effect without considering any confounders. Both parameters might be interesting, so you could easily summarize both of them.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.