Say I’m looking at dating website profiles. I note 20 adjective that are much more likely to be used on females’ profiles than males’ profiles. I note another 20 adjective that are much more likely to be used on males’ profiles than females’ profiles.
I then roll out a design change across the site that I hypothesise will reduce the use of ‘gendered language’. I want to test whether it has done so. Imagine that we push the redesign to only half of our users, so this is a proper RCT. Imagine that we are hoping for a reduction in 'gendered language'.
There are a range of possible outcome measures / methods of testing whether there's been a reduction in gendered language.
Use a list of the same words. The outcome measure would be the difference between the sum of male-coded words and the sum of female coded words, for each candidate (or would a ratio be better?). For example, one candidate could have 7 male-coded words and 4 female-coded words, and their score would be 7-4 = 3. Another candidate could have 2 male-coded words and 9 female coded words, and their score would be 2-9 = -7. We would evaluate the trial based on the difference in mean scores between control and intervention groups. We would evaluate each gender separately. Buuut... what if the 'score' went up for women (so women are getting more male-coded words and/or fewer female coded words) from -6 to -4. Great! But what if I told you that the intervention also made men's 'score' go up, too. From, say, 4 to 7. Now the difference in scores between groups is higher...
Run a proportion test for each word. This would involve 40 proportion tests (20 male-coded words, and 20-female coded words), so we would need to pre-define an analysis strategy to adjust for multiple comparisons. We could also decide to look at just the 2-6 of the ‘most problematic’ or ‘most gendered’ words to reduce comparisons. Buuuuut... We have a similar problem to the above. For example, let's say that the baseline is:
- 1% of males get called 'warm'; 7% of females get called 'warm'. The difference is 6 percentage points and the ratio is 1:7.
Then we intervene. Now: - 7% of males are called warm; 21% of females are called warm. The difference is now 14 percentage points (worse) but the ratio is now only 1:3 (better).
To avoid this thorniness, we would probably want to look at each gender separately. So run a test on proportion of women called 'warm' in control vs. intervention (looking for decrease). And separately run a test on proportion of men called 'warm' in control vs. intervention (looking for increase). But this would now mean 80 comparisons (40 words x 2 genders)...
- Should I use a joint test of orthogonality?
Here’s how I would envision it working: - Using the same list of words that the exploratory analysis has already found, we would make each word an indicator variable which takes the value of 1 if the word is used in a profile and 0 if it is not. We would then run a joint test of orthogonality: - We take our set of 40 words (X1, X2, …, X40) and run the following regression: - Female = a + b1*X1 + b2*X2 + b3*X3 + ….+b40*X40 +u - We then test the joint hypothesis b1=b2=b3=…=b40=0 as a linear regression, with an F-test.
But how should I use the indicator variable for whether the user has been ‘treated’ or not? Interacted with each word-indicator variable?
- Use a blinder oaxaca decomposition?