I have a software that produces a sentiment score between 0 and 1 given an input sentence. I want to test if this system produces biased scores based on the gender of the subject used in the sentence.

I have 50 sentence pairs and for each sentence pair I get a pair of scores from my software. Therefore I have 50 pairs of scores.

I want to give an example sentence pair and score pairs:

Input Sentence: John feels angry    -> score: 0.15
Input Sentence: Marry fells angry   -> score: 0.19

Another example:

Input Sentence: Adam likes music    -> score: 0.10
Input Sentence: Isabel likes music   -> score: 0.34

At the and I create two arrays keeping the scores I obtained with those sentences:

male_scores = [ 0.15,0.10, ....]
female_scores = [ 0.19,0.34, ...]

As I said earlier,now I want to see if these score differences tells me something about the fact that my software have some gender bias or not.

In order to do that, should I use independent t-test or dependent t-test?

  • 2
    $\begingroup$ So each sentence appears twice, with the only modification being the gender of the person mentioned in the sentence, and you want to test if there's a difference in mean between the scores with regards to gender? If the choice is between a paired t-test and a unpaired t-test, then you should use the paired t-test. $\endgroup$ – Phil Nov 7 '18 at 10:33
  • $\begingroup$ @Phil Thank you for your comment, this is exactly what I want to do. Do you have any other suggestion ? $\endgroup$ – zwlayer Nov 7 '18 at 12:05
  • 1
    $\begingroup$ If the t-test is sufficient for your purposes, then do that analysis (but do check the corresponding assumptions behind the paired t-test so that they are fulfilled for your data). If you want to include covariates in your analysis, then you could add them in a regression framework. $\endgroup$ – Phil Nov 7 '18 at 12:10
  • 1
    $\begingroup$ I just realised that you wrote that the outcome is bounded between 0 and 1. That might affect the plausibility of the analysis, so check the distribution of the paired differences to see if they look roughly normal. $\endgroup$ – Phil Nov 7 '18 at 12:12
  • $\begingroup$ @Phil, if they are (almost) normal I use paired-t test right? $\endgroup$ – zwlayer Nov 7 '18 at 12:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.