Significance test for two mean differences of a continuous variable, 4 independent samples There are four unique groups: women (F) and men (M), having taken a course (2) or not (1)
All the groups contain different students.
I want to compare the grades obtained in another course, that's the content of the vectors below.
Specificially I want to compare if the grade increase amongst women is larger than the grade increase amongst men with an appropriate test and get p-value and confidence interval.
F1 <- c(3, 2, 2.5, 1.5, 3, 3.5, 3.5, 4, 3, 3, 3.5, 2.5, 2, 1, 3, 4, 3.5, 3, 5, 4, 3.5, 4.5, 4, 3) 
F2 <- c(4.5, 5, 4.5, 4, 3, 3, 3.33, 4.5, 5, 4.5, 5, 4.5, 5, 4.5, 4.33, 4)

M1 <- c(4, 3.5, 3, 5, 4.5, 4, 4.67, 3.5, 4.5, 3, 4, 4.5, 3.5, 3.5, 3, 2.5, 3.5, 3.5, 3, 3.5, 5)  
M2 <- c(5, 3.5, 4, 4.5, 3.5, 5, 5, 4.5, 5, 4.5, 4.5, 5, 4, 4, 3)
  
# With independent t-test I can test the grade increase for women. It is significant. 
FF <- rbind(as.data.frame(cbind(grade=F1,id=10)), as.data.frame(cbind(grade=F2,id=2)))  
# (Note: id above set to 10 (larger than 2) just to get a positive confidence interval)
t.test(FF$grade ~ FF$id)
  
# And similar for men. 
MM <- rbind(as.data.frame(cbind(grade=M1,id=10)), as.data.frame(cbind(grade=M2,id=2)))
t.test(MM$grade ~ MM$id)

With the numbers above:

*

*for women the increase is  4.29 - 3.15 = 1.14. The 95% confidence interval is [0.63, 1.66]

*for  men  the increase is  4.33 - 3.77 = 0.67. The 95% confidence interval is [0.10, 1.03]

What I really want to test, is if the increase for women is larger than the increase for men at 95% confidence.
And I would also like a confidence interval in case it is so.
Simply by looking at the two intervals, I see there is overlap, so with the numbers above I suspect I cannot make such a claim.
And if they did not overlap, say [0.90, 1.30] vs [0.40, 0.70], what claim could I then make?
But this is a bit of handwaving.
There is probably a more appropriate way, a test, to settle the issue?
 A: Assume four groups $i=1,\ldots,4$. The number of observations in each group is called $n_i$, distribution of observations in each group i.i.d. ${\cal N}(\mu_i,\sigma^2_i)$. Want to test $H_0:\ \mu_2-\mu_1=\mu_4-\mu_3$. Both of the following generalise the standard theory for two samples to four samples.
Version 1 using asymptotic normality: Let $\bar X_i$ be the mean of the observations in group $i$ and $S_i^2$ the estimated variance in group $i$. The distribution of $(\bar X_2-\bar X_1)-(\bar X_4-\bar X_3)$ is
$${\cal N}\left((\mu_2-\mu_1)-(\mu_4-\mu_3),\sum_{i=1}^4\frac{\sigma^2_i}{n_i}\right).$$
Under $H_0$ the expected value is zero and the distribution of $T_\sigma=\frac{(\bar X_2-\bar X_1)-(\bar X_4-\bar X_3)}{\sqrt{\sum_{i=1}^4\frac{\sigma^2_i}{n_i}}}$ is ${\cal N}(0,1)$.
We don't know the $\sigma_i^2$, but asymptotically, with $n_i$ large enough in all groups, $T=\frac{(\bar X_2-\bar X_1)-(\bar X_4-\bar X_3)}{\sqrt{\sum_{i=1}^4\frac{S^2_i}{n_i}}}$ will be distributed ${\cal N}(0,1)$. So $T$ can be used as test statistic and the standard normal distribution can be used to compute the p-values. Obviously the issue here is whether the $n_i$ are large enough for this to work.
Version 2 assuming equality of within-group variances: This does not require asymptotic approximation (large $n_i$) if the model assumptions hold. The variance of $(\bar X_2-\bar X_1)-(\bar X_4-\bar X_3)$ can then be estimated by $S^2=\frac{\sum_{i=1}^4 (n_i-1)S_i^2}{n_1+n_2+n_3+n_4-4}$, and $T_S=\frac{(\bar X_2-\bar X_1)-(\bar X_4-\bar X_3)}{\sqrt{S^2}}$ is distributed $t_{n_1+n_2+n_3+n_4-4}$ under $H_0$, so that $T_S$ can be used as test statistic and $t_{n_1+n_2+n_3+n_4-4}$ for computing p-values.
So you basically have the choice to use Version 1 assuming that all $n_i$ are large enough, or Version 2 assuming that the variances are all the same. Deriving a Version 3 for finite samples but different variances is too difficult for me to do on the spot.
Unfortunately with your data all assumptions are questionable. The data are not normal (obviously discrete and constrained between 1 and 5 with skewness in some groups - note that your data are clearly not continuous because all values seem to be either integer numbers or halves between them but this is really the smallest issue here), the sample sizes are not very large, and variances are potentially different. This is quite typical in statistics, so I don't think it is a big worry, just a reason for caution. What I'd probably do (without investing time to come up with something for small samples and different variances, or even with something nonparametric - both of which could probably be done as a research project) is to compute both versions, and if both either clearly do not reject or clearly reject, I'd say that there is clearly evidence, or no evidence against the $H_0$. If they don't agree and/or both are close to your significance level, 0.05, say, I'd probably say that this is borderline/inconclusive.
By the way, what you could also do is to run a two-way ANOVA with M/F and group 1/2 as factors and test whether there is an interaction between the two factors (and what sign it has). This again requires equal variances, and is probably equivalent to Version 2 (I haven't tried to prove this), or very similar at least.
