Is the t-test adequate?

Question

I am comparing two sets of samples that are of different sizes (n) and are from two different populations. Both sets have undergone a similar intervention at different time periods (one from August to December and the other from January to April). Although both sets have treatment groups, this is not the typical one control and one experiment scenario. My objective is to study the mean of both samples.

To achieve this, I am using a two-sample t-test assuming unequal variances to deduce the p-value. If the p-value is less than 0.05, I will conclude that the difference is significant. Is this the right approach? Should I be using a t-test, and if so, which one? Additionally, would calculating Cohen's D be valuable in this case?

Please let me know if my approach is correct or if there are any other suggestions you would recommend.

Answer 1

Suppose that we model the data in the form of normal random variables $X$ and $Y$. To be more precise, let $\pmb{X} = \left(X_1, \dots, X_m\right)$ form a random sample of $m$ observations from a normal distribution with mean $\mu_1$ and variance $\sigma_{1}^2$. And let $\pmb{Y} = \left(Y_1, \dots, Y_n\right)$ form an random sample of $n$ observations from another normal distribution with mean $\mu_2$ and variance $\sigma_{2}^{2}$. Furthermore, $\pmb{X}$ and $\pmb{Y}$ must be independent given the values of $\mu_{1}, \mu_{2}, \sigma_{1}^{2}, \sigma_{2}^{2}$.

Let's say we want to test the following hypotheses at a chosen level of significance $\alpha_0$ (typically set to 0.01 or 0.05): \begin{equation}\tag{1} H_0: \mu_1 = \mu_2, \quad \text{ versus } \quad H_1: \mu_1 \neq \mu_2. \end{equation}
If we assume that $\sigma_{1}^{2} = \sigma_{2}^{2}$, different t-tests are available to test hypotheses like (1).

Otherwise, if we do not assume that $\sigma_{1}^{2} = \sigma_{2}^{2}$, and also if the value of the ratio $\sigma_{1}^{2} / \sigma_{2}^{2}$ is unknown, then only approximate tests are available. The most popular of the tests proposed in this case is the approximation procedure known as Welch's t-test; see Wiki. An alternative to the Welch's t-test is the likelihood ratio test, which is somewhat more complicated. To find out if Cohen's d is helpful for the Welch t-test, see the responses to this question at Cross Validated.