Which part of calculating the T statistic relies on assumption of normal distribution? Recently, I got into an argument with a professor at my University. As part of a discussion, I made a statement that the calculation of the T statistics (independent sample T test: (mu1 - mu2)/PooledSD) does not by itself have any assumptions about the distribution of the data. My point was that the statistic itself is a simple signal to noise ratio kind of measure and it is the calculation of the p value which relies on the normal distribution. I went on to say that the p value could alternatively be calculated without making any assumptions by using permutation testing to build a distribution and find the p value. The professor on the other hand argued that my understanding was totally incorrect and that the calculation of the T statistic assumes a normal distribution.
My question is: which part of the independent sample t test makes the assumption of the normal distribution?
On a related note: is the normal distribution assumption about the data or the residuals of the data?
 A: Let $Z \sim \mathcal{N}(0,1)$ and $U \sim \chi^2_n$, and further assume $Z$ is independent of $U$. Then, the random variable
$$X = \frac{Z}{\sqrt{U/n}} \sim t_{n}.$$
In other words, $X$ is distributed according to a $t$ distribution with $n$ degrees of freedom if it is the ratio of a standard normal and the square root of a chi-squared random variable over its degrees of freedom. This is a definition we'll appeal to later.
Specifically, consider two independent samples from two normal populations $\mathcal{N}(\mu_X, \sigma^2)$, and $\mathcal{N}(\mu_Y, \sigma^2)$. Denote the first sample as I.I.D normal random variables $X_1, X_2, ..., X_n$, and the second sample as $Y_1, Y_2, ..., Y_m$.
The test statistic that we use (as I'm sure you're familiar with) is
$$T = \frac{\bar{X} - \bar{Y} - (\mu_X - \mu_Y)}{s_p\sqrt{\frac{1}{n} + \frac{1}{m}}}$$
where $s_p = \sqrt{\frac{(n-1)s_X^2 + (m-1)s_Y^2}{m + n - 2}}$, and $s_X^2$ and $s_Y^2$ are the sample (unbiased) standard deviations of each of our samples. We multiply $\sigma\sqrt{(1/n) + (1/m)}$ in the numerator and denominator of $T$ and simplify to get
$$T = \frac{\frac{\bar{X} - \bar{Y} - (\mu_X - \mu_Y)}{\sigma\sqrt{\frac{1}{n} + \frac{1}{m}}}}{\sqrt{\frac{\frac{(n-1)s_X^2}{\sigma^2} + \frac{(m-1)s_Y^2}{\sigma^2}}{m+n-2}}}$$.
It is standard fare in many statistics textbooks that (under the specific conditions we lay out here, but not generally)
$$\frac{(n-1)s_X^2}{\sigma^2} + \frac{(m-1)s_Y^2}{\sigma^2} \sim \chi_{m+ n-2}^2.$$
Further, by properties of the normal distribution,
$$\frac{\bar{X} - \bar{Y} - (\mu_X - \mu_Y)}{\sigma\sqrt{\frac{1}{n} + \frac{1}{m}}} \sim \mathcal{N}(0,1).$$
So, indeed, by our definition at the very beginning, $T \sim t_{m+n-2}$.

Now, to address your question (if it's not obvious already!) We assume that our samples are normal because it allows us to say
$$\frac{\bar{X} - \bar{Y} - (\mu_X - \mu_Y)}{\sigma\sqrt{\frac{1}{n} + \frac{1}{m}}} \sim \mathcal{N}(0,1)$$
and
$$\frac{(n-1)s_X^2}{\sigma^2} + \frac{(m-1)s_Y^2}{\sigma^2} \sim \chi_{m+ n-2}^2$$
exactly. There are no approximations here. This is because each sample is normally distributed and independent of each other, so their sums, averages, differences are normally distributed too! Therefore, our statistic $T$ is distributed exactly $t_{m+n-2}$, by our definition at the beginning. This is the detail which your teacher is referencing.
Clearly, though, if your sample size is large enough, we can appeal to the central limit theorem and say that the distribution of the sample means of each of your samples are approximately normal. Therefore, $T$ is approximately $t_{m+n-2}$. For many applications, this is good enough!
