A little change in notation might help in answering your question: What you call $\mu$ is often called $\mu_{0}$ because it is the population mean under the null hypothesis, whereas $\mu$ is the actual population mean - which is unknown because we don't know whether the null hypothesis is true. Also, what you call $\sigma$ is usually called $s$ following the convention that population parameters get greek letters, and sample parameters are denoted by latin letters.
Note that $s / \sqrt{n}$ is the standard error of the mean $\bar{X}$ (SEM) - an estimate of the variablity of $\bar{X}$, where $\bar{X}$ is understood as a random variable. So we have
$t = \frac{\bar{X} - \mu_{0}}{s / \sqrt{n}}$
Now, for a given sample, you have a fixed empirical $\bar{X}_{emp}$, and thus a fixed difference $d_{emp} = \bar{X}_{emp} - \mu_{0}$. Part of the confusion seems to be related to the idea that "a bigger sample size (higher $n$) should give sample mean closer to population mean". This should be rephrased to conditional on the null hypothesis being true ($\mu = \mu_{0}$), the probability of observing a difference $d = \bar{X} - \mu_{0}$ that is at least as large as the already observed $d_{emp}$ becomes smaller when $n$ increases. This is because the "accuracy" of our estimator $\bar{X}$ then increases (variability decreases).
I guess the main point is that you already have a fixed $\bar{X}_{emp}$ and thus $d_{emp}$, and $t$ just tells you how "big that difference is" measured in (estimated) units of variability of $\bar{X}$. When the units become smaller in absolute numbers, the same absolute difference $d_{emp}$ will "be worth more units" and will thus count as "more suprisingly high" (= less likely to occur) if $\mu = \mu_{0}$.
