As @Antoine R pointed out the idea is that sample means provide an unbiased estimate of the population mean. If we could sample unlimited samples of one single observation (or subject) (n = 1) we would reproduce the entire population, and its exact mean.
I wanted to contribute some simulations to illustrate the point about consistency: the sample mean converges to the population mean as the number of observations tends to infinity. Conversely, sample means with small number of observations can deviate substantially, perhaps explaining the counterintuitive impression you describe in your question.
If we obtain 10,000 samples out of a standard normal ($N(0,1)$) distribution each sample containing one additional subject than the previous, and we plot the results we get the following plot,
sample_means <- NULL
for (i in 1:1000) sample_means[i] <- mean(rnorm(i))
plot(sample_means, cex = 0.2, pch = 19, col = 'red')
showing the convergence to the true mean (0) with increasing sample size.
Or, another way of play with the concept would be to obtain 10,000 samples with n = 10, and then 10,000 samples with n = 10,000, plotting the histograms without altering the x axis limits,
s_means_10 <- NULL
for(i in 1:1e4) s_means_10[i] <- mean(rnorm(10))
hist(s_means_10,col="dark red", border = 'white', main = "Distr. Means of Samples of 10 rand. numbers")
s_means_10k <- NULL
for(i in 1:1e4) s_means_10k[i] <- mean(rnorm(1e4))
hist(s_means_10k,xlim=c(-4,4),border = "dark red",col="dark red", main = "Distr. Means of Samples of 10,000 numbers")
showing again the convergence with sample size, without any shift in the mean, which is always centered at zero.