The one-sample t-Test/z-test and inference validity The CLT tells us that as we collect the means of different samples, the sampling distribution resembles a normal distribution and this way we can infer with a CI on the sampling distribution, the population mean. Now, this can be done via bootstrapping or manual sampling, in both cases the inference is totally valid.
Now, the one-sample t-test also allows you to infer on the population mean with a confidence interval, but how is it valid if it doesn't sample with replacement (bootstrapping) or with manual sampling; meaning; how is the t-test able to infer without incurring in the required additional sampling process that the CLT needs, stated in the previous paragraph? The same question applies to a proportion z-test on categorical data.
 A: The CLT says the sampling distribution of the sample mean is asymptotically normal.
From a single sample, we can estimate the mean and the standard error.  These are our best guesses for each parameter. Given that each is an unbiased estimate for the mean and the standard deviation of the sampling distribution, we apply them as if they were the truth.
So, the knowledge of the asymptotic behaviour coupled with unbiased (and hopefully high precision) estimates allows us to make the inference.  Inference in the case where we have a null about the mean is even simpler.  We have a null (the mean is 0) and since we know the asymptotic behaviour of the sampling distribution, we can compute p values for seeing test stats at least as large as what we've seen.
A: I think one thing that is slightly tricky with the CLT is the word “asymptotic”. I used to think that it meant infinity in sample size, causing confusion as to why you would get a normal distribution rather than a single dererministic value (leaving aside whether infinte sampling is even possible, for example it might take a long time before your results become available!) But the CLT includes this “infinte” case and also where you have merely very large samples.
Under certain conditions the CLT says that the sampling distribution of the mean is asymptotically normal, with the same mean as the underlying distribution of the variables but with a variance that is $1/\sqrt{N}$ times as big as the underlying standard deviation. Note that as $N$ tends to infinity the standard deviation shrinks and you would eventually get to zero.
Once you have enough sample size to be in the asymptote, you are in normal distribution territory for the sample mean. But you may well need to use the t-distribution if you want to estimate the mean as you will unlikely know the standard deviation.
Welcome someone like @kjetil-b-halvorsen if would want to make any comments or corrections on the use of the informal summary here.
