Understanding the test statistic in hypothesis testing?

Question

I am trying to explain/understand how one gets from sample observations to making inferences about a population parameter. I want to make sure I explain the logic correctly. Let's take the most beginner example where we have a normal population, a known variance ex ante, and we want to make inferences about the population mean.

When learning, a lot of "properties" are thrown around like CLT, law of large numbers, etc... I am hoping to focus on the most relevant properties. After re-reading a few times, seems like CLT, the fact that the sampling distribution of the sample mean is normally distributed, etc... are just intermediary properties (basically steps within a proof) - not the property that allows us to ultimately make inferences.

What really matters for hypothesis testing is the test statistic itself. The test statistic relates our sample estimate and some value. Usually, we like to substitute what we think the population parameter may be for the value. In this case the test statistic is the Z-statistic. For example, when we look at this:

$Z = \frac{\bar{x} - \mu}{\frac{\sigma}{\sqrt{n}}}$

When it comes to interpretation, what we should be thinking is that given $\bar{x}, n, and \sigma$, the expression above gives us the likelihood of $\mu$ being certain values. The likelihood is given by the standard normal PDF.

So I guess this brings me to my question. If I wanted to do significance testing for any population parameter, would this be adequate steps?

1. Determine a way to estimate the population parameter using sample data. Usually, do we need our sample estimate to be unbiased? What happens if we can't find an unbiased estimate?
2. After doing that, find some sort of expression that relates your estimate with the population parameter so that that expression follows a well known distribution.
3. Assuming the test statistic follows a continuous distribution, the test statistic will give us the likelihood of our population parameter taking on certain values.

---

I know this is a bit overthinking, but when I learned this stuff, we just learned how to make inferences with the Z-statistic via calculate some formulas and then look it up on a Z-table. I just tried to explain it away by saying some stuff like the normal distribution is symmetric and we are basically standardizing some stuff, etc.... I never learned to view the test statistic as an expression that tells us, given sample data, this is the likelihood of our population parameter being specific values. As a result, there was always a gap in logic as to how we actually are able to make inferences about the population using our sample.

Answer 1

Regarding $Z=\frac{\bar x-\mu}{\frac{\sigma}{\sqrt{n}}}$, the numerator is the (possibly negative) distance between the observed $\bar x$ and the $\mu$ from the null hypothesis ($H_0$). The denominator standardises this so that the distribution of $Z$ does not depend anymore on the variance $\sigma^2$ (because otherwise the bigger $\sigma^2$, the bigger a difference $\bar x-\mu$ would be expected).

So $Z$ is a standardised distance between observed $\bar x$ and $\mu$, which would be the true parameter if $H_0$ were true. It is not a likelihood! Now if $H_0$ is indeed true, we can expect $|Z|$ to be small. If $|Z|$ is so big that under $H_0$ it would be very unlikely, we take this as evidence against the $H_0$. The probability that under $H_0$, $|Z|$ is larger than what was actually observed is called p-value(*). If the p-value is very small, that's a (more or less) strong indication against $H_0$. Note that this is a probability for data, assuming that $H_0$ is true. It is not a probability (or likelihood) that $H_0$ is true. Note also that all this does not require $H_0$ to be in fact true, and non-rejection will not prove that it is; it only serves investigating whether the data is compatible with $H_0$.

"1 Determine a way to estimate the population parameter using sample data. Usually, do we need our sample estimate to be unbiased? What happens if we can't find an unbiased estimate?" This is not a problem in principle as long as (a) the distribution of the test statistic can be evaluated (your item 2), and (b) a large distance between the estimator and the parameter still indicates incompatibility of the data with the model.

"2 After doing that, find some sort of expression that relates your estimate with the population parameter so that that expression follows a well known distribution." Correct though sometimes not possible, in which case the distribution can often be simulated in one way or another. Note however that there are some (nonparametric) tests that are defined in different ways.

"3 Assuming the test statistic follows a continuous distribution, the test statistic will give us the likelihood of our population parameter taking on certain values." No, see above.

(*) Actually I have defined a p-value for a two-sided test here; both big positive and big negative values of $Z$ indicate against the model. In a one-sided test one would check whether $Z$ is too big, too small, respectively, depending on whether one would want to detect bigger or smaller $\mu$ than under $H_0$.