1
$\begingroup$

I am trying to explain/understand how one gets from sample observations to making inferences about a population parameter. I want to make sure I explain the logic correctly. Let's take the most beginner example where we have a normal population, a known variance ex ante, and we want to make inferences about the population mean.

When learning, a lot of "properties" are thrown around like CLT, law of large numbers, etc... I am hoping to focus on the most relevant properties. After re-reading a few times, seems like CLT, the fact that the sampling distribution of the sample mean is normally distributed, etc... are just intermediary properties (basically steps within a proof) - not the property that allows us to ultimately make inferences.

What really matters for hypothesis testing is the test statistic itself. The test statistic relates our sample estimate and some value. Usually, we like to substitute what we think the population parameter may be for the value. In this case the test statistic is the Z-statistic. For example, when we look at this:

$Z = \frac{\bar{x} - \mu}{\frac{\sigma}{\sqrt{n}}}$

When it comes to interpretation, what we should be thinking is that given $\bar{x}, n, and \sigma$, the expression above gives us the likelihood of $\mu$ being certain values. The likelihood is given by the standard normal PDF.

So I guess this brings me to my question. If I wanted to do significance testing for any population parameter, would this be adequate steps?

  1. Determine a way to estimate the population parameter using sample data. Usually, do we need our sample estimate to be unbiased? What happens if we can't find an unbiased estimate?
  2. After doing that, find some sort of expression that relates your estimate with the population parameter so that that expression follows a well known distribution.
  3. Assuming the test statistic follows a continuous distribution, the test statistic will give us the likelihood of our population parameter taking on certain values.

I know this is a bit overthinking, but when I learned this stuff, we just learned how to make inferences with the Z-statistic via calculate some formulas and then look it up on a Z-table. I just tried to explain it away by saying some stuff like the normal distribution is symmetric and we are basically standardizing some stuff, etc.... I never learned to view the test statistic as an expression that tells us, given sample data, this is the likelihood of our population parameter being specific values. As a result, there was always a gap in logic as to how we actually are able to make inferences about the population using our sample.

$\endgroup$
2
  • $\begingroup$ Also, now that I think about it, that seems like a very Bayesian way to explain things (since the population parameter is almost like a random variable). Is there a more frequentist way to explain stuff? $\endgroup$ – confused Sep 20 '20 at 19:46
  • $\begingroup$ I'll probably just say, the test statistic gives us the likelihood (or probability) of our sample estimate and population paramter being x far away from each other. And the user can decide if the population paramter is random or the sample statistic is random - whichever way they want to view it. $\endgroup$ – confused Sep 20 '20 at 19:52
1
$\begingroup$

Regarding $Z=\frac{\bar x-\mu}{\frac{\sigma}{\sqrt{n}}}$, the numerator is the (possibly negative) distance between the observed $\bar x$ and the $\mu$ from the null hypothesis ($H_0$). The denominator standardises this so that the distribution of $Z$ does not depend anymore on the variance $\sigma^2$ (because otherwise the bigger $\sigma^2$, the bigger a difference $\bar x-\mu$ would be expected).

So $Z$ is a standardised distance between observed $\bar x$ and $\mu$, which would be the true parameter if $H_0$ were true. It is not a likelihood! Now if $H_0$ is indeed true, we can expect $|Z|$ to be small. If $|Z|$ is so big that under $H_0$ it would be very unlikely, we take this as evidence against the $H_0$. The probability that under $H_0$, $|Z|$ is larger than what was actually observed is called p-value(*). If the p-value is very small, that's a (more or less) strong indication against $H_0$. Note that this is a probability for data, assuming that $H_0$ is true. It is not a probability (or likelihood) that $H_0$ is true. Note also that all this does not require $H_0$ to be in fact true, and non-rejection will not prove that it is; it only serves investigating whether the data is compatible with $H_0$.

"1 Determine a way to estimate the population parameter using sample data. Usually, do we need our sample estimate to be unbiased? What happens if we can't find an unbiased estimate?" This is not a problem in principle as long as (a) the distribution of the test statistic can be evaluated (your item 2), and (b) a large distance between the estimator and the parameter still indicates incompatibility of the data with the model.

"2 After doing that, find some sort of expression that relates your estimate with the population parameter so that that expression follows a well known distribution." Correct though sometimes not possible, in which case the distribution can often be simulated in one way or another. Note however that there are some (nonparametric) tests that are defined in different ways.

"3 Assuming the test statistic follows a continuous distribution, the test statistic will give us the likelihood of our population parameter taking on certain values." No, see above.

(*) Actually I have defined a p-value for a two-sided test here; both big positive and big negative values of $Z$ indicate against the model. In a one-sided test one would check whether $Z$ is too big, too small, respectively, depending on whether one would want to detect bigger or smaller $\mu$ than under $H_0$.

$\endgroup$
7
  • $\begingroup$ I guess I was a bit unclear but my understanding is test statistic itself is not the likelihood, but it is is linked to the likelihood where if you plug the test statistic into the PDF, you get the likelihood. For example, feeding the z-stat into the z-distribution's PDF (standard normal distribution), you can get a likelihood. The z-table is just a CDF as opposed to PDF. $\endgroup$ – confused Sep 21 '20 at 9:48
  • 1
    $\begingroup$ That's not wrong but really doesn't have much implication for the workings of the test. $\endgroup$ – Lewian Sep 21 '20 at 10:50
  • 1
    $\begingroup$ In fact hypothesis tests are criticised regularly by people who hold the "likelihood principle" that roughly states that all information regarding the parameter is in the likelihood. For tests the likelihood at the observed value of $z$ is irrelevant, because it does not enter the computation of $P\{|Z|>z\}$ (at least not if the distribution of $Z$ is continuous). $\endgroup$ – Lewian Sep 21 '20 at 10:53
  • $\begingroup$ Well yes for tests you are using the CDF as opposed to PDF so likelihood doesn't show up. I guess I am just trying to explain how we are able to make inferences about the population parameter intuitively, since when I learned it we skipped some steps. The test-statistic links our sample data with our hypothesized value, and then we can rearrange the test-statistic to produce a confidence interval, or we can just use the test-statistic itself + CDF to make probabilistic statements. $\endgroup$ – confused Sep 21 '20 at 11:27
  • 1
    $\begingroup$ My intuition is that $Z$ measures how far away the data are from the hypothesised value, and the probabilistic statement will tell you "how far is too far" in the sense of too unlikely to happen if the $H_0$ were true. (Ah! I'm using the word "unlikely" here, but I really mean just "too low probability"). $\endgroup$ – Lewian Sep 21 '20 at 12:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.