Testing if a sample follows a given distribution Suppose I have a single sample statistic $y$ (e.g., a scalar computed from the data collected after conducting an experiment) and would like to "test" if $y$ follows a given probability density function $p(x)$ (this is of course ill-posed - see below).

*

*How can I formally express this as a hypothesis testing problem? Also, is this unique? i.e., are there multiple ways of casting this task (described in words as "test if $y \sim p(x)$") as hypothesis testing problems?

*How do I do this for the special cases of (a) $p(x)= \mathcal{N}(\mu,\sigma^2)$ and (b) $p(x) = \chi^2_k$?

 A: There are various ways in which to test whether a sample
might have been randomly chosen from a given distribution.
These might include checking to see if the sample mean is
consistent with the population, and similarly for variances, or other parameters.
I interpret your question to seek a test whether the
distribution of the sample (perhaps as expressed in its
empirical CDF) is consistent with the population distribution.
For example, suppose we happen to have a sample of size
$n = 500$ from an exponential distribution with mean $2$ and rate $1/2$ and so also standard deviation $2.$ A fictitious sample to these specifications is sampled and summarized in R below.
set.seed(2022)
x = rexp(500, 1/2)
summary(x);  sd(x)
     Min.   1st Qu.    Median      Mean   3rd Qu.      Max. 
 0.007349  0.569684  1.371701  1.987886  2.703088 12.677304 
[1] 1.923795  # sample SD

Not knowing how the sample was obtained, you might
have guessed that it could be from the population $\mathsf{Norm}(\mu=2, \sigma=2)$.
boxplot(x, horizontal=T, pch=19, col="skyblue2")


The shape of the boxplot already suggests
that this is not a normal sample.
One could use a Kolmogorov-Smirnov goodness-of-fit test
in R to test the null hypothesis that the population is
$\mathsf{Norm}(\mu=2, \sigma=2).$ This null hypothesis
is strongly rejected with P-value very nearly $0.$
ks.test(x, pnorm, 2, 2)

        One-sample Kolmogorov-Smirnov test

data:  x
D = 0.15955, p-value = 1.762e-11
alternative hypothesis: two-sided

More generally, if you had no idea of the population mean and
standard deviation, you could use one of several tests
to see if the sample is consistent with any normal distribution. One such test is the Shapiro-Wilk test, shown below using R; the null hypothesis of normality is strongly rejected.
shapiro.test(x)

        Shapiro-Wilk normality test

data:  x
W = 0.83826, p-value < 2.2e-16


Note: It is important to remember that the K-S test
is for a particular population distribution with all
parameters known. It is cheating to estimate the parameters
from the sample.
The K-S test does not reject the null hypothesis that the data x are from $\mathsf{Exp}(\mathrm{rate}= 1/2),$ but
strongly rejects that the population might have been
$\mathsf{Exp}(\mathrm{rate}= 1).$
ks.test(x, pexp, 1/2)

        One-sample Kolmogorov-Smirnov test

data:  x
D = 0.020534, p-value = 0.9843
alternative hypothesis: two-sided

ks.test(x, pexp, 1)

        One-sample Kolmogorov-Smirnov test

data:  x
D = 0.25929, p-value < 2.2e-16
alternative hypothesis: two-sided

The K-S test compares the empirical CDF (ECDF) of
the sample with the CDF of the population distribution. The test statistic $D$ is the maximum vertical distance between the two.
The plots below illustrate a good fit between ECDF and CDF at left, and a poor fit at right.

R code for figure:
par(mfrow=c(1,2))
 hdr1="Good fit to EXP(1/2)"
 plot(ecdf(x), main=hdr1)
  curve(pexp(x, 1/2), add=T, col="green3", lwd=3)
 hdr2="Bad fit to EXP(1)"
 plot(ecdf(x), main=hdr2)
  curve(pexp(x, 1), add=T, col="red", lwd=2)
par(mfrow=c(1,1))

A: All standard hypothesis testing follows the following steps:

*

*Formulate a null hypothesis. This should rigorously state conditions under which one can calculate probabilities (e.g. "The observations are all drawn from independent normal distributions with $\mu = 0$ and $\sigma = 1$.")

*Choose an $\alpha$

*Choose a statistical variable $X$

*Collect data

*Calculate the statistic $x$ from the data

*Reject the null if $P(X\ge x|H_0)\le\alpha$ (Note that $X$ is a random variable with a probability distribution, while $x$ is a particular value that that variable takes; $P(X\ge x|H_0)$ is the probability, given the null hypothesis, of getting something as or more extreme than what you actually saw.)

So to test whether a sample follows a given distribution, you need some metric for measuring "distance" from a given distribution, and then you use that as your statistic.

How can I formally express this as a hypothesis testing problem?

All you need is some function $f: ((\mathbb R \rightarrow \mathbb R) \times \mathbb R ^n) \rightarrow \mathbb R$. That is, a function that takes an ordered pair as input and gives a real number as output, where the first element of the ordered pair is a PDF and the second element is a sample of $n$ observations. Once you have such a function, you can find what that function gives for your sample, then calculate the probability, given the null hypothesis (the null hypothesis in this case is that the sample came from the proposed distribution) that you would get a sample for which the function would give a result greater than or equal to what you actually got.

Also, is this unique?

No. Every function gives a different test. Two attributes to look for in choosing the function are tractability (how easy is it to calculate $P(X\ge x|H_0)$) and power (how well does the function correspond to what we think of "agreeing" with the distribution). If you think someone is deliberately trying to fake a particular distribution, then you might want a function tailored to what errors you think they might make (for instance, someone faking a normal distribution might focus on getting the mean and sd correct, but ignore the kurtosis).
