# Goodness of fit to a fitted distribution

Assume we have a sample of data $$\{x_1, \dots, x_n\}$$ and a family of distributions $$(f_\theta)_\theta$$ indexed by some parameter vector $$\theta$$. We would like to fit $$\theta$$ to $$\{x_1, \dots, x_n\}$$ to obtain $$\hat{\theta}$$, then assess the goodness of fit of $$f_{\hat{\theta}}$$ to $$\{x_1, \dots, x_n\}$$.

Is there a general way to do this? Maybe for certain families of distributions? For specific ways of estimating $$\hat{\theta}$$?

The problem is of course that we are using the data we would like to assess the fit of to fit the distribution in the first place. In the specific case where $$(f_\theta)$$ is the family of normal distribution, parameterized by the mean and variance, $$\theta=(\mu, \sigma^2)$$, we can use the Lilliefors test. Is there something that works for other distribution families?

I had originally thought that a Probability Integral Transform (PIT) might work, but a quick simulation shows that it doesn't - the p value histograms look uniform enough in the middle, but they have too little mass for $$p\approx 0$$ and $$p\approx 1$$:

n_sims <- 1e4
nn <- 20

pp_pit_specified <- pp_pit_estimated <- matrix(NA,nrow=n_sims,ncol=20)
pb <- winProgressBar(max=n_sims)
for ( ii in 1:n_sims ) {
setWinProgressBar(pb,ii,paste(ii,"of",n_sims))
set.seed(ii)
sim <- rnorm(nn)
pp_pit_specified[ii,] <- pnorm(sim,mean=0,sd=1)
pp_pit_estimated[ii,] <- pnorm(sim,mean=mean(sim),sd=sd(sim))
}
close(pb)

opar <- par(mfrow=c(1,2))
hist(pp_pit_specified,main="Parameters specified",xlab="",col="lightgray")
hist(pp_pit_estimated,main="Parameters estimated",xlab="",col="lightgray")
par(opar)

This is motivated by Non-uniform distribution of p-values. No, I don't have a specific use case in mind, I'm just curious.

• Could you state more specifically what you mean by "assess the goodness of fit"? In other threads I have noted that different people seem to have different understandings of the term "goodness of fit." In particular, it's apparent you have in mind something more than, say, the usual machinery of MLE to test hypotheses about $\theta.$ But what exactly? – whuber Jul 3 at 22:41
• @whuber: that's a good question. I'm not completely clear about what precisely I want. It could be something NHST, with some kind of test statistic (as in, yes, we could ML fit a normal distribution to beta(0.1,0.1) data, but the distributional fit would make no sense whatsoever). – S. Kolassa - Reinstate Monica Jul 4 at 6:20