Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.
Sign up to join this community
Often a theoretical text will say something like, 'a probability distribution may be used to model the data' or, 'assume a probability distribution such as normal or Lognormal for the outputs'.
What is being conveyed when someone mentions that 'a distribution is being used to model the outputs'?
By assuming some probability for our data we mean that we use some statistical distribution (i.e. mathematical function) as approximation of our data.
Sir Francis Galton in 1886 used data on parents height and their adult children height to describe regression towards mediocrity. This was one of the first times when correlation and regression was used. We can use this data to illustrate how distribution can be used to describe data.
We can approximate parents heights distribution by a normal distribution. As you can see from the plot below, normal distribution (red line) does not fit perfectly to the data, but it fits reasonably well. You could ask why would we like to use any approximation in here? The major advantage is that it makes things simpler because we can describe whole distribution of heights just by two parameters: $\mu$ for mean height and $\sigma$ for standard deviation. Knowing these two parameters not only lets us approximately re-create the data values, but also allows us to assign probabilities to values that were not observed in our dataset.
Galton used this data to show regression to the mean phenomenon. What he described is that heights of parents and their children are related and there is some rate that describes the relation. The rate of how parent's height influences child's height can be summarized using a single number $\rho$ for correlation coefficient or regression slope. The $\rho$ parameter can be also thought as a parameter of bivariate normal distribution. So in this case, distribution not only approximates the individual variable, but also relation of two variables, i.e. it leads to a statistical model of the observed phenomenon.
This leads us to replace the original counts
with a statistical distribution (red-yellow blob marks kernel density and gray ellipses mark bivariate normal density)
By saying that we assume distribution for data we mean that we use this distribution to build a model. Such model is obviously wrong, because it is an approximation, but it is useful, since it describes the complicated phenomenon in simple mathematical terms.
Galton, F. (1886). Regression Towards Mediocrity in Hereditary Stature. Journal of the Anthropological Institute, 15, 246-263.
Hanley, J. A. (2004). “Transmuting” women into men: Galton's family data on human stature. The American Statistician, 58(3), 237-243.
