# Based on the ideas of Parameter Estimation and Fitting Probability Distributions, what stops us from making any function be a PDF(PMF)?

Currently I am doing an introduction to parameter estimation and fitting probability distributions to sets of data. So in a small synopsis what I understand the whole process to be like is the following:

1) We collect a large amount of raw data, which comes from an underlying probability distribution. We then "graph" the data (perhaps in the form of a bar chart or something similar at least in the 2D and 3D cases).

2) Observing this visual presentation we go through our list of existing probability distributions and form an opinion on which distribution appears to fit the data most precisely.

3) We then take a large sample from this data and attempt to estimate the parameters of our chosen probability distribution by using the array of techinques available at our disposal.

I have a few questions:

i) Is the outline above the procedure used to get parameter estimates?

ii) (more important) What stops us from making any function a probability distribution? What I mean is we have this visual representation of the data, perhaps none of the known probability distributions that we have presently align with the data. What stops us from just saying "this continuous function will now be a distribution as long as it satisfies the necessary axioms." Is there something more rigourous to this? (perhpas I just haven't arrived there yet in my studies).

• To your last question: A probability distribution do not need to be any of the "named" distributions. Any nonnegative function that integrates to 1 can be used as a probability density. Maybe you should look for nonparametric methods. – kjetil b halvorsen Feb 3 '19 at 21:36

1) We collect a large amount of raw data, which comes from an underlying probability distribution. We then "graph" the data (perhaps in the form of a bar chart or something similar at least in the 2D and 3D cases).

If the data has more then two dimensions (usually the case), then you cannot graph it. You can graph only the marginal distribution, but not the joint distribution.

2) Observing this visual presentation we go through our list of existing probability distributions and form an opinion on which distribution appears to fit the data most precisely.

No. First of all, as stated above, graphs don't tell you the whole story. Second, many distributions can look very similar. Third, there is no such a thing as "list of existing distributions". You can go through the list of popular distributions, but the list of all possible distributions is infinite (you can come up with your own distribution, you can define mixtures of any number of any distributions -- this alone makes the list infinite).

Usually based on what you know about the data (given plots, summary statistics, knowledge on what the data represents, how it was collected) you choose some distribution or few distributions that make sense for this data. For example, if it is a count of independent binary things in fixed number of trials, then most likely you will be using binomial distribution. To understand better when what distributions make sense, you can check the Statistics 110 lectures by Joe Blitzstein.

Moreover, even if you would try several different distributions, then you wouldn't do it based on how the data looks, but rather based on model fit statistics (see questions tagged as ).

3) We then take a large sample from this data and attempt to estimate the parameters of our chosen probability distribution by using the array of techinques available at our disposal.

Generally yes, if possible.

ii) (more important) What stops us from making any function a probability distribution? What I mean is we have this visual representation of the data, perhaps none of the known probability distributions that we have presently align with the data. What stops us from just saying "this continuous function will now be a distribution as long as it satisfies the necessary axioms." Is there something more rigourous to this? (perhpas I just haven't arrived there yet in my studies).

If the function follows the mathematical definition of probability density function, or probability mass function, then it is the function. Usually it is not about finding the distribution that looks exactly like your data. If you wanted it to look like your data, then you would use empirical distribution function or things like kernel density, that would look exactly like your data. In most cases we choose simpler distributions, that look approximately like the empirical distribution. We use the distributions to build simplified models of reality, that can be extended beyond the data you collected. Saying it differently, you don't want the distribution to overfitt to your data.

Here you can find example: What is meant by using a probability distribution to model the output data for a regression problem?

• Thank you very much for this explanation. It really does provide me with the clarity that I was looking for. – dc3rd Feb 3 '19 at 21:45