3
$\begingroup$

this is a conceptual question. In data mining, the problem often arises that scientists/data engineers are using an expert guess for the underlying distribution of their data. Often their assumption is based on previous results of academic research and does not really reflect the real distribution of the dataset.

However, I would like to get the real distribution of my dataset? Any recommendations where to start? Does this even make sense to get the real distribution? Can thisv task be automated?

I appreciate your answer!

btw I am mostly using R as my data analysis tool

$\endgroup$
3
  • 3
    $\begingroup$ This is a frequent question here. Generally speaking, you can't. Let's say your data look exponential. That doesn't mean they are exponential. You can never prove it by looking at the data. As you look at larger and larger samples sizes, you may be able to exclude many alternatives, but there will always be an infinite number of alternatives that will fit the data about as well even if the data are truly exponential. $\endgroup$
    – Glen_b
    Commented Aug 17, 2014 at 9:46
  • 1
    $\begingroup$ E.g. see the first few paragraphs here, and some of the comments here $\endgroup$
    – Glen_b
    Commented Aug 17, 2014 at 9:56
  • $\begingroup$ You can sometimes find out when data is inconsistent with some distribution, but you can't say that it is from some distribution. $\endgroup$
    – Glen_b
    Commented Aug 17, 2014 at 10:04

2 Answers 2

2
$\begingroup$

There is no "real distribution" of your data. There are models of the distribution of your data, and some are more useful than others. There are several ways to compare distributions.

  1. Fit distributional models to data using maximum likelihood. Compare the models based on their information criterion scores (e.g., AIC, BIC, DIC, WAIC).

  2. Fit distribution models to data using Bayesian methods. Once you have recovered the Bayesian posterior distribution, choose a series of test statistics, which are not parameters or sufficient statistics in the distribution model. Compute the posterior probability that the test statistic is at least as extreme as what is found in the data. The distribution model with the fewest "significant" Bayesian p-values is the best model.

Of course, there is also the possibility of including all of the distribution models within some super Bayesian estimation model, which includes parameters dictating the relative weights of each of these models. That would be the best thing to do, but is often completely unfeasible.

$\endgroup$
2
$\begingroup$

The topic you're looking for is 'density estimation'.

The first thing to note is that you will probably not find that a mathematically familiar distribution fits your data (unless you work on radioactive decay, when the Poisson really is what you want...)

The second thing to note is that one may, subject to various theorem-specific technical conditions, approximate any distribution arbitrarily well given enough data with a mixture of more familiar distributions e.g. from the exponential family.

For example, a simple approach to estimating the distribution of continuous data might be a finite mixture of Gaussians. Maximum Likelihood estimates for parameters are available using an EM algorithm. (EM algorithms are easiest to implement for exponential family). Naturally other methods are possible. McLachlan et al. 2000 is a good place to start reading. The R package flexmix will help you fit these models.

The basic issues that arise in this exercise involve not having enough samples in all parts of the data space - the curse of dimensionality - and consequently the question of how to trade-off bias and variance in one's density model.

Density estimation is in this sense inherently harder than building conditional models such as regression and classification. For example, in the latter problem one needs only to characterise the area of the data distributon where the class decision boundaries lie rather than all of it.

Finally, note that for many problems the distribution of your data is not relevant. A standard motivation for regression modeling is to observe that some P(X, Y) can be decomposed into P(X) the density of the input data and P(Y | X) the conditional density of the target data. If it is further assumed that the parameters governing these two distributions are a priori independent (as they will be when distinct mechanisms assign X and generate Y from it) then the density P(X) can be ignored when trying to learn about P(Y | X), which is convenient because the latter is typically a much lower dimensional distribution to characterise.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.