2
$\begingroup$

What are the traditional distributions for assigning probabilities to model parameters? For instance, assume that we have a binomial distribution: $$y \sim Bin(n,\theta)$$ Then we can distribute $\theta$ parameter with a Beta distribution and parameters $\alpha$ and $\beta$ equal to 1. $$\theta \sim Beta(1,1)$$ I consider that if we would like to derive the distribution of the mean of a Gaussian probability distribution, then we could use the central limit theorem and assume that: $$\mu \sim N(0,1)$$ But what about other distributions? For instance, what is the distribution of $\lambda$ in the Weibull distribution? Or what is the distribution of $k$ for Chi-squared distribution? I'm not sure where to find papers on this topic because in most cases the usage of techniques involving parameters distributions seems like more of an intuitive methodology than some kind of rule.

Update: I'm asking this question for the Bayesian parameters estimation and Bayesian model selection, where I calculate posterior with: $$P(\theta|y) = P(y|\theta)P(\theta)$$ So I need to find the prior $P(\theta)$, that's what I was interested in.

Improve this question
$\endgroup$
10
  • 1
    $\begingroup$ This is about conjugate priors for the exponential family of probability distributions (which includes many distributions): people.eecs.berkeley.edu/~jordan/courses/260-spring10/… $\endgroup$
    – Eskapp
    Commented Dec 15, 2016 at 20:47
  • 1
    $\begingroup$ @Eskapp thanks a lot for the link, will definitely work on this paper. $\endgroup$
    – olejnik_
    Commented Dec 15, 2016 at 20:52
  • 1
    $\begingroup$ Wikipedia has a useful table for conjugate priors en.wikipedia.org/wiki/Conjugate_prior $\endgroup$
    – Jon
    Commented Dec 15, 2016 at 21:31
  • 1
    $\begingroup$ The beta is a conjugate prior to the binomial. But nowhere in the question do you mention Bayesian methods. Maybe one can infer it since you are talking about distributions (presumably posterior distributions) for parameters. But it is puzzling to me when you talk about the mean of a normal distribution where you invoke the central limit theorem but don't mention any prior. $\endgroup$
    – Michael R. Chernick
    Commented Dec 15, 2016 at 22:09
  • 1
    $\begingroup$ So $\alpha$ and $\beta$ are your prior parameters. Those do not come directly from your data. There are a few ways to estimate those prior parameters using your observed data set, but they're generally supposed to come from "prior" information. $\endgroup$
    – Jon
    Commented Dec 18, 2016 at 19:34

1 Answer 1

Reset to default
3
$\begingroup$

You seem to be asking for the distribution to model certain parameters. However, there is no single distribution for certain types of parameters. There are common distributions used for parameters such as the Beta distribution to model a probability. No matter how common they are, they may not be the most accurate distribution to describe your problem. Oftentimes, the "common" distributions for parameters are common due to convenience of some sort. Either they're conjugate or have other nice mathematics associated with them.

I would caution against using these distributions just for convenience sake as there oftentimes better distributions for your task. For example, the Beta distribution cannot be made to be trimodal. If you're modeling probabilities that you think must be in a neighborhood of 0.0, 0.5, or 1.0 this may not be sufficient. Another example is the Normal distibrution as a prior, which has really small mass in extreme values. Oftentimes, these extreme values aren't really as implausible as your Normal prior is suggesting (t-distribution is a good alternative). If you want to really put good priors on your parameters, you're going to have think about your problem and think about what makes sense and not just simply go with what's common.

Now, there are good reasons to go with the common distributions for reasons other than they most accurately describe your problem. It's entirely possible that you may have to make some sacrifices for computational or numerical efficiency, but to default to these types of "common" distributions I think is misguided.

Improve this answer
$\endgroup$
4
  • $\begingroup$ Thank you for your answer. For now I'm not good with statistics (have more experience with machine learning than with Bayesian approaches, for instance), so I think that looking for most common techniques as for the first step will be okay for me as long as I'm not proficient enough to encode my own solutions from scratch for particular tasks. $\endgroup$
    – olejnik_
    Commented Dec 15, 2016 at 21:49
  • 1
    $\begingroup$ When applying Bayesian methods conjugate priors were a convenience because they give closed form distributions for the posterior. With the advent of Markov Chain Monte Carlo there is much more flexibility available to pick priors for parameters and hence different posterior distributions can occur. But @olejnik think back to your question. Did you intend to look at the Bayesian approach? Did you intend to suggest conjugate priors? Lastly was the N(0,1) distribution for the mean parameter intended to be a prior or posterior distribution. $\endgroup$
    – Michael R. Chernick
    Commented Dec 15, 2016 at 22:21
  • $\begingroup$ I am not sure that the answer given completely answers what you want. Also you don't mention which distribution (prior or posterior) you are referring to for the chi square and Weibull parameters. $\endgroup$
    – Michael R. Chernick
    Commented Dec 15, 2016 at 22:22
  • $\begingroup$ @MichaelChernick for now I'm working on the Bayesian approaches, I mean Bayesian model selection and Bayesian parameter estimation, where I calculate posterior $P(\theta | y) = P(y | \theta)P(\theta)$. The distribution I need is prior, $P(\theta)$. I already understood that I need to work with conjugate priors, thank you for this advice. The $N(0,1)$ distribution was meant to be the prior. $\endgroup$
    – olejnik_
    Commented Dec 16, 2016 at 13:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.