11
$\begingroup$

I have been reading some papers regarding statistics and I seem to be confusing the terms priors and likelihood.

Would it be possible to explain the difference between the two terms? I am interested in both a "down-to-earth" approach with examples and the mathematical and statistical aspects.

Thanks.

Improve this question
$\endgroup$
6
  • $\begingroup$ The likelihood is the joint density of the data, given a parameter value and the prior is the marginal distribution of the parameter. Something tells me you're asking something more though-- can you elaborate? $\endgroup$
    – Macro
    Commented May 19, 2012 at 22:51
  • $\begingroup$ @Macro technically you gave a complete answer to the question. Since Andrew is probably not familiar with the Bayesian methodology or at least the terminology i think it is appropriate to elaborate a little. Bayesians do inference based on treating unknown models parameters as having probabilities. The likelihood is a probability density for the data given a value for the parameter. The likelihood can be used by frequentists to do inference about the parameter without making assumptions about the parameter. $\endgroup$
    – Michael R. Chernick
    Commented May 19, 2012 at 23:15
  • $\begingroup$ The Bayesian cannot determine a probability distribution for the parameter based on the likelihood alone. But prior to collecting the data they can form an opinion about the parameter and express it in terms of a probability that the call a prior because it is derived prior to collecting data. Then they can combine the prior with the likelihood to get a posterior distribution for the parameter. It is called posterior because it is obtained after observing the data. In this context applying Bayes rulegives the posterior as an appropriately normalized product of prior with likelihood. $\endgroup$
    – Michael R. Chernick
    Commented May 19, 2012 at 23:24
  • $\begingroup$ Bayesians and frequentist both use the likelihood for inference. The idfference is that the Bayesians add a prior and use a posterior distribution for inference while the frequentists only use the data. Their interpretation of probability is fundamentally different. $\endgroup$
    – Michael R. Chernick
    Commented May 19, 2012 at 23:30
  • $\begingroup$ a frequentist does not "only use the data" its just that the method of using prior information is different, such as the parameterisation they want unbiasness or fisher efficient estimator. prior information for frequentist usually comes in the from of choice of hypothesis to compare, statistic to use, and significance level. $\endgroup$
    – probabilityislogic
    Commented May 20, 2012 at 2:21

1 Answer 1

Reset to default
9
$\begingroup$

The likelihood relates your data to a set of parameters. It is typically written as: $P(D | \theta)$ (or $\mathcal{L}(\theta | D)$ because the likelihood can be viewed as a function of the parameters - holding the data constant).

where $\theta$ contains all of the parameters necessary for the model. For example, consider we have a bunch of iid data $X = \{x_1, ..., x_n\}$ and we want to see how well this fits to a Normal distribution. $\theta = \{\mu, \sigma\}$, and $P(D | \theta) = \prod_i \mathcal{N}(x_i; \mu, \sigma)$. One approach to fitting this model would be to maximize the parameter values according to maximum likelihood. This is exactly what it sounds like. We take the likelihood function, and attempt to maximize it by changing the parameter settings (keeping the observed data constant). This is usually done by computing the derivative of the likelihood w.r.t. each parameter, setting to 0 and solving (side note: it is common to first take the logarithm of the likelihood function to make the derivatives easier to solve).

Alternatively, we could take a Bayesian approach and assign a prior probability distribution over the parameters and compute the posterior distribution to fit the parameters: $P(\theta | D) \propto P(D | \theta) P(\theta)$. In this case we treat the parameters as random variables ad thus must define a distribution over their possible values. The prior distribution can encode any prior knowledge we may have about the variables. For instance, we may have a good idea of the possible ranges for $\mu$ and could thus assign a prior distribution that pushes the $\mu$ slightly toward these values.

To recap: Likelihood: $P(D | \theta)$ links data to parameters Prior: $P(\theta)$ distribution over possible parameter values (used in Bayesian analysis)

Improve this answer
$\endgroup$
2
  • $\begingroup$ Good, but I've always seen it written: $\mathrm P(D\mid\theta)$ can be written as $\mathcal L(\theta\mid D)$. $\endgroup$
    – Neil G
    Commented May 20, 2012 at 4:36
  • 2
    $\begingroup$ @NeilG, yes, you're absolutely correct - I've edited my answer to reflect this. Its sometimes written $\mathcal{L}(\theta | D)$ because the likelihood can be viewed as a function of the parameters holding the data constant. $\endgroup$
    – Nick
    Commented May 22, 2012 at 20:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.