Questions tagged [likelihood]
Given a random variable $X$ which arise from a parameterized distribution $F(X;θ)$, the likelihood is defined as proportional to the probability of observed data as a function of $θ$: $\operatorname{L}(θ | x)=\operatorname{P}(X=x \mid θ)$
1,549
questions
667
votes
12
answers
491k
views
What is the difference between "likelihood" and "probability"?
The wikipedia page claims that likelihood and probability are distinct concepts.
In non-technical parlance, "likelihood" is usually a synonym for "probability," but in statistical usage there is a ...
104
votes
5
answers
55k
views
What is the reason that a likelihood function is not a pdf?
What is the reason that a likelihood function is not a pdf (probability density function)?
97
votes
7
answers
43k
views
Why to optimize max log probability instead of probability
In most machine learning tasks where you can formulate some probability $p$ which should be maximised, we would actually optimize the log probability $\log p$ instead of the probability for some ...
73
votes
5
answers
102k
views
Why do we minimize the negative likelihood if it is equivalent to maximization of the likelihood?
This question has puzzled me for a long time. I understand the use of 'log' in maximizing the likelihood so I am not asking about 'log'.
My question is, since maximizing log likelihood is equivalent ...
71
votes
2
answers
31k
views
What is the difference between a partial likelihood, profile likelihood and marginal likelihood?
I see these terms being used and I keep getting them mixed up. Is there a simple explanation of the differences between them?
64
votes
5
answers
134k
views
How to calculate pseudo-$R^2$ from R's logistic regression?
Christopher Manning's writeup on logistic regression in R shows a logistic regression in R as follows:
...
55
votes
3
answers
12k
views
What kind of information is Fisher information?
Suppose we have a random variable $X \sim f(x|\theta)$. If $\theta_0$ were the true parameter, the the likelihood function should be maximized and the derivative equal to zero. This is the basic ...
53
votes
7
answers
8k
views
Why would someone use a Bayesian approach with a 'noninformative' improper prior instead of the classical approach?
If the interest is merely estimating the parameters of a model (pointwise and/or interval estimation) and the prior information is not reliable, weak, (I know this is a bit vague but I am trying to ...
48
votes
10
answers
7k
views
Why do people use p-values instead of computing probability of the model given data?
Roughly speaking a p-value gives a probability of the observed outcome of an experiment given the hypothesis (model). Having this probability (p-value) we want to judge our hypothesis (how likely it ...
40
votes
5
answers
184k
views
How to derive the likelihood function for binomial distribution for parameter estimation?
According to Miller and Freund's Probability and Statistics for Engineers, 8ed (pp.217-218), the likelihood function to be maximised for binomial distribution (Bernoulli trials) is given as
$L(p) = \...
36
votes
5
answers
3k
views
Wikipedia entry on likelihood seems ambiguous
I have a simple question regarding "conditional probability" and "Likelihood". (I have already surveyed this question here but to no avail.)
It starts from the Wikipedia page on likelihood. They say ...
33
votes
3
answers
4k
views
How to rigorously define the likelihood?
The likelihood could be defined by several ways, for instance :
the function $L$ from $\Theta\times{\cal X}$ which maps $(\theta,x)$ to $L(\theta \mid x)$ i.e. $L:\Theta\times{\cal X} \rightarrow \...
32
votes
4
answers
10k
views
Theoretical motivation for using log-likelihood vs likelihood
I'm trying to understand at a deeper level the ubiquity of log-likelihood (and perhaps more generally log-probability) in statistics and probability theory. Log-probabilities show up all over the ...
31
votes
8
answers
10k
views
Bayes' Theorem Intuition
I've been trying to develop an intuition based understanding of Bayes' theorem in terms of the prior, posterior, likelihood and marginal probability. For that I use the following equation:
$$P(B|A) = \...
31
votes
1
answer
7k
views
Computation of the marginal likelihood from MCMC samples
This is a recurring question (see this post, this post and this post), but I have a different spin.
Suppose I have a bunch of samples from a generic MCMC sampler. For each sample $\theta$, I know the ...
30
votes
5
answers
32k
views
Why is everything based on likelihoods even though likelihoods are so small?
Suppose I generate some random numbers from a specific normal distribution in R:
set.seed(123)
random_numbers <- rnorm(50, mean = 5, sd = 5)
These numbers look ...
29
votes
4
answers
5k
views
Maximum likelihood function for mixed type distribution
In general we maximize a function
$$ L(\theta; x_1, \ldots, x_n) = \prod_{i=1}^n f(x_i \mid \theta) $$
where $f$ is probability density function if the underlying distribution is continuous, and a ...
29
votes
3
answers
4k
views
Is there any difference between Frequentist and Bayesian on the definition of Likelihood?
Some sources say likelihood function is not conditional probability, some say it is. This is very confusing to me.
According to most sources I have seen, the likelihood of a distribution with ...
29
votes
3
answers
3k
views
What are some illustrative applications of empirical likelihood?
I have heard of Owen's empirical likelihood, but until recently paid it no heed until I came across it in a paper of interest (Mengersen et al. 2012).
In my efforts to understand it, I have gleaned ...
29
votes
6
answers
8k
views
Why do people use $\mathcal{L}(\theta|x)$ for likelihood instead of $P(x|\theta)$?
According to the Wikipedia article Likelihood function, the likelihood function is defined as:
$$
\mathcal{L}(\theta|x)=P(x|\theta),
$$
with parameters $\theta$ and observed data $x$. This equals $p(...
27
votes
1
answer
14k
views
Converting (normalizing) very small likelihood values to probability
I am writing an algorithm where, given a model, I compute likelihoods for a list of datasets and then need to normalize (to probability) each one of the likelihood. So something like [0.00043, 0.00004,...
27
votes
8
answers
7k
views
Posterior very different to prior and likelihood
If the prior and the likelihood are very different from each other, then sometimes a situation occurs where the posterior is similar to neither of them. See for example this picture, which uses normal ...
26
votes
5
answers
10k
views
What does "likelihood is only defined up to a multiplicative constant of proportionality" mean in practice?
I'm reading a paper where the authors are leading from a discussion of maximum likelihood estimation to Bayes' Theorem, ostensibly as an introduction for beginners.
As a likelihood example, they ...
23
votes
2
answers
4k
views
What are the disadvantages of the profile likelihood?
Consider a vector of parameters $(\theta_1, \theta_2)$, with $\theta_1$ the parameter of interest, and $\theta_2$ a nuisance parameter.
If $L(\theta_1, \theta_2 ; x)$ is the likelihood constructed ...
22
votes
6
answers
1k
views
An example where the likelihood principle *really* matters?
Is there an example where two different defensible tests with proportional likelihoods would lead one to markedly different (and equally defensible) inferences, for instance, where the p-values are ...
22
votes
3
answers
9k
views
Comparing AIC of a model and its log-transformed version
The essence of my question is this:
Let $Y \in \mathbb{R}^n$ be a multivariate normal random variable with mean $\mu$ and covariance matrix $\Sigma$. Let $Z := \log(Y)$, i.e. $Z_i = \log(Y_i), i \...
22
votes
2
answers
4k
views
What would be an example of a really simple model with an intractable likelihood?
Approximate Bayesian computation is a really cool technique for fitting basically any stochastic model, intended for models where the likelihood is intractable (say, you can sample from the model if ...
22
votes
2
answers
2k
views
If the likelihood principle clashes with frequentist probability then do we discard one of them?
In a comment recently posted here one commenter pointed to a blog by Larry Wasserman who points out (without any sources) that frequentist inference clashes with the likelihood principle.
The ...
21
votes
4
answers
4k
views
How is the bayesian framework better in interpretation when we usually use uninformative or subjective priors?
It is often argued that the bayesian framework has a big advantage in interpretation (over frequentist), because it computes the probability of a parameter given the data - $p(\theta|x)$ instead of $p(...
21
votes
2
answers
15k
views
What is bits per dimension (bits/dim) exactly (in pixel CNN papers)?
If it is for the lack of my effort to search, I apologize in advance but I couldn't find a explicit definition of bits per dimension (bits/dim).
The first mention of its definition I found was from ...
20
votes
3
answers
16k
views
In a GLM, is the log likelihood of the saturated model always zero?
As part of the output of a generalised linear model, the null and residual deviance are used to evaluate the model. I often see the formulas for these quantities expressed in terms of the log ...
19
votes
4
answers
2k
views
In Bayesian statistics, data is considered nonrandom but can have a probability or be conditioned on. How?
In Bayesian statistics, parameters are said to be random variables while data are said to be nonrandom. Yet if we look at the Bayesian updating formula
$$
p(\theta|y)=\frac{p(\theta)p(y|\theta)}{p(y)},...
19
votes
2
answers
22k
views
Can the likelihood take values outside of the range [0, 1]? [duplicate]
I got a log-likelihood value of -34.82, so I am not getting whether the answer which I have got is right or not.
Can the likelihood take values outside of the range $[0, 1]$?
19
votes
4
answers
6k
views
Finding the MLE for a univariate exponential Hawkes process
The univariate exponential Hawkes process is a self-exciting point process with an event arrival rate of:
$ \lambda(t) = \mu + \sum\limits_{t_i<t}{\alpha e^{-\beta(t-t_i)}}$
where $ t_1,..t_n $ ...
19
votes
2
answers
987
views
What is the frequentist take on the voltmeter story?
What is the frequentist take on the voltmeter story and its variations? The idea behind it is that a statistical analysis that appeals to hypothetical events would have to be revised if it was later ...
18
votes
3
answers
10k
views
Likelihood vs conditional distribution for Bayesian analysis
We can write Bayes' theorem as
$$p(\theta|x) = \frac{f(X|\theta)p(\theta)}{\int_{\theta} f(X|\theta)p(\theta)d\theta}$$
where $p(\theta|x)$ is the posterior, $f(X|\theta)$ is the conditional ...
17
votes
2
answers
13k
views
Why is posterior density proportional to prior density times likelihood function?
According to Bayes' theorem, $P(y|\theta)P(\theta) = P(\theta|y)P(y)$. But according to my econometric text, it says that $P(\theta|y) \propto P(y|\theta)P(\theta)$. Why is it like this? I don't get ...
17
votes
1
answer
2k
views
Questions about Likelihood Principle
I currently try to understand Likelihood Principle and I frankly don't get it at all. So, I will write all my question as a list, even if those might be pretty basic questions.
What exactly does "all ...
16
votes
1
answer
17k
views
Recalculate log-likelihood from a simple R lm model
I'm simply trying to recalculate with dnorm() the log-likelihood provided by the logLik function from a lm model (in R).
It works (almost perfectly) for high number of data (eg n=1000) :
...
16
votes
1
answer
1k
views
Do you have to adhere to the likelihood principle to be a Bayesian?
This question is spurred from the question: When (if ever) is a frequentist approach substantively better than a Bayesian?
As I posted in my solution to that question, in my opinion, if you are a ...
16
votes
2
answers
975
views
Are we frequentists really just implicit/unwitting Bayesians?
For a given inference problem, we know that a Bayesian approach usually differ in both form and results from a fequentist approach. Frequentists (usually includes me) often point out that their ...
16
votes
1
answer
422
views
What is the "direct likelihood" point of view in statistics?
I am reading a Springer title from 1997 called Applied Generalized Linear Models by James K. Lindsey. In the preface, Lindsey writes
For this text, the reader is assumed to have knowledge of basic ...
15
votes
6
answers
2k
views
If you use a point estimate that maximizes $P(x | \theta)$, what does that say about your philosophy? (frequentist or Bayesian or something else?)
If somebody said
"That method uses the MLE the point estimate for the parameter which maximizes $\mathrm{P}(x|\theta)$, therefore it is frequentist; and further it is not Bayesian."
would you agree?...
15
votes
1
answer
33k
views
What is the log of the PDF for a Normal Distribution?
I am learning Maximum Likelihood Estimation.
Per this post, the log of the PDF for a normal distribution looks like this:
$$
\log{\left(f\left(x_i;\,\mu,\sigma^2\right)\right)}
=
- \frac{n}{2} \log{\...
15
votes
5
answers
10k
views
What are the differences between stochastic and fixed regressors in linear regression model?
If we have stochastic regressors, we are drawing random pairs $(y_i,\vec{x}_i)$ for a bunch of $i$, the so-called random sample, from a fixed but unknown probabilistic distribution $(y,\vec{x})$. ...
15
votes
1
answer
9k
views
LogLikelihood Parameter Estimation for Linear Gaussian Kalman Filter
I have written some code that can do Kalman filtering (using a number of different Kalman-type filters [Information Filter et al.]) for Linear Gaussian State Space Analysis for an n-dimensional state ...
15
votes
1
answer
21k
views
Log marginal likelihood for Gaussian Process
Log marginal likelihood for Gaussian Process as per Rasmussen's Gaussian Processes for Machine Learning equation 2.30 is:
$$\log p(y|X) = -\frac{1}{2}y^T(K+\sigma^2_n I)^{-1}y - \frac{1}{2}\log|K+\...
14
votes
1
answer
6k
views
What is the intuition behind the score function? [duplicate]
Wikipedia tells us that the score plays an important role in the Cramér–Rao inequality. It also phrases out the definition:
$$V = \frac{\partial}{\partial \theta} \log{L(\theta; X)}$$
However, I ...
14
votes
2
answers
70k
views
Interpreting log likelihood
I have difficulty interpreting some results. I am doing an hierarchical related regression with ecoreg. If I enter the code I receive output with odds ratios, ...
13
votes
2
answers
11k
views
Fisher's score function has mean zero - what does that even mean?
I'm trying to follow the princeton review of likelihood theory.
They define Fisher’s score function as The first derivative of the log-likelihood function, and they ...