Questions tagged [likelihood]

Given a random variable $X$ which arise from a parameterized distribution $F(X;θ)$, the likelihood is defined as proportional to the probability of observed data as a function of $θ$: $\operatorname{L}(θ | x)=\operatorname{P}(X=x \mid θ)$

Filter by
Sorted by
Tagged with
667 votes
12 answers
491k views

What is the difference between "likelihood" and "probability"?

The wikipedia page claims that likelihood and probability are distinct concepts. In non-technical parlance, "likelihood" is usually a synonym for "probability," but in statistical usage there is a ...
Douglas S. Stones's user avatar
104 votes
5 answers
55k views

What is the reason that a likelihood function is not a pdf?

What is the reason that a likelihood function is not a pdf (probability density function)?
John Doe's user avatar
  • 1,556
97 votes
7 answers
43k views

Why to optimize max log probability instead of probability

In most machine learning tasks where you can formulate some probability $p$ which should be maximised, we would actually optimize the log probability $\log p$ instead of the probability for some ...
Albert's user avatar
  • 1,245
73 votes
5 answers
102k views

Why do we minimize the negative likelihood if it is equivalent to maximization of the likelihood?

This question has puzzled me for a long time. I understand the use of 'log' in maximizing the likelihood so I am not asking about 'log'. My question is, since maximizing log likelihood is equivalent ...
Tony's user avatar
  • 1,803
71 votes
2 answers
31k views

What is the difference between a partial likelihood, profile likelihood and marginal likelihood?

I see these terms being used and I keep getting them mixed up. Is there a simple explanation of the differences between them?
Rob Hyndman's user avatar
  • 56.7k
64 votes
5 answers
134k views

How to calculate pseudo-$R^2$ from R's logistic regression?

Christopher Manning's writeup on logistic regression in R shows a logistic regression in R as follows: ...
dfrankow's user avatar
  • 3,376
55 votes
3 answers
12k views

What kind of information is Fisher information?

Suppose we have a random variable $X \sim f(x|\theta)$. If $\theta_0$ were the true parameter, the the likelihood function should be maximized and the derivative equal to zero. This is the basic ...
Stan Shunpike's user avatar
53 votes
7 answers
8k views

Why would someone use a Bayesian approach with a 'noninformative' improper prior instead of the classical approach?

If the interest is merely estimating the parameters of a model (pointwise and/or interval estimation) and the prior information is not reliable, weak, (I know this is a bit vague but I am trying to ...
user avatar
48 votes
10 answers
7k views

Why do people use p-values instead of computing probability of the model given data?

Roughly speaking a p-value gives a probability of the observed outcome of an experiment given the hypothesis (model). Having this probability (p-value) we want to judge our hypothesis (how likely it ...
Roman's user avatar
  • 584
40 votes
5 answers
184k views

How to derive the likelihood function for binomial distribution for parameter estimation?

According to Miller and Freund's Probability and Statistics for Engineers, 8ed (pp.217-218), the likelihood function to be maximised for binomial distribution (Bernoulli trials) is given as $L(p) = \...
Ébe Isaac's user avatar
  • 1,082
36 votes
5 answers
3k views

Wikipedia entry on likelihood seems ambiguous

I have a simple question regarding "conditional probability" and "Likelihood". (I have already surveyed this question here but to no avail.) It starts from the Wikipedia page on likelihood. They say ...
Creatron's user avatar
  • 1,655
33 votes
3 answers
4k views

How to rigorously define the likelihood?

The likelihood could be defined by several ways, for instance : the function $L$ from $\Theta\times{\cal X}$ which maps $(\theta,x)$ to $L(\theta \mid x)$ i.e. $L:\Theta\times{\cal X} \rightarrow \...
Stéphane Laurent's user avatar
32 votes
4 answers
10k views

Theoretical motivation for using log-likelihood vs likelihood

I'm trying to understand at a deeper level the ubiquity of log-likelihood (and perhaps more generally log-probability) in statistics and probability theory. Log-probabilities show up all over the ...
ratsalad's user avatar
  • 683
31 votes
8 answers
10k views

Bayes' Theorem Intuition

I've been trying to develop an intuition based understanding of Bayes' theorem in terms of the prior, posterior, likelihood and marginal probability. For that I use the following equation: $$P(B|A) = \...
Anas Ayubi's user avatar
31 votes
1 answer
7k views

Computation of the marginal likelihood from MCMC samples

This is a recurring question (see this post, this post and this post), but I have a different spin. Suppose I have a bunch of samples from a generic MCMC sampler. For each sample $\theta$, I know the ...
lacerbi's user avatar
  • 5,186
30 votes
5 answers
32k views

Why is everything based on likelihoods even though likelihoods are so small?

Suppose I generate some random numbers from a specific normal distribution in R: set.seed(123) random_numbers <- rnorm(50, mean = 5, sd = 5) These numbers look ...
Uk rain troll's user avatar
29 votes
4 answers
5k views

Maximum likelihood function for mixed type distribution

In general we maximize a function $$ L(\theta; x_1, \ldots, x_n) = \prod_{i=1}^n f(x_i \mid \theta) $$ where $f$ is probability density function if the underlying distribution is continuous, and a ...
bonifaz's user avatar
  • 1,085
29 votes
3 answers
4k views

Is there any difference between Frequentist and Bayesian on the definition of Likelihood?

Some sources say likelihood function is not conditional probability, some say it is. This is very confusing to me. According to most sources I have seen, the likelihood of a distribution with ...
CyberPlayerOne's user avatar
29 votes
3 answers
3k views

What are some illustrative applications of empirical likelihood?

I have heard of Owen's empirical likelihood, but until recently paid it no heed until I came across it in a paper of interest (Mengersen et al. 2012). In my efforts to understand it, I have gleaned ...
Sameer's user avatar
  • 1,014
29 votes
6 answers
8k views

Why do people use $\mathcal{L}(\theta|x)$ for likelihood instead of $P(x|\theta)$?

According to the Wikipedia article Likelihood function, the likelihood function is defined as: $$ \mathcal{L}(\theta|x)=P(x|\theta), $$ with parameters $\theta$ and observed data $x$. This equals $p(...
danijar's user avatar
  • 990
27 votes
1 answer
14k views

Converting (normalizing) very small likelihood values to probability

I am writing an algorithm where, given a model, I compute likelihoods for a list of datasets and then need to normalize (to probability) each one of the likelihood. So something like [0.00043, 0.00004,...
Ikram Ullah's user avatar
27 votes
8 answers
7k views

Posterior very different to prior and likelihood

If the prior and the likelihood are very different from each other, then sometimes a situation occurs where the posterior is similar to neither of them. See for example this picture, which uses normal ...
Rónán Daly's user avatar
26 votes
5 answers
10k views

What does "likelihood is only defined up to a multiplicative constant of proportionality" mean in practice?

I'm reading a paper where the authors are leading from a discussion of maximum likelihood estimation to Bayes' Theorem, ostensibly as an introduction for beginners. As a likelihood example, they ...
kmm's user avatar
  • 557
23 votes
2 answers
4k views

What are the disadvantages of the profile likelihood?

Consider a vector of parameters $(\theta_1, \theta_2)$, with $\theta_1$ the parameter of interest, and $\theta_2$ a nuisance parameter. If $L(\theta_1, \theta_2 ; x)$ is the likelihood constructed ...
ocram's user avatar
  • 21.8k
22 votes
6 answers
1k views

An example where the likelihood principle *really* matters?

Is there an example where two different defensible tests with proportional likelihoods would lead one to markedly different (and equally defensible) inferences, for instance, where the p-values are ...
user avatar
22 votes
3 answers
9k views

Comparing AIC of a model and its log-transformed version

The essence of my question is this: Let $Y \in \mathbb{R}^n$ be a multivariate normal random variable with mean $\mu$ and covariance matrix $\Sigma$. Let $Z := \log(Y)$, i.e. $Z_i = \log(Y_i), i \...
Stijn's user avatar
  • 1,880
22 votes
2 answers
4k views

What would be an example of a really simple model with an intractable likelihood?

Approximate Bayesian computation is a really cool technique for fitting basically any stochastic model, intended for models where the likelihood is intractable (say, you can sample from the model if ...
Rasmus Bååth's user avatar
22 votes
2 answers
2k views

If the likelihood principle clashes with frequentist probability then do we discard one of them?

In a comment recently posted here one commenter pointed to a blog by Larry Wasserman who points out (without any sources) that frequentist inference clashes with the likelihood principle. The ...
Michael Lew's user avatar
  • 15.1k
21 votes
4 answers
4k views

How is the bayesian framework better in interpretation when we usually use uninformative or subjective priors?

It is often argued that the bayesian framework has a big advantage in interpretation (over frequentist), because it computes the probability of a parameter given the data - $p(\theta|x)$ instead of $p(...
Tomas's user avatar
  • 6,153
21 votes
2 answers
15k views

What is bits per dimension (bits/dim) exactly (in pixel CNN papers)?

If it is for the lack of my effort to search, I apologize in advance but I couldn't find a explicit definition of bits per dimension (bits/dim). The first mention of its definition I found was from ...
le4m's user avatar
  • 470
20 votes
3 answers
16k views

In a GLM, is the log likelihood of the saturated model always zero?

As part of the output of a generalised linear model, the null and residual deviance are used to evaluate the model. I often see the formulas for these quantities expressed in terms of the log ...
Alex's user avatar
  • 4,382
19 votes
4 answers
2k views

In Bayesian statistics, data is considered nonrandom but can have a probability or be conditioned on. How?

In Bayesian statistics, parameters are said to be random variables while data are said to be nonrandom. Yet if we look at the Bayesian updating formula $$ p(\theta|y)=\frac{p(\theta)p(y|\theta)}{p(y)},...
Richard Hardy's user avatar
19 votes
2 answers
22k views

Can the likelihood take values outside of the range [0, 1]? [duplicate]

I got a log-likelihood value of -34.82, so I am not getting whether the answer which I have got is right or not. Can the likelihood take values outside of the range $[0, 1]$?
user avatar
19 votes
4 answers
6k views

Finding the MLE for a univariate exponential Hawkes process

The univariate exponential Hawkes process is a self-exciting point process with an event arrival rate of: $ \lambda(t) = \mu + \sum\limits_{t_i<t}{\alpha e^{-\beta(t-t_i)}}$ where $ t_1,..t_n $ ...
Dave Anderson's user avatar
19 votes
2 answers
987 views

What is the frequentist take on the voltmeter story?

What is the frequentist take on the voltmeter story and its variations? The idea behind it is that a statistical analysis that appeals to hypothetical events would have to be revised if it was later ...
Praxeolitic's user avatar
18 votes
3 answers
10k views

Likelihood vs conditional distribution for Bayesian analysis

We can write Bayes' theorem as $$p(\theta|x) = \frac{f(X|\theta)p(\theta)}{\int_{\theta} f(X|\theta)p(\theta)d\theta}$$ where $p(\theta|x)$ is the posterior, $f(X|\theta)$ is the conditional ...
kzoo's user avatar
  • 181
17 votes
2 answers
13k views

Why is posterior density proportional to prior density times likelihood function?

According to Bayes' theorem, $P(y|\theta)P(\theta) = P(\theta|y)P(y)$. But according to my econometric text, it says that $P(\theta|y) \propto P(y|\theta)P(\theta)$. Why is it like this? I don't get ...
bayes-problem's user avatar
17 votes
1 answer
2k views

Questions about Likelihood Principle

I currently try to understand Likelihood Principle and I frankly don't get it at all. So, I will write all my question as a list, even if those might be pretty basic questions. What exactly does "all ...
Karel Bílek's user avatar
16 votes
1 answer
17k views

Recalculate log-likelihood from a simple R lm model

I'm simply trying to recalculate with dnorm() the log-likelihood provided by the logLik function from a lm model (in R). It works (almost perfectly) for high number of data (eg n=1000) : ...
Gilles San Martin's user avatar
16 votes
1 answer
1k views

Do you have to adhere to the likelihood principle to be a Bayesian?

This question is spurred from the question: When (if ever) is a frequentist approach substantively better than a Bayesian? As I posted in my solution to that question, in my opinion, if you are a ...
RustyStatistician's user avatar
16 votes
2 answers
975 views

Are we frequentists really just implicit/unwitting Bayesians?

For a given inference problem, we know that a Bayesian approach usually differ in both form and results from a fequentist approach. Frequentists (usually includes me) often point out that their ...
user avatar
16 votes
1 answer
422 views

What is the "direct likelihood" point of view in statistics?

I am reading a Springer title from 1997 called Applied Generalized Linear Models by James K. Lindsey. In the preface, Lindsey writes For this text, the reader is assumed to have knowledge of basic ...
Hugo's user avatar
  • 263
15 votes
6 answers
2k views

If you use a point estimate that maximizes $P(x | \theta)$, what does that say about your philosophy? (frequentist or Bayesian or something else?)

If somebody said "That method uses the MLE the point estimate for the parameter which maximizes $\mathrm{P}(x|\theta)$, therefore it is frequentist; and further it is not Bayesian." would you agree?...
Aaron McDaid's user avatar
15 votes
1 answer
33k views

What is the log of the PDF for a Normal Distribution?

I am learning Maximum Likelihood Estimation. Per this post, the log of the PDF for a normal distribution looks like this: $$ \log{\left(f\left(x_i;\,\mu,\sigma^2\right)\right)} = - \frac{n}{2} \log{\...
JJJohn's user avatar
  • 1,865
15 votes
5 answers
10k views

What are the differences between stochastic and fixed regressors in linear regression model?

If we have stochastic regressors, we are drawing random pairs $(y_i,\vec{x}_i)$ for a bunch of $i$, the so-called random sample, from a fixed but unknown probabilistic distribution $(y,\vec{x})$. ...
Kun's user avatar
  • 502
15 votes
1 answer
9k views

LogLikelihood Parameter Estimation for Linear Gaussian Kalman Filter

I have written some code that can do Kalman filtering (using a number of different Kalman-type filters [Information Filter et al.]) for Linear Gaussian State Space Analysis for an n-dimensional state ...
MoonKnight's user avatar
15 votes
1 answer
21k views

Log marginal likelihood for Gaussian Process

Log marginal likelihood for Gaussian Process as per Rasmussen's Gaussian Processes for Machine Learning equation 2.30 is: $$\log p(y|X) = -\frac{1}{2}y^T(K+\sigma^2_n I)^{-1}y - \frac{1}{2}\log|K+\...
pkj's user avatar
  • 573
14 votes
1 answer
6k views

What is the intuition behind the score function? [duplicate]

Wikipedia tells us that the score plays an important role in the Cramér–Rao inequality. It also phrases out the definition: $$V = \frac{\partial}{\partial \theta} \log{L(\theta; X)}$$ However, I ...
blubb's user avatar
  • 2,630
14 votes
2 answers
70k views

Interpreting log likelihood

I have difficulty interpreting some results. I am doing an hierarchical related regression with ecoreg. If I enter the code I receive output with odds ratios, ...
Keizer's user avatar
  • 151
13 votes
2 answers
11k views

Fisher's score function has mean zero - what does that even mean?

I'm trying to follow the princeton review of likelihood theory. They define Fisher’s score function as The first derivative of the log-likelihood function, and they ...
ihadanny's user avatar
  • 3,290

1
2 3 4 5
31