11
$\begingroup$

In Bayesian theorem, $$p(y|x) = \frac{p(x|y)p(y)}{p(x)}$$, and from the book I'm reading, $p(x|y)$ is called the likelihood, but I assume it's just the conditional probability of $x$ given $y$, right?

The maximum likelihood estimation tries to maximize $p(x|y)$, right? If so, I'm badly confused, because $x,y$ are both random variables, right? To maximize $p(x|y)$ is just to find out the $\hat y$? One more problem, if these 2 random variables are independent, then $p(x|y)$ is just $p(x)$, right? Then maximizing $p(x|y)$ is to maximize $p(x)$.

Or maybe, $p(x|y)$ is a function of some parameters $\theta$, that is $p(x|y; \theta)$, and MLE tries to find the $\theta$ which can maximize $p(x|y)$? Or even that $y$ is actually the parameters of the model, not random variable, maximizing the likelihood is to find the $\hat y$?

UPDATE

I'm a novice in machine learning, and this problem is a confusion from the stuff I read from a machine learning tutorial. Here it is, given an observed dataset $\{x_1,x_2,...,x_n\}$, the target values are $\{y_1,y_2,...,y_n\}$, and I try to fit a model over this dataset, so I assume that, given $x$, $y$ has a form of distribution named $W$ parameterized by $\theta$, that is $p(y|x; \theta)$, and I assume this is the posterior probability, right?

Now to estimate the value of $\theta$, I use MLE. OK, here comes my problem, I think the likelihood is $p(x|y;\theta)$, right? Maximizing the likelihood means I should pick the right $\theta$ and $y$?

If my understanding of likelihood is wrong, please show me the right way.

$\endgroup$
  • $\begingroup$ I think the confusion is this: Bayes' theorem is just the manipulation of the conditional probabilities as you give at the beginning of your question. The Bayesian Estimation makes use of the Bayes theorem to make parameter estimations. It is only in the latter, do the maximum likelihood estimation (MLE) and the parameter theta, etc. come into play. $\endgroup$ – Zhubarb Oct 22 '13 at 9:00
  • $\begingroup$ @Berkan, well I actually try to figure out what likelihood is, given $x,y,\theta$. $\endgroup$ – avocado Oct 22 '13 at 14:09
  • 1
    $\begingroup$ I see, I would recommend you to have a look at this great set of introductory lecture slides in parameter estimation. $\endgroup$ – Zhubarb Oct 22 '13 at 15:09
  • 1
    $\begingroup$ Another great topic to read about is Empirical Bayes' Estimators. We just learned about those in my class :) biostat.jhsph.edu/~fdominic/teaching/bio656/labs/labs09/… $\endgroup$ – bdeonovic Oct 22 '13 at 20:46
16
$\begingroup$

I think the core misunderstanding stems from questions you ask in the first half of your question. I approach this answer as contrasting MLE and Bayesian inferential paradigms. A very approachable discussion of MLE can be found in chapter 1 of Gary King, Unifying Political Methodology. Gelman's Bayesian Data Analysis can provide details on the Bayesian side.

In Bayes' theorem, $$p(y|x)=\frac{p(x|y)p(y)}{p(x)}$$ and from the book I'm reading, $p(x|y)$ is called the likelihood, but I assume it's just the conditional probability of $x$ given $y$, right?

The likelihood is a conditional probability. To a Bayesian, this formula describes the distribution of the parameter $y$ given data $x$ and prior $p(y)$. But since this notation doesn't reflect your intention, henceforth I will use ($\theta$,$y$) for parameters and $x$ for your data.

But your update indicates that $x$ are observed from some distribution $p(x|\theta,y)$. If we place our data and parameters in the appropriate places in Bayes' rule, we find that these additional parameters pose no problems for Bayesians: $$p(\theta|x,y)=\frac{p(x,y|\theta)p(\theta)}{p(x,y)}$$

I believe this expression is what you are after in your update.

The maximum likelihood estimation tries to maximize $p(x,y|\theta)$, right?

Yes. MLE posits that $$p(x,y|\theta) \propto p(\theta|x,y)$$ That is, it treats the term $\frac{p(\theta,y)}{p(x)}$ as an unknown (and unknowable) constant. By contrast, Bayesian inference treats $p(x)$ as a normalizing constant (so that probabilities sum/integrate to unity) and $p(\theta,y)$ as a key piece of information: the prior. We can think of $p(\theta,y)$ as a way of incurring a penalty on the optimization procedure for "wandering too far away" from the region we think is most plausible.

If so, I'm badly confused, because $x,y,\theta$ are random variables, right? To maximize $p(x,y|\theta)$ is just to find out the $\hat{\theta}$?

In MLE, $\hat{\theta}$ is assumed to be a fixed quantity that is unknown but able to be inferred, not a random variable. Bayesian inference treats $\theta$ as a random variable. Bayesian inference puts probability density functions in and gets probability density functions out, rather than point summaries of the model, as in MLE. That is, Bayesian inference looks at the full range of parameter values and the probability of each. MLE posits that $\hat{\theta}$ is an adequate summary of the data given the model.

$\endgroup$
  • 1
    $\begingroup$ Thanks for your answer, I update my post, please see my update. $\endgroup$ – avocado Oct 22 '13 at 13:59
  • $\begingroup$ This update radically changed my understanding of the question. Initially, I thought you were regarding $y$ as a parameter and $x$ as your data. Now it appears that $(x,y)$ are data and you are interested in constructing a model that describes the relationship between $x$ and $y$. I'll modify my response as I have time. $\endgroup$ – Sycorax Oct 22 '13 at 14:03
  • $\begingroup$ +1 This is still a great answer: I hope you keep it largely intact even if you modify it to match the changes in the question. $\endgroup$ – whuber Oct 22 '13 at 14:22
  • $\begingroup$ I have updated my response to reflect your updated question. I hope these details help. I really do recommend referring to the references that I mention. And I hope @whuber still approves. ;-) $\endgroup$ – Sycorax Oct 22 '13 at 19:47
  • $\begingroup$ Thanks you so much for the update, so you mean although I pick up a form of distribution for $p(y|x)$, I should treat $x,y$ both as observed data when I'm trying to estimate the $\theta$? $\endgroup$ – avocado Oct 22 '13 at 23:26
3
$\begingroup$

Normally $p(x|y)$ is a function of the parameter $y$. Consider the following reformulation of Bayes theorem:

$$p(\theta|x) = \frac{p(x|\theta)p(\theta)}{p(x)}$$

Or even more explicitly (with regards to the notion of the likelihood):

$$p(\theta|x) = \frac{L(\theta;x)p(\theta)}{p(x)}$$

For a concrete example, consider the model

$$ X|\theta \sim Binomial(\theta) \\ \theta \sim Beta(\alpha,\beta) $$

$\endgroup$
  • $\begingroup$ So, typically $y$ is not the random variable but $x$, right? $\endgroup$ – avocado Oct 22 '13 at 6:49
  • $\begingroup$ Y is usually a parameter on the pdf of X. In a frequentist setting y is normally a fixed value. In a Bayesian setting, Y is itself a random variable (as in the example I gave). X|Y can also be a conditional probability in the sense you mean, I was trying to give you the motivation behind why that quantity is called the likelihood. $\endgroup$ – David Marx Oct 22 '13 at 6:56
  • $\begingroup$ With respect to the concrete example given in your answer, do you mean $\theta$ is actually a random variable, but in $X$'s distribution, it's taken as a parameter? $\endgroup$ – avocado Oct 22 '13 at 6:59
  • $\begingroup$ Just because something is a random variable doesn't mean it can't be a parameter. Welcome to the wonderful world of bayesian probability :) $\endgroup$ – David Marx Oct 22 '13 at 16:07
0
$\begingroup$
  • "...$p(x|y)$ is called the likelihood..."

$p(x|y)$ is the likelihood of y given x. Saying what it is the likelihood of is important. And yes, it is just the conditional probability of $x$ given $y$.

  • "...if these 2 random variables are independent, then $p(x|y)$ is just $p(x)$, right? Then maximizing $p(x|y)$ is to maximize $p(x)$..."

If they are independent, i.e. $p(x|y) = p(x)$, the $p(x)$ is constant with respect to $y$. Be careful here, as you don't specify what you are maximising with respect to - from what you wrote earlier, I would assume you are maximising with respect to $y$.

  • ...Or maybe, $p(x|y)$ is a function of some parameters $\theta$, that is $p(x|y;\theta)$, and MLE tries to find the $\theta$ which can maximize $p(x|y)$? Or even that y is actually the parameters of the model, not random variable, maximizing the likelihood is to find the $\hat{y}$?...

Introducing $\theta$ makes this an entirely new problem. In general, the answer to most of this question here seems to be 'it depends'. We could denote parameters as $y$ if we wanted, and maximise with respect to them. Equally, we could have a situation where we maximise $p(x|y;\theta)$ with respect to parameters $\theta$ if that was a sensible way of approaching the problem at hand.

$\endgroup$
  • $\begingroup$ The reason why I introduce $\theta$ is this, in the machine learning book I'm reading, given a dataset $x$, and $y$ is the corresponding target value, so to fit a model to this dataset, I can use MLE to estimate $\theta$ which is the parameter of the model, right? $\endgroup$ – avocado Oct 22 '13 at 13:43
0
$\begingroup$

From STAN reference manual:

If the prior is uniform, the posterior mode corresponds to the maximum likelihood estimate (MLE) of the parameters. If the prior is not uniform, the posterior mode is sometimes called the maximum a posterior (MAP) estimate.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.