4
$\begingroup$

I was watching a lecture on coursera on 'Bayesian Methods on Machine Learning' and I came across a statement that: MAP(Maximum a posteriori) is not invariant to reparametrization. I didn't quite understand:

  1. What does reparametrization mean here and why is reparametrization important?

  2. How is MAP 'not invariant to reparametrization'?

  3. Why 'not being invariant to reparametrization' a problem and How do Conjugate priors help solve this problem?

Please explain the answers in an easy and intuitive way. I looked for other similar questions to mine and they are way too mathematical and I don not have a solid theoretical foundation in bayesian statistics.

PS: Please do not mark this question as a duplicate. I have read other similar questions on StackOverflow and other sites, however, those questions don't answer my questions completely and clearly.

Edit: Even after writing a special note that I have viewed every possible question similar to this, and those do not answer my questions, This question was marked duplicate.

Improve this question
$\endgroup$
8
  • $\begingroup$ Hi, sorry I can't provide an answer to your questions, but a tip where you might find help. Coursera has their own Forum implemented, have you tried to search there if this questions was already asked? $\endgroup$
    – Björn
    Commented Sep 9, 2019 at 11:32
  • 1
    $\begingroup$ @PythonBeginner Yes, I did. There was one answer but It wasn't satisfactory.But their discussion forum is almost dead. People take a month to reply to a doubt. $\endgroup$
    – Akash Dubey
    Commented Sep 9, 2019 at 11:48
  • 3
    $\begingroup$ Possible duplicate of What is an example of a transformation on a posterior distribution such that the MAP estimate will be non-invariant? and several more stats.stackexchange.com/search?q=map+invariant $\endgroup$
    – Sycorax ♦
    Commented Sep 9, 2019 at 12:22
  • 5
    $\begingroup$ The content of the duplicate answers (2) directly. The links answer (1). Question (3) might be unique to this website. But answering 2/3rds of your question seems like a reason to edit your post to focus on the new content, not the duplicates. If you don't understand the other answers, then you can edit your question to explain what you know and what you don't know/where you're stuck and we can help you get unstuck. But writing "those questions don't answer my question completely" is not a description of what information you are seeking and what you want to understand. Be specific. $\endgroup$
    – Sycorax ♦
    Commented Sep 9, 2019 at 14:00
  • 3
    $\begingroup$ @AkashDubey if, as you say, the question has been asked on the coursera forums, and you don't understand the answer, you should probably paste that answer and explain exactly what confuses you rather than asking the question again. My $0.02- some ML topics just can't be explained/understood/used without a solid mathematical statistics foundation. $\endgroup$
    – AdamO
    Commented Sep 9, 2019 at 14:32

1 Answer 1

Reset to default
1
$\begingroup$

MAP estimation relies at its heart using optimization of the posterior (ignoring the constant term) across the values of your parameters $\theta$. Mathematically: $\theta^{MAP} = argmax_{\theta} p(x|\theta)p(\theta)$. Now suppose we would like to reparameterize $k= log(1-exp(\theta))$ and solve for k. Well quite simply the maximization problem will not give the same results even solving back for k i.e. $k^{MAP}= argmax_k p(x|k)p(k)$ and then $ \theta^{new} = log(1-e^{k^{MAP}})\neq \theta^{MAP}$ for most combinations of likelihood and priors in the original model.

Why reparameterization like the above (meaning do algebraic manipulation to get a nicer form for the parameter of interest)? Well sometimes using the above reparameterization have nice computational properties because for normals for example exponential components can get very large very quickly and the computer can overflow. Checking probabilities in the distribution can be very important to check how strong your model is.

Lastly being not invariant to reparameterization makes a method less attractive. If you can easily manipulate your problem into a simpler or more computer-friendly method there can be large gains to efficiency. This efficiency might mean that things can be estimated more accurately or much more quickly. As to the last part of your question 3, I'm not completely sure. Perhaps the posterior being in the same family as the prior ensures that when you "retranslate" back to your original parameter you will get the same answer had you not reparameterized, but I'm not completely sure and I'd have to see it worked out to know for certain.

Improve this answer
$\endgroup$
5
  • 1
    $\begingroup$ Just to add a few cents: the mathematical property that the reparametrisation needs to have for being compatible with argmax is called „monotonous“... I.e. when you reparametrize using a monotonous function like log or so „computing argmax“ and „reparametrisation“ will be interchangeable. $\endgroup$
    – Fabian Werner
    Commented Sep 9, 2019 at 15:48
  • $\begingroup$ Thanks @FabianWerner for the addition! This is right I believe, although usually we say "monotonic" transformation. This is because a monotonic transformation of an objective function will have a the same gradient as the original. Log is a good example of a monotonic transformation hence log likelihoods. $\endgroup$
    – JoeTheShmoe
    Commented Sep 9, 2019 at 16:05
  • $\begingroup$ @JoeTheShmoe What does Reparameterisation mean here? What exactly does Reparameterisation mean? What is the intuition ? $\endgroup$
    – Akash Dubey
    Commented Sep 9, 2019 at 17:13
  • 1
    $\begingroup$ Reparameterization simply means to rewrite the parameter of interest (ex: mean of a guassian in a different but equivalent form). Think of the change of variables trick for integration, it's very similar. Under one the integral might not be tractable, under the other it will. The idea is the same. So if the mean of the Gaussian is usually written as $\mu$ we might want to rewrite it in terms of a different variable say $k= exp(1+log(\mu))$ as explained above, and solve for that variable in the original problem because it's easier to work with. $\endgroup$
    – JoeTheShmoe
    Commented Sep 9, 2019 at 18:24
  • $\begingroup$ @JoeTheShmoe Thank you. I get it now. Can you please check for the third question, If possible? $\endgroup$
    – Akash Dubey
    Commented Sep 9, 2019 at 21:05

Not the answer you're looking for? Browse other questions tagged or ask your own question.