Just wonder about the simplest example, I have a simple Normal prior on my scalar parameter $w$.

$$P(w|D) \sim P(D|w) P(w)$$

$$P(w) = \mathcal{N}(0,\alpha^2)$$

When I increase $\alpha$ from 0 to $\infty$, how would the curvature of $P(w|D)$ changes?

  • $\begingroup$ When you ask about the "curvature", are you interested specifically in the curvature in the strict sense (related to the second derivative of the function), or do you mean something else? $\endgroup$ – Reinstate Monica Jul 15 '18 at 11:59
  • $\begingroup$ Yes! The second order derivative, in the 1D case is fine! $\endgroup$ – ArtificiallyIntelligence Jul 15 '18 at 13:27
  • $\begingroup$ In general wouldn't this depend on $ P(D \mid w) $? $\endgroup$ – Kevin Li Jul 15 '18 at 17:37
  • $\begingroup$ @KevinLi, yes. But I want to know, when P(D|w) is fixed, how does regularization prior affect the curvature. $\endgroup$ – ArtificiallyIntelligence Jul 16 '18 at 0:10

From your clarification in the comments, it appears that you are interested in the second derivative of the posterior, not the curvature in its strict sense. Obtaining this is just a matter of basic calculus.

To facilitate our analysis, let's use the alternative notation $\pi_D(w) \propto L_D(w) \pi_0 (w)$ so that we are not using generic function notation. From your specification we have $\pi_0 (w) = \text{N}(w|0,\alpha^2)$. With a bit of simple calculus on the normal density, the derivative of the specified prior density is:

$$\begin{equation} \begin{aligned} \frac{d \pi_0}{dw}(w) = \frac{d \text{N}}{dw}(w|0,\alpha^2) &= - \frac{w}{\alpha^2}\cdot \text{N}(w|0,\alpha^2) = - \frac{w}{\alpha^2}\cdot \pi_0(w). \end{aligned} \end{equation}$$

Then, using repeated application of the product rule you get:

$$\begin{equation} \begin{aligned} \frac{d \pi_D}{dw}(w) &= \frac{d L_D}{dw}(w) \cdot \pi_0(w) + L_D(w) \cdot \frac{d \pi_0}{dw}(w) \\[6pt] &= \Bigg[ \frac{d L_D}{dw}(w) - \frac{w}{\alpha^2} \cdot L_D(w) \Bigg] \pi_0(w), \\[10pt] \frac{d^2 \pi_D}{dw^2}(w) &= \Bigg[ \frac{d^2 L_D}{dw^2}(w) - \frac{1}{\alpha^2} \cdot L_D(w) - \frac{w}{\alpha^2} \cdot \frac{dL_D}{dw}(w) \Bigg] \pi_0(w) + \Bigg[ \frac{d L_D}{dw}(w) - \frac{w}{\alpha^2} \cdot L_D(w) \Bigg] \frac{d\pi_0}{dw}(w) \\[6pt] &= \frac{1}{\alpha^2} \Bigg[ \alpha^2 \cdot \frac{d^2 L_D}{dw^2}(w) + \Big( \frac{w^2}{\alpha^2}-1 \Big) L_D(w) - 2w \cdot \frac{dL_D}{dw}(w) \Bigg] \pi_0(w). \\[6pt] \end{aligned} \end{equation}$$

Taking the limit as $\alpha \rightarrow \infty$ you get:

$$\begin{equation} \begin{aligned} \lim_{\alpha \rightarrow \infty} \frac{d \pi_D}{dw}(w) &= \frac{d L_D}{dw}(w) \cdot \pi_0(w), \\[8pt] \lim_{\alpha \rightarrow \infty} \frac{d^2 \pi_D}{dw^2}(w) &= \frac{d^2 L_D}{dw^2}(w) \cdot \pi_0(w). \end{aligned} \end{equation}$$


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.