# Return to Answer

 4 replaced http://stats.stackexchange.com/ with https://stats.stackexchange.com/ edited Apr 13 '17 at 12:44 The basic idea of Bayesian updating is that given some data $$X$$ and prior over parameter of interest $$\theta$$, where the relation between data and parameter is described using likelihood function, you use Bayes theorem to obtain posterior $$p(\theta \mid X) \propto p(X \mid \theta) \, p(\theta)$$ This can be done sequentially, where after seeing first data point $$x_1$$ prior $$\theta$$ becomes updated to posterior $$\theta'$$, next you can take second data point $$x_2$$ and use posterior obtained before $$\theta'$$ as your prior, to update it once again etc. Let me give you an example. Imagine that you want to estimate mean $$\mu$$ of normal distribution and $$\sigma^2$$ is known to you. In such case we can use normal-normal model. We assume normal prior for $$\mu$$ with hyperparameters $$\mu_0,\sigma_0^2:$$ \begin{align} X\mid\mu &\sim \mathrm{Normal}(\mu,\ \sigma^2) \\ \mu &\sim \mathrm{Normal}(\mu_0,\ \sigma_0^2) \end{align} Since normal distribution is a conjugate prior for $$\mu$$ of normal distribution, we have closed-form solution to update the prior \begin{align} E(\mu' \mid x) &= \frac{\sigma^2\mu + \sigma^2_0 x}{\sigma^2 + \sigma^2_0} \\[7pt] \mathrm{Var}(\mu' \mid x) &= \frac{\sigma^2 \sigma^2_0}{\sigma^2 + \sigma^2_0} \end{align} Unfortunately, such simple closed-form solutions are not available for more sophisticated problems and you have to rely on optimization algorithms (for point estimates using maximum a posteriori approach), or MCMC simulation. Below you can see data example: n <- 1000 set.seed(123) x <- rnorm(n, 1.4, 2.7) mu <- numeric(n) sigma <- numeric(n) mu <- (10000*x[i] + (2.7^2)*0)/(10000+2.7^2) sigma <- (10000*2.7^2)/(10000+2.7^2) for (i in 2:n) { mu[i] <- ( sigma[i-1]*x[i] + (2.7^2)*mu[i-1] )/(sigma[i-1]+2.7^2) sigma[i] <- ( sigma[i-1]*2.7^2 )/(sigma[i-1]+2.7^2) }  If you plot the results, you'll see how posterior approaches the estimated value (it's true value is marked by red line) as new data is accumulated. For learning more you can check those slides and Conjugate Bayesian analysis of the Gaussian distribution paper by Kevin P. Murphy. Check also http://stats.stackexchange.com/questions/200982/do-bayesian-priors-become-irrelevant-with-large-sample-size/201059#201059Do Bayesian priors become irrelevant with large sample size? You can also check those notes and this blog entry for accessible step-by-step introduction to Bayesian inference. The basic idea of Bayesian updating is that given some data $$X$$ and prior over parameter of interest $$\theta$$, where the relation between data and parameter is described using likelihood function, you use Bayes theorem to obtain posterior $$p(\theta \mid X) \propto p(X \mid \theta) \, p(\theta)$$ This can be done sequentially, where after seeing first data point $$x_1$$ prior $$\theta$$ becomes updated to posterior $$\theta'$$, next you can take second data point $$x_2$$ and use posterior obtained before $$\theta'$$ as your prior, to update it once again etc. Let me give you an example. Imagine that you want to estimate mean $$\mu$$ of normal distribution and $$\sigma^2$$ is known to you. In such case we can use normal-normal model. We assume normal prior for $$\mu$$ with hyperparameters $$\mu_0,\sigma_0^2:$$ \begin{align} X\mid\mu &\sim \mathrm{Normal}(\mu,\ \sigma^2) \\ \mu &\sim \mathrm{Normal}(\mu_0,\ \sigma_0^2) \end{align} Since normal distribution is a conjugate prior for $$\mu$$ of normal distribution, we have closed-form solution to update the prior \begin{align} E(\mu' \mid x) &= \frac{\sigma^2\mu + \sigma^2_0 x}{\sigma^2 + \sigma^2_0} \\[7pt] \mathrm{Var}(\mu' \mid x) &= \frac{\sigma^2 \sigma^2_0}{\sigma^2 + \sigma^2_0} \end{align} Unfortunately, such simple closed-form solutions are not available for more sophisticated problems and you have to rely on optimization algorithms (for point estimates using maximum a posteriori approach), or MCMC simulation. Below you can see data example: n <- 1000 set.seed(123) x <- rnorm(n, 1.4, 2.7) mu <- numeric(n) sigma <- numeric(n) mu <- (10000*x[i] + (2.7^2)*0)/(10000+2.7^2) sigma <- (10000*2.7^2)/(10000+2.7^2) for (i in 2:n) { mu[i] <- ( sigma[i-1]*x[i] + (2.7^2)*mu[i-1] )/(sigma[i-1]+2.7^2) sigma[i] <- ( sigma[i-1]*2.7^2 )/(sigma[i-1]+2.7^2) }  If you plot the results, you'll see how posterior approaches the estimated value (it's true value is marked by red line) as new data is accumulated. For learning more you can check those slides and Conjugate Bayesian analysis of the Gaussian distribution paper by Kevin P. Murphy. Check also http://stats.stackexchange.com/questions/200982/do-bayesian-priors-become-irrelevant-with-large-sample-size/201059#201059 You can also check those notes and this blog entry for accessible step-by-step introduction to Bayesian inference. The basic idea of Bayesian updating is that given some data $$X$$ and prior over parameter of interest $$\theta$$, where the relation between data and parameter is described using likelihood function, you use Bayes theorem to obtain posterior $$p(\theta \mid X) \propto p(X \mid \theta) \, p(\theta)$$ This can be done sequentially, where after seeing first data point $$x_1$$ prior $$\theta$$ becomes updated to posterior $$\theta'$$, next you can take second data point $$x_2$$ and use posterior obtained before $$\theta'$$ as your prior, to update it once again etc. Let me give you an example. Imagine that you want to estimate mean $$\mu$$ of normal distribution and $$\sigma^2$$ is known to you. In such case we can use normal-normal model. We assume normal prior for $$\mu$$ with hyperparameters $$\mu_0,\sigma_0^2:$$ \begin{align} X\mid\mu &\sim \mathrm{Normal}(\mu,\ \sigma^2) \\ \mu &\sim \mathrm{Normal}(\mu_0,\ \sigma_0^2) \end{align} Since normal distribution is a conjugate prior for $$\mu$$ of normal distribution, we have closed-form solution to update the prior \begin{align} E(\mu' \mid x) &= \frac{\sigma^2\mu + \sigma^2_0 x}{\sigma^2 + \sigma^2_0} \\[7pt] \mathrm{Var}(\mu' \mid x) &= \frac{\sigma^2 \sigma^2_0}{\sigma^2 + \sigma^2_0} \end{align} Unfortunately, such simple closed-form solutions are not available for more sophisticated problems and you have to rely on optimization algorithms (for point estimates using maximum a posteriori approach), or MCMC simulation. Below you can see data example: n <- 1000 set.seed(123) x <- rnorm(n, 1.4, 2.7) mu <- numeric(n) sigma <- numeric(n) mu <- (10000*x[i] + (2.7^2)*0)/(10000+2.7^2) sigma <- (10000*2.7^2)/(10000+2.7^2) for (i in 2:n) { mu[i] <- ( sigma[i-1]*x[i] + (2.7^2)*mu[i-1] )/(sigma[i-1]+2.7^2) sigma[i] <- ( sigma[i-1]*2.7^2 )/(sigma[i-1]+2.7^2) }  If you plot the results, you'll see how posterior approaches the estimated value (it's true value is marked by red line) as new data is accumulated. For learning more you can check those slides and Conjugate Bayesian analysis of the Gaussian distribution paper by Kevin P. Murphy. Check also Do Bayesian priors become irrelevant with large sample size? You can also check those notes and this blog entry for accessible step-by-step introduction to Bayesian inference. 3 some formatting for easier readability edited Jan 12 '17 at 16:59 gung♦ 112k3434 gold badges277277 silver badges547547 bronze badges The basic idea of Bayesian updating is that given some data $$X$$ and prior over parameter of interest $$\theta$$, where the relation between data and parameter is described using likelihood function, you use Bayes theorem to obtain posterior $$p(\theta \mid X) \propto p(X \mid \theta) \, p(\theta)$$ This can be done sequentially, where after seeing first data point $$x_1$$ prior $$\theta$$ becomes updated to posterior $$\theta'$$, next you can take second data point $$x_2$$ and use posterior obtained before $$\theta'$$ as your prior, to update it once again etc. Let me give you an example. Imagine that you want to estimate mean $$\mu$$ of normal distribution and $$\sigma^2$$ is known to you. In such case we can use normal-normal model. We assume normal prior for $$\mu$$ with hyperparameters $$\mu_0,\sigma_0^2:$$ $$X\mid\mu \sim \mathrm{Normal}(\mu, \sigma^2) \\ \mu \sim \mathrm{Normal}(\mu_0, \sigma_0^2)$$\begin{align} X\mid\mu &\sim \mathrm{Normal}(\mu,\ \sigma^2) \\ \mu &\sim \mathrm{Normal}(\mu_0,\ \sigma_0^2) \end{align} Since normal distribution is a conjugate prior for $$\mu$$ of normal distribution, we have closed-form solution to update the prior $$E(\mu' \mid x) = \frac{\sigma^2\mu + \sigma^2_0 x}{\sigma^2 + \sigma^2_0} \\ \mathrm{Var}(\mu' \mid x) = \frac{\sigma^2 \sigma^2_0}{\sigma^2 + \sigma^2_0}$$\begin{align} E(\mu' \mid x) &= \frac{\sigma^2\mu + \sigma^2_0 x}{\sigma^2 + \sigma^2_0} \\[7pt] \mathrm{Var}(\mu' \mid x) &= \frac{\sigma^2 \sigma^2_0}{\sigma^2 + \sigma^2_0} \end{align} Unfortunately, such simple closed-form solutions are not available for more sophisticated problems and you have to rely on optimization algorithms (for point estimates using maximum a posteriori approach), or MCMC simulation. Below you can see data example: n <- 1000 set.seed(123) x <- rnorm(n, 1.4, 2.7) mu <- numeric(n) sigma <- numeric(n) mu <- (10000*x[i] + (2.7^2)*0)/(10000+2.7^2) sigma <- (10000*2.7^2)/(10000+2.7^2) for (i in 2:n) { mu[i] <- ( sigma[i-1]*x[i] + (2.7^2)*mu[i-1] )/(sigma[i-1]+2.7^2) sigma[i] <- ( sigma[i-1]*2.7^2 )/(sigma[i-1]+2.7^2) }  If you plot the results, you'll see how posterior approaches the estimated value (it's true value is marked by red line) as new data is accumulated. For learning more you can check those slides and Conjugate Bayesian analysis of the Gaussian distribution paper by Kevin P. Murphy. Check also http://stats.stackexchange.com/questions/200982/do-bayesian-priors-become-irrelevant-with-large-sample-size/201059#201059 You can also check those notes and this blog entry for accessible step-by-step introduction to Bayesian inference. The basic idea of Bayesian updating is that given some data $$X$$ and prior over parameter of interest $$\theta$$, where the relation between data and parameter is described using likelihood function, you use Bayes theorem to obtain posterior $$p(\theta \mid X) \propto p(X \mid \theta) \, p(\theta)$$ This can be done sequentially, where after seeing first data point $$x_1$$ prior $$\theta$$ becomes updated to posterior $$\theta'$$, next you can take second data point $$x_2$$ and use posterior obtained before $$\theta'$$ as your prior, to update it once again etc. Let me give you an example. Imagine that you want to estimate mean $$\mu$$ of normal distribution and $$\sigma^2$$ is known to you. In such case we can use normal-normal model. We assume normal prior for $$\mu$$ with hyperparameters $$\mu_0,\sigma_0^2:$$ $$X\mid\mu \sim \mathrm{Normal}(\mu, \sigma^2) \\ \mu \sim \mathrm{Normal}(\mu_0, \sigma_0^2)$$ Since normal distribution is a conjugate prior for $$\mu$$ of normal distribution, we have closed-form solution to update the prior $$E(\mu' \mid x) = \frac{\sigma^2\mu + \sigma^2_0 x}{\sigma^2 + \sigma^2_0} \\ \mathrm{Var}(\mu' \mid x) = \frac{\sigma^2 \sigma^2_0}{\sigma^2 + \sigma^2_0}$$ Unfortunately, such simple closed-form solutions are not available for more sophisticated problems and you have to rely on optimization algorithms (for point estimates using maximum a posteriori approach), or MCMC simulation. Below you can see data example: n <- 1000 set.seed(123) x <- rnorm(n, 1.4, 2.7) mu <- numeric(n) sigma <- numeric(n) mu <- (10000*x[i] + (2.7^2)*0)/(10000+2.7^2) sigma <- (10000*2.7^2)/(10000+2.7^2) for (i in 2:n) { mu[i] <- (sigma[i-1]*x[i] + (2.7^2)*mu[i-1])/(sigma[i-1]+2.7^2) sigma[i] <- (sigma[i-1]*2.7^2)/(sigma[i-1]+2.7^2) }  If you plot the results, you'll see how posterior approaches the estimated value (it's true value is marked by red line) as new data is accumulated. For learning more you can check those slides and Conjugate Bayesian analysis of the Gaussian distribution paper by Kevin P. Murphy. Check also http://stats.stackexchange.com/questions/200982/do-bayesian-priors-become-irrelevant-with-large-sample-size/201059#201059 You can also check those notes and this blog entry for accessible step-by-step introduction to Bayesian inference. The basic idea of Bayesian updating is that given some data $$X$$ and prior over parameter of interest $$\theta$$, where the relation between data and parameter is described using likelihood function, you use Bayes theorem to obtain posterior $$p(\theta \mid X) \propto p(X \mid \theta) \, p(\theta)$$ This can be done sequentially, where after seeing first data point $$x_1$$ prior $$\theta$$ becomes updated to posterior $$\theta'$$, next you can take second data point $$x_2$$ and use posterior obtained before $$\theta'$$ as your prior, to update it once again etc. Let me give you an example. Imagine that you want to estimate mean $$\mu$$ of normal distribution and $$\sigma^2$$ is known to you. In such case we can use normal-normal model. We assume normal prior for $$\mu$$ with hyperparameters $$\mu_0,\sigma_0^2:$$ \begin{align} X\mid\mu &\sim \mathrm{Normal}(\mu,\ \sigma^2) \\ \mu &\sim \mathrm{Normal}(\mu_0,\ \sigma_0^2) \end{align} Since normal distribution is a conjugate prior for $$\mu$$ of normal distribution, we have closed-form solution to update the prior \begin{align} E(\mu' \mid x) &= \frac{\sigma^2\mu + \sigma^2_0 x}{\sigma^2 + \sigma^2_0} \\[7pt] \mathrm{Var}(\mu' \mid x) &= \frac{\sigma^2 \sigma^2_0}{\sigma^2 + \sigma^2_0} \end{align} Unfortunately, such simple closed-form solutions are not available for more sophisticated problems and you have to rely on optimization algorithms (for point estimates using maximum a posteriori approach), or MCMC simulation. Below you can see data example: n <- 1000 set.seed(123) x <- rnorm(n, 1.4, 2.7) mu <- numeric(n) sigma <- numeric(n) mu <- (10000*x[i] + (2.7^2)*0)/(10000+2.7^2) sigma <- (10000*2.7^2)/(10000+2.7^2) for (i in 2:n) { mu[i] <- ( sigma[i-1]*x[i] + (2.7^2)*mu[i-1] )/(sigma[i-1]+2.7^2) sigma[i] <- ( sigma[i-1]*2.7^2 )/(sigma[i-1]+2.7^2) }  If you plot the results, you'll see how posterior approaches the estimated value (it's true value is marked by red line) as new data is accumulated. For learning more you can check those slides and Conjugate Bayesian analysis of the Gaussian distribution paper by Kevin P. Murphy. Check also http://stats.stackexchange.com/questions/200982/do-bayesian-priors-become-irrelevant-with-large-sample-size/201059#201059 You can also check those notes and this blog entry for accessible step-by-step introduction to Bayesian inference. 2 added 40 characters in body edited Sep 28 '16 at 10:07 Tim♦ 63.7k99 gold badges142142 silver badges241241 bronze badges The basic idea of Bayesian updating is that given some data $$X$$ and prior over parameter of interest $$\theta$$, where the relation between data and parameter is described using likelihood function, you use Bayes theorem to obtain posterior $$p(\theta \mid X) \propto p(X \mid \theta) \, p(\theta)$$ This can be done sequentially, where after seeing first data point $$x_1$$ prior $$\theta$$ becomes updated to posterior $$\theta'$$, next you can take second data point $$x_2$$ and use posterior obtained before $$\theta'$$ as your prior, to update it once again etc. Let me give you an example. Imagine that you want to estimate mean $$\mu$$ of normal distribution and $$\sigma^2$$ is known to you. In such case we can use normal-normal model. We assume normal prior for $$\mu$$ with hyperparameters $$\mu_0,\sigma_0^2:$$ $$X\mid\mu \sim \mathrm{Normal}(\mu, \sigma^2) \\ \mu \sim \mathrm{Normal}(\mu_0, \sigma_0^2)$$ Since normal distribution is a conjugate prior for $$\mu$$ of normal distribution, we have closed-form solution to update the prior $$E(\mu' \mid x) = \frac{\sigma^2\mu + \sigma^2_0 x}{\sigma^2 + \sigma^2_0} \\ \mathrm{Var}(\mu' \mid x) = \frac{\sigma^2 \sigma^2_0}{\sigma^2 + \sigma^2_0}$$ Unfortunately, such simple closed-form solutions are not available for more sophisticated problems and you have to rely on optimization algorithms (for point estimates using maximum a posteriori approach), or MCMC simulation. Below you can see data example: n <- 1000 set.seed(123) x <- rnorm(n, 1.4, 2.7) mu <- numeric(n) sigma <- numeric(n) mu <- (10000*x[i] + (2.7^2)*0)/(10000+2.7^2) sigma <- (10000*2.7^2)/(10000+2.7^2) for (i in 2:n) { mu[i] <- (sigma[i-1]*x[i] + (2.7^2)*mu[i-1])/(sigma[i-1]+2.7^2) sigma[i] <- (sigma[i-1]*2.7^2)/(sigma[i-1]+2.7^2) }  If you plot the results, you'll see how posterior approaches the estimated value (it's true value is marked by red line) as new data is accumulated. For learning more you can check those slides and Conjugate Bayesian analysis of the Gaussian distribution paper by Kevin P. Murphy. Check also http://stats.stackexchange.com/questions/200982/do-bayesian-priors-become-irrelevant-with-large-sample-size/201059#201059 You can also check those notes and this blog entry for accessible step-by-step introduction to Bayesian inference. The basic idea of Bayesian updating is that given some data $$X$$ and prior over parameter of interest $$\theta$$, where the relation between data and parameter is described using likelihood function, you use Bayes theorem to obtain posterior $$p(\theta \mid X) \propto p(X \mid \theta) \, p(\theta)$$ This can be done sequentially, where after seeing first data point $$x_1$$ prior $$\theta$$ becomes updated to posterior $$\theta'$$, next you can take second data point $$x_2$$ and use posterior obtained before $$\theta'$$ as your prior, to update it once again etc. Let me give you an example. Imagine that you want to estimate mean $$\mu$$ of normal distribution and $$\sigma^2$$ is known to you. In such case we can use normal-normal model. We assume normal prior for $$\mu$$ with hyperparameters $$\mu_0,\sigma_0^2:$$ $$X\mid\mu \sim \mathrm{Normal}(\mu, \sigma^2) \\ \mu \sim \mathrm{Normal}(\mu_0, \sigma_0^2)$$ Since normal distribution is a conjugate prior for $$\mu$$ of normal distribution, we have closed-form solution to update the prior $$E(\mu' \mid x) = \frac{\sigma^2\mu + \sigma^2_0 x}{\sigma^2 + \sigma^2_0} \\ \mathrm{Var}(\mu' \mid x) = \frac{\sigma^2 \sigma^2_0}{\sigma^2 + \sigma^2_0}$$ Unfortunately, such simple closed-form solutions are not available for more sophisticated problems and you have to rely on optimization algorithms (for point estimates using maximum a posteriori approach), or MCMC simulation. Below you can see data example: n <- 1000 set.seed(123) x <- rnorm(n, 1.4, 2.7) mu <- numeric(n) sigma <- numeric(n) mu <- (10000*x[i] + (2.7^2)*0)/(10000+2.7^2) sigma <- (10000*2.7^2)/(10000+2.7^2) for (i in 2:n) { mu[i] <- (sigma[i-1]*x[i] + (2.7^2)*mu[i-1])/(sigma[i-1]+2.7^2) sigma[i] <- (sigma[i-1]*2.7^2)/(sigma[i-1]+2.7^2) }  If you plot the results, you'll see how posterior approaches the estimated value as new data is accumulated. For learning more you can check those slides and Conjugate Bayesian analysis of the Gaussian distribution paper by Kevin P. Murphy. Check also http://stats.stackexchange.com/questions/200982/do-bayesian-priors-become-irrelevant-with-large-sample-size/201059#201059 You can also check those notes and this blog entry for accessible step-by-step introduction to Bayesian inference. The basic idea of Bayesian updating is that given some data $$X$$ and prior over parameter of interest $$\theta$$, where the relation between data and parameter is described using likelihood function, you use Bayes theorem to obtain posterior $$p(\theta \mid X) \propto p(X \mid \theta) \, p(\theta)$$ This can be done sequentially, where after seeing first data point $$x_1$$ prior $$\theta$$ becomes updated to posterior $$\theta'$$, next you can take second data point $$x_2$$ and use posterior obtained before $$\theta'$$ as your prior, to update it once again etc. Let me give you an example. Imagine that you want to estimate mean $$\mu$$ of normal distribution and $$\sigma^2$$ is known to you. In such case we can use normal-normal model. We assume normal prior for $$\mu$$ with hyperparameters $$\mu_0,\sigma_0^2:$$ $$X\mid\mu \sim \mathrm{Normal}(\mu, \sigma^2) \\ \mu \sim \mathrm{Normal}(\mu_0, \sigma_0^2)$$ Since normal distribution is a conjugate prior for $$\mu$$ of normal distribution, we have closed-form solution to update the prior $$E(\mu' \mid x) = \frac{\sigma^2\mu + \sigma^2_0 x}{\sigma^2 + \sigma^2_0} \\ \mathrm{Var}(\mu' \mid x) = \frac{\sigma^2 \sigma^2_0}{\sigma^2 + \sigma^2_0}$$ Unfortunately, such simple closed-form solutions are not available for more sophisticated problems and you have to rely on optimization algorithms (for point estimates using maximum a posteriori approach), or MCMC simulation. Below you can see data example: n <- 1000 set.seed(123) x <- rnorm(n, 1.4, 2.7) mu <- numeric(n) sigma <- numeric(n) mu <- (10000*x[i] + (2.7^2)*0)/(10000+2.7^2) sigma <- (10000*2.7^2)/(10000+2.7^2) for (i in 2:n) { mu[i] <- (sigma[i-1]*x[i] + (2.7^2)*mu[i-1])/(sigma[i-1]+2.7^2) sigma[i] <- (sigma[i-1]*2.7^2)/(sigma[i-1]+2.7^2) }  If you plot the results, you'll see how posterior approaches the estimated value (it's true value is marked by red line) as new data is accumulated. For learning more you can check those slides and Conjugate Bayesian analysis of the Gaussian distribution paper by Kevin P. Murphy. Check also http://stats.stackexchange.com/questions/200982/do-bayesian-priors-become-irrelevant-with-large-sample-size/201059#201059 You can also check those notes and this blog entry for accessible step-by-step introduction to Bayesian inference. 1 answered Sep 27 '16 at 8:02 Tim♦ 63.7k99 gold badges142142 silver badges241241 bronze badges