Skip to main content
Cross Validated

Return to Answer

added 1 character in body
Source Link
Ben
Ben
  • 132.9k
  • 7
  • 255
  • 588

Using your data to construct the prior is bad; using it for starting points for MCMC is okay

While you don't ask an explicit question here, I will take it that your inquiry comes out of a confusion between what you take to be a tension between the pieces of advice given by myself and Frank. Here I will explain why I think that there is no contradiction in the various pieces of advice you have been given.

My view is that all these pieces of advice are correct (though obviously I gave one of those pieces of advice, so I am just repeating it here) and so you are likely to just get agreements with these ideas here, rather than any lively discussion. Both Frank and I caution against incorporating information from the data into the prior distribution (e.g., by using the MLE to form the prior) and we do so for effectively the same reasons. Incorporating information from the data into the prior distribution breaks the proper rules for Bayesian updating and can lead to bias in your analysis. In particular, Frank is correct to point out that the type of bias it induces is that there is inflation of effective sample sizes. This occurs because the data is effectively being incorporated twice --- once to form the prior and then again to update the prior to a posterior. (You could reasonably regard this as a special type of "overfitting" of a model to data.)

As to Frank's suggestion that you could use the MLE as a starting point for MCMC simulation of the posterior, this is also a good suggestion. To understand why this is okay, it is important to note that the posterior distribution simulated in MCMC is determined by the asymptotic properties of the Markov chain, not by the starting point of the chain. The starting point for iterations of MCMC methods does not change the posterior distribution being simulated, so it does not constitute a situation where you would be considered to be "double-counting" your data. However, using a well-informed starting point (e.g., the MLE) does allow you to start in an area of the posterior distribution that is likely to be "in the body" of the distribution, which will mean that the MCMC iteration might search around the main part of the distribution a bit more quickly and efficiently. That is really just an advantage in making your numerical simulations a bit more efficient, but it is potentially an advantage nonetheless.

It is worth noting that similar methods are used in other areas of numerical simulation in statistics, where we use the data to get a good starting point for a numerical simulation that is going to compute some estimator that substantively depends on the data. For example, even when computing the MLE in an estimation problem, it is usually the case that this does not have a closed forform solution and so it is computed with iterative numerical methods (e.g., Newton-Raphson). In this computation, it is often useful to start the iterative method at some value that is a crude closed-form estimator of the parameters that is determined by the data. Again, the idea is to get a starting value that is "close" to the main area of where you want to iterate in your numerical method. The fact that the starting point for iteration depends on the data does not breach standard estimation rules in this case because it does not change the substantive asymptotic properties of the iterative method (it just makes it a bit faster and more efficient).

Using your data to construct the prior is bad; using it for starting points for MCMC is okay

While you don't ask an explicit question here, I will take it that your inquiry comes out of a confusion between what you take to be a tension between the pieces of advice given by myself and Frank. Here I will explain why I think that there is no contradiction in the various pieces of advice you have been given.

My view is that all these pieces of advice are correct (though obviously I gave one of those pieces of advice, so I am just repeating it here) and so you are likely to just get agreements with these ideas here, rather than any lively discussion. Both Frank and I caution against incorporating information from the data into the prior distribution (e.g., by using the MLE to form the prior) and we do so for effectively the same reasons. Incorporating information from the data into the prior distribution breaks the proper rules for Bayesian updating and can lead to bias in your analysis. In particular, Frank is correct to point out that the type of bias it induces is that there is inflation of effective sample sizes. This occurs because the data is effectively being incorporated twice --- once to form the prior and then again to update the prior to a posterior. (You could reasonably regard this as a special type of "overfitting" of a model to data.)

As to Frank's suggestion that you could use the MLE as a starting point for MCMC simulation of the posterior, this is also a good suggestion. To understand why this is okay, it is important to note that the posterior distribution simulated in MCMC is determined by the asymptotic properties of the Markov chain, not by the starting point of the chain. The starting point for iterations of MCMC methods does not change the posterior distribution being simulated, so it does not constitute a situation where you would be considered to be "double-counting" your data. However, using a well-informed starting point (e.g., the MLE) does allow you to start in an area of the posterior distribution that is likely to be "in the body" of the distribution, which will mean that the MCMC iteration might search around the main part of the distribution a bit more quickly and efficiently. That is really just an advantage in making your numerical simulations a bit more efficient, but it is potentially an advantage nonetheless.

It is worth noting that similar methods are used in other areas of numerical simulation in statistics, where we use the data to get a good starting point for a numerical simulation that is going to compute some estimator that substantively depends on the data. For example, even when computing the MLE in an estimation problem, it is usually the case that this does not have a closed for solution and so it is computed with iterative numerical methods (e.g., Newton-Raphson). In this computation, it is often useful to start the iterative method at some value that is a crude closed-form estimator of the parameters that is determined by the data. Again, the idea is to get a starting value that is "close" to the main area of where you want to iterate in your numerical method. The fact that the starting point for iteration depends on the data does not breach standard estimation rules in this case because it does not change the substantive asymptotic properties of the iterative method (it just makes it a bit faster and more efficient).

Using your data to construct the prior is bad; using it for starting points for MCMC is okay

While you don't ask an explicit question here, I will take it that your inquiry comes out of a confusion between what you take to be a tension between the pieces of advice given by myself and Frank. Here I will explain why I think that there is no contradiction in the various pieces of advice you have been given.

My view is that all these pieces of advice are correct (though obviously I gave one of those pieces of advice, so I am just repeating it here) and so you are likely to just get agreements with these ideas here, rather than any lively discussion. Both Frank and I caution against incorporating information from the data into the prior distribution (e.g., by using the MLE to form the prior) and we do so for effectively the same reasons. Incorporating information from the data into the prior distribution breaks the proper rules for Bayesian updating and can lead to bias in your analysis. In particular, Frank is correct to point out that the type of bias it induces is that there is inflation of effective sample sizes. This occurs because the data is effectively being incorporated twice --- once to form the prior and then again to update the prior to a posterior. (You could reasonably regard this as a special type of "overfitting" of a model to data.)

As to Frank's suggestion that you could use the MLE as a starting point for MCMC simulation of the posterior, this is also a good suggestion. To understand why this is okay, it is important to note that the posterior distribution simulated in MCMC is determined by the asymptotic properties of the Markov chain, not by the starting point of the chain. The starting point for iterations of MCMC methods does not change the posterior distribution being simulated, so it does not constitute a situation where you would be considered to be "double-counting" your data. However, using a well-informed starting point (e.g., the MLE) does allow you to start in an area of the posterior distribution that is likely to be "in the body" of the distribution, which will mean that the MCMC iteration might search around the main part of the distribution a bit more quickly and efficiently. That is really just an advantage in making your numerical simulations a bit more efficient, but it is potentially an advantage nonetheless.

It is worth noting that similar methods are used in other areas of numerical simulation in statistics, where we use the data to get a good starting point for a numerical simulation that is going to compute some estimator that substantively depends on the data. For example, even when computing the MLE in an estimation problem, it is usually the case that this does not have a closed form solution and so it is computed with iterative numerical methods (e.g., Newton-Raphson). In this computation, it is often useful to start the iterative method at some value that is a crude closed-form estimator of the parameters that is determined by the data. Again, the idea is to get a starting value that is "close" to the main area of where you want to iterate in your numerical method. The fact that the starting point for iteration depends on the data does not breach standard estimation rules in this case because it does not change the substantive asymptotic properties of the iterative method (it just makes it a bit faster and more efficient).

Source Link
Ben
Ben
  • 132.9k
  • 7
  • 255
  • 588

Using your data to construct the prior is bad; using it for starting points for MCMC is okay

While you don't ask an explicit question here, I will take it that your inquiry comes out of a confusion between what you take to be a tension between the pieces of advice given by myself and Frank. Here I will explain why I think that there is no contradiction in the various pieces of advice you have been given.

My view is that all these pieces of advice are correct (though obviously I gave one of those pieces of advice, so I am just repeating it here) and so you are likely to just get agreements with these ideas here, rather than any lively discussion. Both Frank and I caution against incorporating information from the data into the prior distribution (e.g., by using the MLE to form the prior) and we do so for effectively the same reasons. Incorporating information from the data into the prior distribution breaks the proper rules for Bayesian updating and can lead to bias in your analysis. In particular, Frank is correct to point out that the type of bias it induces is that there is inflation of effective sample sizes. This occurs because the data is effectively being incorporated twice --- once to form the prior and then again to update the prior to a posterior. (You could reasonably regard this as a special type of "overfitting" of a model to data.)

As to Frank's suggestion that you could use the MLE as a starting point for MCMC simulation of the posterior, this is also a good suggestion. To understand why this is okay, it is important to note that the posterior distribution simulated in MCMC is determined by the asymptotic properties of the Markov chain, not by the starting point of the chain. The starting point for iterations of MCMC methods does not change the posterior distribution being simulated, so it does not constitute a situation where you would be considered to be "double-counting" your data. However, using a well-informed starting point (e.g., the MLE) does allow you to start in an area of the posterior distribution that is likely to be "in the body" of the distribution, which will mean that the MCMC iteration might search around the main part of the distribution a bit more quickly and efficiently. That is really just an advantage in making your numerical simulations a bit more efficient, but it is potentially an advantage nonetheless.

It is worth noting that similar methods are used in other areas of numerical simulation in statistics, where we use the data to get a good starting point for a numerical simulation that is going to compute some estimator that substantively depends on the data. For example, even when computing the MLE in an estimation problem, it is usually the case that this does not have a closed for solution and so it is computed with iterative numerical methods (e.g., Newton-Raphson). In this computation, it is often useful to start the iterative method at some value that is a crude closed-form estimator of the parameters that is determined by the data. Again, the idea is to get a starting value that is "close" to the main area of where you want to iterate in your numerical method. The fact that the starting point for iteration depends on the data does not breach standard estimation rules in this case because it does not change the substantive asymptotic properties of the iterative method (it just makes it a bit faster and more efficient).