I'm trying to understand Gaussian Processes. Could anyone tell me:

  1. Why we need to use the log marginal likelihood?
  2. Why using log, the marginal likelihood can be decomposed to 3 terms (including a fitting term and a penalty term)?
  • Con you please add some more details? However, the marginal likelihood is generally used to have a measure of how the model fitting. You can find the marginal likelihood of a process as the marginalization over the set of parameters that govern the process This integral is generally not available and cannot be computed in closed form. However, an approximation can be found with the sum of the complete likelihood and a penalization term, that I suppose is the decomposition that you mentioned on point 2. The likelihood is generally computed in logarithmic scale for numerical stability reason. – niandra82 Jul 16 '14 at 21:21
  • @niandra82 thank you so much for your comment~ Please could you explain why we compute the marginal likelihood in logarithmic scale for numerical stability reason? How can such a form increase the stability~ thank you for your help once more. – kakanana Jul 19 '14 at 11:17
  • I can try to explain. COnsider a computer that can store only numbers between 99,000 and 0.001 (only three decimal) plus the sign. If you compute a density and in some point this has value 0.0023456789, in the computer this will be stored as 0.003 losing part of the real value, if you compute it in log scale the $log(0.0023456789)=-6.05518$ will be stored as $-6.055$ losing less than in original scale. If you multiply a lot of small values, the situation get worst: consider $0.0023456789^2$ that will be store as $0$ while $log(0.0023456789^2)=-12.11036$ – niandra82 Jul 19 '14 at 15:08
  • If you think this satisfy your question, tell me and i put the comment in an "answer" – niandra82 Jul 19 '14 at 15:10
  • @niandra82 thanks for your comments~~ please put them in the 'answer'~ by the way, would you mind to explain how can Cross Validation and Gaussian Process overcome over-fitting problems? – kakanana Jul 20 '14 at 9:42
up vote 4 down vote accepted

The marginal likelihood is generally used to have a measure of how the model fitting. You can find the marginal likelihood of a process as the marginalization over the set of parameters that govern the process This integral is generally not available and cannot be computed in closed form. However, an approximation can be found with the sum of the complete likelihood and a penalization term, that I suppose is the decomposition that you mentioned on point 2.

The likelihood is generally computed in logarithmic scale for numerical stability reason: consider a computer that can store only numbers between 99,000 and 0.001 (only three decimal) plus the sign. If you compute a density and in some point this has value 0.0023456789, in the computer this will be stored as 0.003 losing part of the real value, if you compute it in log scale the log(0.0023456789)=−6.05518 will be stored as −6.055 losing less than in original scale. If you multiply a lot of small values, the situation get worst: consider 0.00234567892 that will be store as 0 while log(0.00234567892)=−12.11036

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.