5
$\begingroup$

I'm looking for references of Gaussian Processes and identification issues that may occur.

For example, in Kennedy and O'Hagan's (2001) Bayesian Calibration of Computer Models, we have $$y_i=\eta(x_i,\theta)+\delta(x_i)+e_i$$ where $\delta$ has a gaussian process prior, and $y_i, x_i$ are observable data. Then, on the discussion of this paper, several researchers point to the potential lack of identifiability when we try to estimate $\eta$ and the process governing $\delta$, even when have $e_i=0$ (which we don't).

The response/answer of the original authors to this is to rewrite the previous equation as $$y_i=\eta(x_i,\theta)+\epsilon(x_i)$$ and simply say that these components are intrinsically well-defined as in any regression problem... Why is that?Also, this doesn't seem to truly answer the criticism...

Someone told me that the same identification issue would happen if, instead of considering a Gaussian Process, we were considering a random effect. Is this true? any references?

Improve this question
$\endgroup$
3
  • 1
    $\begingroup$ +1 This has been discussed in the kriging literature for at least 30 years. I recall that Andre Journel (Stanford U.) and colleagues recognized that the decomposition of a random field into a deterministic "drift" $\eta,$ a random component $\delta,$ and (optionally) iid measurement errors $\epsilon$ was a modeling decision, rather than something that could be determined from data, in part because of the lack of identifiability. This became apparent in various studies that had analysts independently analyze common datasets. $\endgroup$
    – whuber ♦
    Commented Aug 27, 2019 at 13:27
  • $\begingroup$ @whuber thanks for the comment. By any chance do you have a specific reference in mind? $\endgroup$
    – An old man in the sea.
    Commented Aug 27, 2019 at 13:38
  • $\begingroup$ I remember commenting on an article of Journel's in J. Env. Ecol. Stats c. 1992-1994 that focused on this issue. (That was so long ago I can't find any reference to it on the Web.) I recall reading about this topic more recently but can't remember any article specifically. $\endgroup$
    – whuber ♦
    Commented Aug 27, 2019 at 14:09

1 Answer 1

Reset to default
2
$\begingroup$

There has been a lot of work, since the original Kennedy and O'Hagan paper, discussing the lack of identifiability of the Bayesian model calibration framework. One paper that I like in particular is this one by Arendt, Apley and Chen. From the abstract:

One of the main challenges in model updating is the difficulty in distinguishing between the effects of calibration parameters versus model discrepancy. We illustrate this identifiability problem with several examples, explain the mechanisms behind it, and attempt to shed light on when a system may or may not be identifiable. In some instances, identifiability is achievable under mild assumptions, whereas in other instances, it is virtually impossible.

Here is one of the figures from the paper.

Figure 8 - Apley et al (2012)

This figure demonstrates that in many cases, the estimates of the calibration parameters $\theta$ (bottom right panel) and the estimate of the discrepancy function $\delta$ (bottom left panel) fail to capture the true values. Yet the fit to the data (top left panel) is arbitrarily good nonetheless.

Another related paper is this one by Brynjarsdottir and O'Hagan (same O'Hagan as the original 2001 reference). They show how the identifiability can often be improved by incorporating meaningful constraints into the prior distributions.

As to your last question about random effects: I can't give you a definitive answer, but I don't immediately see why that would be the case. This paper, Adding Spatially-Correlated Errors Can Mess Up the Fixed Effect You Love, may be relevant.

In order of appearance in the discussion:

Arendt, Paul D., Daniel W. Apley, and Wei Chen. "Quantification of model uncertainty: Calibration, model discrepancy, and identifiability." Journal of Mechanical Design 134.10 (2012): 100908.

Brynjarsdóttir, Jenný, and Anthony OʼHagan. "Learning about physical parameters: The importance of model discrepancy." Inverse problems 30.11 (2014): 114007.

Hodges, James S., and Brian J. Reich. "Adding spatially-correlated errors can mess up the fixed effect you love." The American Statistician 64.4 (2010): 325-334.

Improve this answer
$\endgroup$
6
  • 1
    $\begingroup$ Thanks! I had completely forgotten about this question of mine. Just to complement it, I later found some more references. For identifiability of GP, see Orthogonal Gaussian Processes by Plumlee. For a more updated view of identifiability in UQ using GPs, Arendt et al wrote a chapter in the Handbook of Uncertainty Quantification. There were a few other references which I can't remember right now... $\endgroup$
    – An old man in the sea.
    Commented Sep 28, 2019 at 11:17
  • $\begingroup$ Incidentally, what's the topic of your thesis? ;) $\endgroup$
    – An old man in the sea.
    Commented Sep 28, 2019 at 11:18
  • 1
    $\begingroup$ The Plumlee reference is another good one. I'm working on robust strategies for UQ when the inverse problem (parameter estimation) is the primary goal. I am actually almost finished with a paper on improving identifiability on a subset of calibration parameters. I'll add the link here if I ever put it onto Arxiv. (: $\endgroup$
    – knrumsey
    Commented Sep 28, 2019 at 16:02
  • $\begingroup$ Hey, today I got a +1 on this question, and I thought I would ask you how your phd/paper is going. I hope everything is going well. $\endgroup$
    – An old man in the sea.
    Commented Jul 31, 2020 at 23:57
  • $\begingroup$ @Anoldmaninthesea. Thanks for asking! I just defended my dissertation a few weeks ago. The paper in question has been accepted for publication in Journal of Uncertainty Quantification and should be available in a few weeks. I'll replace this comment with a link when it's available. $\endgroup$
    – knrumsey
    Commented Aug 1, 2020 at 19:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.