What are the limitations of Gaussian process regression and gaussian response surface methodologies? In which scenarios other techniques might be preferable over Gaussian process regression?
GPs assume a gaussian uncertainty on the y-values. However, this may not be the type of uncertainty that you have. For example let us assume the output values are strictly positive, or bounded between two values, then the gaussian prior would be inappropriate (or used only as an approximation).
SVMs are somewhat similar as they are kernel based regression models with which you can choose your loss function. However they don't offer a probabilistic interpretation (which is a big no no if you are a die hard bayesian for example).
Kernel methods versus random forests or neural nets have other trade offs. A GP kernel allows us to specify a prior on our function space which can be extremely useful especially when we have little data. However, a poor choice in kernel which specifies misconceptions about the function space can make conversion slow. Specifying appropriate kernels beyond the most basic requires some mathematical understanding. On the other hand random forests and neural nets are completely frequentist (in general) and so usually require more data in order to get decent predictive performance.