Both Gaussian processes (GP) and Neural Networks (NN) are regression techniques that assume no underlying functional form of the response.

How can these two methodologies be compared?

What are the unique feature that a Gaussian process has that neural networks don’t and vice versa?

Also in the application of multivariate regression, both GP and NN can handle multiple outputs? But in what cases should the GP be used vs NN ?


1 Answer 1


Gaussian processes are suitable for modelling small datasets where some prior knowledge of the generative process exists. GPs do require assumptions about the functional form of the underlying response. GPs do not scale well in terms of dimensionality. They may provide well calibrated uncertainty output.

Neural networks are, on the other hand, more suitable for large and very large data sets where little knowledge about the underlying process or suitable features exist. NNs scale well. Work is being done to enable neural networks to output calibrated uncertainty estimates.

  • $\begingroup$ Could you elaborate on what you mean when you say GPs don't scale in dimensionality? A kernel like the standard RBF only interacts with dimensionality by computing a subtraction and an inner product, and the linear kernel is just an inner product. On the other hand, GPs have cubic complexity in terms of the number of observations, which would seem to be the larger concern. $\endgroup$
    – Sycorax
    Mar 6, 2017 at 17:59
  • $\begingroup$ Standard RBF without simplifying assumptions has $1+D^2$ hyperparameters - evaluating the Gram matrix with a specific kernel/hyperparameter tuple is just fine, its inference of the kernel/hyperparameters that becomes tricky. $\endgroup$
    – Oxonon
    Mar 6, 2017 at 18:17
  • $\begingroup$ As to the cubic in observations, approximations exist that make that ~ linear. $\endgroup$
    – Oxonon
    Mar 6, 2017 at 18:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.