1
$\begingroup$

After training a Gaussian Process (GP), predictions for any input is the mean a probability distribution (posterior) and the uncertainty in making this prediction is the variance of this posterior. However, in other ML models (like neural networks) we have no direct way to extract uncertainty (an indirect way would be to use dropout when making predictions, see here).

My question is: is there a clear/agreed-upon terminology to differentiate between models that offer a direct way to compute uncertainty (like GPs) and those that don't (neural networks)? Intuitively, I referred to models like GPs as probabilistic models (since predictions are probability distributions) and models like neural networks as non-probabilistic. However, a friend of mine objected saying that a neural network employing stochastic gradient descent in training is also a probabilistic model.

$\endgroup$
2
  • 1
    $\begingroup$ What is your friend's definition of a probabilistic model? When I hear the term I think of a model based on random variables and their distributions. I don't feel that it is enough to have the word "stochastic" nearby. $\endgroup$
    – einar
    Jan 20, 2021 at 10:37
  • $\begingroup$ I agree. The mere use of a stochastic optimisation algorithm does not make the model probabilistic. To me, a probabilistic model is a model that outputs a distribution for any prediction. I don't know if this is an agreed upon terminology. $\endgroup$
    – Saleh
    Jan 20, 2021 at 18:40

1 Answer 1

1
$\begingroup$

Some people like to refer to GPs as "emulators" (and GP regression as "emulation") for the reasons you describe.

As far as I'm aware, this term was coined by Tony O'Hagan. You can find a page indexing his academic work here.

$\endgroup$
2
  • $\begingroup$ Thanks for the answer. Are there any other emulators? i.e. are there other ML model where you can directly quantify the uncertainty of a prediction? $\endgroup$
    – Saleh
    Jan 20, 2021 at 18:45
  • $\begingroup$ I've personally only ever heard the term applied to GP models, but this might be more of a reflection on the literature I'm familiar with than it is the applicability of the term! $\endgroup$
    – rxFt20
    Jan 21, 2021 at 14:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.