# Can we get probabilistic predictions evaluable by proper scoring rules from bayesian inference without evaluating the marginal likelihood?

Let's say we have a vector of inputs, $$X=[x_0,\dots, x_{n-1}]$$, and a vector of outputs, $$Y=[y_0, \dots, y_{n-1}]$$.

We would like to predict the distribution of a new output ,$$\hat{y}$$, given a new input, $$\hat{x}$$.

Bayesian inference gives us a way to do that by way of the posterior predictive distribution which in this case would be

$$p(\hat{y}|\hat{x}, X, Y)=\int p(\hat{y}|\hat{x}, \theta)p(\theta|X,Y)\text{d}\theta$$

with the posterior being further defined as $$p(\theta|X,Y)=\frac{p(Y|X,\theta)p(\theta)}{\int p(Y|X,\theta)p(\theta)\text{d}\theta}.$$

With the denominator of $$p(\theta|X,Y)$$ being sometimes difficult or time-consuming to evaluate numerically when there isn't a closed form or conjugate prior, is there a way to be able to get some sort of density prediction without evaluating the denominator and which can then be evaluated by a proper scoring rule such as the negatively-oriented log score?

Defined as follows for a density prediction, $$f_{\hat{y}}(y)$$, and an observed value, $$y$$:

$$L(y, f_{\hat{y}}) = -\text{log}(f_{\hat{y}}(y)).$$

I feel like we would run into problems with the density prediction not being a true density (i.e. not integrating to 1) and this would give us an incorrect evaluation when evaluating the scoring rule using it even if we could still technically calculate it.

• Interesting question. I would assume that this does not work, because the denominator turns into an additive offset in the log score through the logging, and competing predictions will of course differ in exactly this normalizing denominator... so if we don't normalize the densities by dividing, we don't know whether differences in the score come from better density predictions or from differences in this denominator. I would be happy to be proven wrong! Commented Apr 19 at 6:40
• Good point, @StephanKolassa I think you might be correct. I'm eager to see if there's another way to approach it or if just figuring out how to better evaluate those integrals is the direction to go in :) Commented Apr 20 at 2:52

You can do this, but as Stephan Kolassa argued in the comment, not with the logarithmic scoring rule. Instead, for a predictive density $$f$$ which is twice continuously differentiable, you can use the Hyvärinen score, given by $$S(f,y) = 2 \frac{f''(y)}{f(y)} - \left( \frac{f'(y)}{f(y)} \right)^2$$ It is a strictly proper scoring rule and can be calculated when knowing $$f$$ only up to a normalizing constant, since $$S(cf,y) = S(f,y)$$ for all $$c>0$$. Apart from the logarithmic score it is probably the most prominent example of a local proper scoring rule, i.e. a scoring rule which evaluates the forecast only at the observation $$y$$.