Probably the easiest to digest example is a simple detection task.
Say you make a measurement $X_m$ of some system and you want to detect whether or not a DC (constant) signal with amplitude $\mu$ is present.
However, there is additive white noise in your measurement $\eta \sim \mathcal{N}(0,\sigma^2_\eta)$.
If the signal is present, your measurement is $X_{m|S} = \mu + \eta$, but if not, $X_{m|0}=\eta$.
$X_{m|S}\sim \mathcal{N}(\mu,\sigma^2_\eta)$.
Naturally, you form two hypotheses.
- the null hypothesis---the signal is not present.
- the alternative hypotheses---the signal is present.
The optimal approach is to look up the likelihoods of $X_m$ under each hypothesis and compare them via a log-likelihood ratio:
$$\text{LLR}=\log\frac{\mathcal L(X_m|H_1)}{\mathcal L(X_m|H_0)}$$
If the LLR is greater than or equal, then reject the null hypothesis, otherwise accept the null. Writing this out explicitly for our example:
$$
\begin{align*}
\text{LLR}=&\log \frac{\exp \Big(-\frac{(x-\mu)^2}{2\sigma_\eta^2} \Big)}
{\exp \Big(-\frac{x^2}{2\sigma_\eta^2} \Big)}\\
=&\frac{x^2}{2\sigma_\eta^2} -\frac{(x-\mu)^2}{2\sigma_\eta^2}.
\end{align*}
$$
This will return the LLR for a specific outcome $x$.
However, if we want to know how detectable our signal is on average, we can take the expectation of the LLR under the alternative hypothesis:
$$\mathbb E[\text{LLR}|H_1]=\mathcal L(X_m|H_1) \log\frac{\mathcal L(X_m|H_1)}{\mathcal L(X_m|H_0)},$$
which is the definition for KL-divergence $D_{KL}(\ \mathcal p(X_m|H_1)\ ||\ \mathcal p(X_m|H_0))$.
Alternatively, if we wanted to find how undetected our signal is instead, we would commute the arguments for KL divergence (expectation of LLR under the null hypothesis).
Explicitly we can write out the full statement for our example as
$$\mathbb E[\text{LLR}|H_1]=\int_{-\infty}^\infty dx\ \frac{1}{\sqrt{2\pi}\sigma} \exp \Big(-\frac{(x-\mu)^2}{2\sigma_\eta^2} \Big)\Big(\frac{x\mu}{\sigma_\eta^2} -\frac{\mu^2}{2\sigma_\eta^2}\Big),$$
which simplifies to
$$\frac{\mu^2}{2\sigma^2}=\frac{1}{2}\text{SNR}.$$
As far as I'm aware, the SNR under this specific definition is only proportional to KL divergence for additive white noise.
We can also write this as
$\frac{(\mu-\mu_\eta)^2}{2\sigma^2},$
where $\mu_\eta$ is the DC offset for the noise.
Take the square root and you have
$\frac{1}{\sqrt{2}}\frac{|\mu-\mu_\eta|}{\sigma}=\frac{1}{\sqrt{2}}d',$
where $d'$ (d-prime) is called the sensitivity index which is the favorite statistic used in signal detection theory.
If you take $\Phi^{-1}\Big(\frac{d'}{2}\Big)$, $\Phi^{-1}$ being the inverse normal CDF, you would get the proportion of trials you would expect to get this correct $\text{PC}$ for multiple draws of $X_m$, that is $$PC=\frac{\text{true positives }+ \text{true negatives}}{\text{total trials}}.$$
As far as predictive modelling goes, you can think of noise as making your signal less correlated/indicative of the outcome.
When you train your model, you are training your model to first detect these correlations. Prediction is then an extrapolation from what has been learned. If your model has a hard time detecting the signal in known conditions, its going to have an even harder time detecting the signal in new conditions. Note that the above formulations are considered optimal (que Neyman-Pearson lemma). Thus, something like SNR will give you an upper bound on your expected prediction performance.
