Formula for "true distribution" in machine learning problems

Question

In machine learning we often use the concept of a "true distribution" $p_{\mathrm{true}}$ which is sometimes called "nature's distribution" or "target distribution". I tried to express it analytically using the function $\tilde y(x) = \mathrm{target}(x)$, which maps every sample $x \in \mathcal{X}$ to its true label $\tilde y(x) \in \mathcal{Y}$.

So, whether the following statements are true?

1. In the multiclass classification task we have: $p_{\mathrm{true}}(y|x) = \begin{cases}1, &\text{if } y = \tilde y(x) \\ 0, &\text{else}\end{cases};$
2. In the regression task we have $p_{\mathrm{true}}(y|x) = \delta(y - \tilde{y}(x))$, where $\delta(z)$ is the Dirac delta function.

In addition, I will be happy if you give me some links where I can read about this in detail.

Answer 1

"True distribution" is the distribution of your data, it doesn't have analytical form. Moreover, you write it as some kind of distribution that considers function of $X$, while this doesn't have to be the case. For example, you could use ice-cream sales to predict sunny weather, this doesn't mean that there's a causal relationship that makes the weather sunny when we sell more ice cream. The trud distribution in here, is the actual distribution of those two events, their dependence structure and interdependences with all the other, possibly unobserved, factors that take part in this process. Another example, the "true distribution" of the weather is an extremely complicated, chaotic, process that cannot be observed directly. We don't know and can't define the "true" function that produces weather. There's no analytic form for that. The true distribution is an abstract concept.

As about your notation, you seem to be stating the tautology “$y$ is $y$ when it’s $y$”. For those to describe the distribution, you’d need to know the “true” $p(y|x)$.