What is probability distribution for a machine learning task? I am new to machine learning. I feel confused of how to understand the probability distribution of the training set, like $p(y|x)$ and $p(x)$, where $x$ is a training sample and $y$ is a label. Is there any concrete example to understand this? Further, it's said that the "ground truth" distribution is unknown and once you know it, there is nothing to learning. Could you help me clarify these words? Moreover, is it fair to say that a neural network or a deep learning model is learning the distribution of data? Thank you!
 A: $p(y|x)$ represents the probability of a label given a sample, sometimes called the posterior, while $p(x)$ represents the probability (or density) of obtaining that specific sample. For example, in classification, we typically choose the class that maximize the posterior, i.e. $p(y|x)$. For example, $x$ can be a feature vector consisting of word counts in an e-mail and $y$ can be the label of it being spam or not.

it's said that the "ground truth" distribution is unknown and once you
  know it, there is nothing to learning

Let's say you've some data (e.g. think 1D), sampled from a Gaussian distribution. Even if you knew that the data is sampled from a Gaussian, you'd still need to estimate the distribution, i.e. its mean and variance and decide on some kind of estimates of true $\mu,\sigma$, i.e. $\hat{\mu},\hat{\sigma}$. Then, you can claim that the data distribution can be well approximated with $\mathcal{N}(\hat{\mu},\hat{\sigma})$, however it's not the true distribution. If you had known the true values $\mu,\sigma$, then you wouldn't need to estimate (i.e. learn) them from data, which means you don't need to learn anything. Typically, we even don't know the true distribution and assume forms. The discussion follows similarly for the joint, $p(x,y)$.

Moreover, is it fair to say that a neural network or a deep learning model is learning the distribution of data? Thank you!

More or less any statistical method aims to do that, either explicitly or implicitly. Deep learning as one of them certainly tries to model the data distribution, especially if you look from Bayesian perspective.
