I am new to machine learning. I feel confused of how to understand the probability distribution of the training set, like $p(y|x)$ and $p(x)$, where $x$ is a training sample and $y$ is a label. Is there any concrete example to understand this? Further, it's said that the "ground truth" distribution is unknown and once you know it, there is nothing to learning. Could you help me clarify these words? Moreover, is it fair to say that a neural network or a deep learning model is learning the distribution of data? Thank you!
1 Answer
$\begingroup$
$\endgroup$
3
$p(y|x)$ represents the probability of a label given a sample, sometimes called the posterior, while $p(x)$ represents the probability (or density) of obtaining that specific sample. For example, in classification, we typically choose the class that maximize the posterior, i.e. $p(y|x)$. For example, $x$ can be a feature vector consisting of word counts in an e-mail and $y$ can be the label of it being spam or not.
it's said that the "ground truth" distribution is unknown and once you know it, there is nothing to learning
Let's say you've some data (e.g. think 1D), sampled from a Gaussian distribution. Even if you knew that the data is sampled from a Gaussian, you'd still need to estimate the distribution, i.e. its mean and variance and decide on some kind of estimates of true $\mu,\sigma$, i.e. $\hat{\mu},\hat{\sigma}$. Then, you can claim that the data distribution can be well approximated with $\mathcal{N}(\hat{\mu},\hat{\sigma})$, however it's not the true distribution. If you had known the true values $\mu,\sigma$, then you wouldn't need to estimate (i.e. learn) them from data, which means you don't need to learn anything. Typically, we even don't know the true distribution and assume forms. The discussion follows similarly for the joint, $p(x,y)$.
Moreover, is it fair to say that a neural network or a deep learning model is learning the distribution of data? Thank you!
More or less any statistical method aims to do that, either explicitly or implicitly. Deep learning as one of them certainly tries to model the data distribution, especially if you look from Bayesian perspective.
-
$\begingroup$ Thank you for your answer! For "Bayesian perspective", do you mean to maximize $p(y|x)$? For the first question, I think I need to make more clear of the basic assumption. When we talk about the probability, what is the sample space here and what are the events? For example, if we try to do object recognition for $N$ pictures of size $n\times n$ and $x$ represents a picture. Do all pictures form the sample space? Is $p(x)$ simply equals to $1/N$? $\endgroup$– luwCommented Oct 1, 2019 at 21:27
-
$\begingroup$ $x$ conceptually represents a picture, but it is a collection of pixels, just as in my spam example (collection of words = e-mail). Concretely, it is $x=[x_1,x_2,...,x_{n^2}]$ since there are $n^2$ pixels (RVs). $p(x)$ doesn't depend on how many training samples you have, i.e. $N$. It represents the whole population and in this case it is a density if pixels are continuous (if image is binary, then it is discrete but still it's not 1/N). Also, it doesn't have to be uniform, it's our prior belief about what the data could be, mostly it's assumed, not known and correspond to regularization. $\endgroup$– gunesCommented Oct 2, 2019 at 7:23
-
$\begingroup$ By "Bayesian perspective" I mean performing our analyses under a stochastic framework (possibly with prior assumptions), not just from an optimization perspective. For ex, in linear reg. ($y=X\beta +\epsilon$), the LS solution is $\hat{\beta}=(X^TX)^{-1}X^Ty$, and you could find the same answer via pure optimization, i.e. w/o diving into likelihoods etc. The same ans. can be found via maximum likelihood, i.e. assuming noise as normal RV, and therefore $y|x$ as normal RVs, and we maximize $\prod p(x|y)$. If priors are involved (i.e. we assume some $p(x)$), we maximize $\prod p(y|x)$. $\endgroup$– gunesCommented Oct 2, 2019 at 7:32