# General question regarding probability distribution in machine learning

This question is in similar spirit to Probability distribution in data mining

I am going through some material online, and the source says that given a set of training data and labels $\{(x_n,y_n)\}$, $n = 1, \ldots, N$, we make the assumption that:

$x_n$ is generated by some "distribution" $P(x)$, or written as $x \sim P(x)$

1. How is the probability distribution related to cumulative density function, probability mass function or probability density function?

2. It is assumed that such probability distribution is unknown. But can we have an idea as to how complex or simple such probability distribution could be?

3. What is the use/utility of such unknown probability distribution? For example, at any moment we can measure a number of features about our body, say temperature, heart rate, rate of breathing, etc. Let $x_n$ be a vector that captures all these variables, then it could possibly be generated by some extremely high dimensional probability distribution (joint probability, mixed random variables, non-independent, non-zero correlation, covariance, mean, variance, etc.). We couldn't model such a probability distribution it even if we tried, so what benefit does it bring to us by making such an assumption?

4. Would it be correct to say that training data $x_n$ are equivalent to random variable evaluated at some outcome of an underlying event space? If so, what is the event space?

• Could you put a link to that source ?
– user83346
Commented Sep 27, 2017 at 6:59

On 4. Yes, it is correct that this is the intention. The event space is $\Omega$ and can be imagined as the collection of states ('all information the universe has to offer') the universe could get to. Let us make this very concrete:

Let us say that we have a small universe. It has three inhabitants: Adam, Barney and Charley. Let us say that in our universe, we only have the time unit 'day' and our universe only lives for $2$ consecutive days and has only two pieces of information to offer: the weather (either 'S' = sunny or 'R' = rainy) and the general mood (either 'H' = happy or 'U' = unhappy). Then each $\omega$ will encode all the information for all the inhabitants at every day. For example: one very concrete $\omega$ could be

$$\omega = (S, H, yes, S, H, yes, S, H, yes, R, U, yes, R, H, yes, R, U, yes)$$

meaning that the universe has gotten into a state where everybody played golf all the time (yes everywhere), the weather on the first day is sunny and all inhabitants were happy and on the second day the weather was rainy and Adam and Charly were happy but Barney was unhappy.

Let us say that we have observed our universe for $2$ days and on the first day, the weather was sunny and on the second day, the weather was rainy. We observed that on the first day, Adam and Barney played golf and on the second day, Adam played Golf but Barney did not. However, we do never have access to something weird as the 'general mood' not even of a single person. So we do know that the $\omega$ that produced the state that we actually observed looks like this:

$$\omega = (S,*,yes, S,*,yes, S,*,*, R,*,yes, S,*,no,R,*,*)$$

and we cannot fill in the values for '*': we do never know about the mood and we never know anything about Charly.

In the language of a data table our table looks like this:

weather | inhabitant | play golf?
sunny   | Barney     | yes
sunny   | Barney     | no


Now $X_1$ is the projection to the first coordinate (weather on the first day for where Adam lives). $X_2$ is the projection to the $4$-th coordinate (weather on the first day where Barney lives). $X_3$ is the projection to the $10$-th coordinate (weather for the second day where Adam lives) and so on. The $Y_i$ are analogously the projections on the $3$-rd, $6$-th, $12$-th and $15$-th coordinate. In Machine Learning we only have a chance to do something if each $Y_i$ is not only the projection but rather is linked to the features. For example: the 'real rule' or link could be

## play golf iff. the weather is good or the mood is good [except for Adam: he plays golf iff. the weather is good independent of the mood]

i.e. the assertion whether or not an inhabitant is about to play golf only depends on the features the universe has to offer. This however, is very natural: if it did not depend on those then our universe would be slightly bigger and we would have restarted that after including this information into $\omega$ as well. The goal of machine learning is to uncover this rule, sometimes written as $Y = f(X)$ and meaning precisely that $Y(\omega) = f(X(\omega))$ where $X$ is the collection of all features (observed and unobserved).

By the data table observed above we can learn some rule like

## play golf iff. the weather is good

(excluding single cases probably leads to overfitting, i.e. a rule that is too complicated to believe). Now here are two sources of errors: Some single cases we will never predict accurately because we do not want to include them into our rule (overfitting). The other case is that the rule would be simple and natural but it depends on features that we will never be able to observer (like the mood).

In short: Yes, $x_i = X_i(\omega)$ but the space $\Omega$ is too big to study and $\omega$ consists mostly of unobserved features. Both are hopeless to study, hence, we ignore these players mostly and consider the things we can see: the behaviour of $X$ and the Relation of $Y$ to $X$ on the target space of $X$, mostly $\mathbb{R}^n$. This is the reason why you will only see distributions and so on in the space $X$ maps to and only rarely do stuff on the underlying event space $\Omega$ directly.

1. $P$ is sometimes used to denote probability density function. Saying that something has some distribution is just saying that this random variable has density $P$
2. There are different types of 'unknown'. You can assume that the form of distribution is known (parametric statistics) or it's unknown at all (nonparametric statistics). In machine learning linear regression is an example of parametric method, and kNN and trees are examples of nonparametric methods.
3. If you assume that this distribution is in some family of distributions indexed by some parameter $\theta$ (for example normal distribution - $\mathcal{N}(\theta, 1)$ you can ask questions: what is $\theta$? How confident can you be about this estimate? That's an example of parametric statistics. If you don't have any clue about the distribution you can do density estimation - example of nonparametric methods.
4. What do you mean 'evaluated'? Random variable is something that is assumed to be possible to sample from, the 'distribution' part saying that when you sample, let's say from discrete distribution, then probability of an outcome $a$ is $P(a)$.