What the relation between a random variable and a sample (or dataset) in machine learning?

Question

I'm having trouble with the machine learning vocabulary, especially with the concept of random variables.

Given a sample $X$ (with features $x_1, x_2, \dots, x_n$) that you train your algorithm on (or predict), what is the random variable? Is it $X$? Or is it any of its feature $x_1, x_2, \dots, x_n$?

Quoting The Deep Learning book (by Ian Goodfellow):

A random variable is a variable that can take on different values randomly. We typically denote the random variable itself with a lowercase letter in plain typeface

Where as in the definition of a Random Variable in the Wikipedia article:

A random variable is a measurable function from a set of possible outcomes to a measurable space E

Then we also have the definition of multivariate random variable

A multivariate random variable is a column vector (or its transpose, which is a row vector) whose components are scalar-valued random variables on the same probability space as each other.

Should a sample be in fact considered as a multivariate random variable?

Answer 1

A dataset is a sample of the population. A dataset is not a random variable, which has a precise mathematical definition, i.e. it is a (measurable) function (you can ignore the "measurable" adjective!): you can simply think of a random variable as a map between outcomes of e.g. an experiment and real numbers.

A dataset (or a sample) contains $N$ "realisations" (or "outcomes") of one or more random variables, where $N$ is the size of the dataset. More precisely, each value associated with each feature of the dataset is (usually) or can be associated with one random variable: when you sample from the population (i.e., you get one row of the dataset), each of the random variables is "realized", that is, you obtain one of the concrete outcomes that the associated random variable can take.

A dataset can thus be considered the realization(s) of a random variable, if there is only one feature, or of multiple random variables (or, equivalently, of a multi-dimensional or multi-variate random variable), if there are multiple features.

In your example, if $X=(x_1, x_2, \dots, x_M)$ is one row of your dataset, then $X$ can be associated with the "realization" of $M$ random variables: $x_1, x_2, \dots, x_M$ are the realizations of these $M$ random varables (which we can denote by $X_1, X_2, \dots, X_M)$, that is, they are the outcomes of these random variables. You may have more than one row or, equivalently, maybe $X$ is actually a matrix, where $x_i$ is a column vector of size $N$ (i.e. the size of the dataset) which contains $N$ realizations of the random variable associated with $x_i$ (which can be denoted by $X_i$).

In conclusion, the term "feature" is (usually) equivalent to the term "random variable". A random variable (or feature) can have several realisations (or outcomes).

Answer 2

Considering the available dataset (sample), the features are random variables and the sample includes examples or instances of the problem. Each example or instance takes a value at every random variable (or feature or covariate). All these definitions are equal. The features are indeed random variables because we assume that their possible values are outcomes of a random phenomenon and they follow a specific distribution that we might do not know.

Answer 3

A random variable is a measurable function $\Omega\to X$ where $\Omega$ is the set of possible outcomes and $X$ is a measurable space. Simply put, a random variable gives a value to each outcome where an outcome might be a temperature reading, a win or loss, or the distance a runner can jump.

Example:

Consider a coin toss where the possible outcome is heads or tails. We can make a random variable with these outcomes such that heads $= 1$ and tails $= 0$. Assuming the coin is fair the probability of that $\mathbb{P}(X=1)=\frac{1}{2}$ and $\mathbb{P}(X=0)=\frac{1}{2}$.

Now what if we have something like temperature readings? In this case we can simply map each outcome (the temperature reading) to be that same value. Therefore if the outcome is 76 degrees Fahrenheit the value of the random variable is 76.

In the context of machine learning $X$ can be a random variable where the number of dimensions $k$ equals the number of features. An example of a $k$ dimensional random variable is a $k$ dimensional Gaussian distribution. More concretely when $k=2$ we have a bell "hump" instead of a bell curve, where the height of the hump is the probability of two features having values $(x_1, x_2)$. Relating this back to a real world example $x_1$ might be 76 for temperature and $x_2$ might be $0.8$ for humidity.

By the way if you are learning machine learning for the first time I suggest getting a good undergraduate text on probability to supplement your studies. Most machine learning techniques build on classical probability and statistics. Unfortunately I don't have a favorite. If you have studied measure theory I have some suggestions for more advanced texts.

Answer 4

From machinelearningmastery:

Given a table of data, such as in excel, each row represents a separate observation or event, and each column represents a separate random variable.