Let $X$ be a random variable with some distribution $f_X(x)$, and assume that we generate $X_1, \ldots, X_N$ i.i.d. samples from this distribution. For simplicity, assume $X_n \in \mathbb{R}$.

Then it is clear from elementary probability theory:

$$\mathbb{E}(X) = \int\limits_{-\infty}^\infty x f_X(x) dx$$

If we have a function $g$, then

$$\mathbb{E}(g(X)) = \int\limits_{-\infty}^\infty g(x) f_X(x) dx$$

Suppose now we associate each sample $X_i$ with a label $y_i \in \{-1,+1\}$ and form the set:

$$\mathcal{D} = \{(X_1, y_1), \ldots, (X_N, y_N)\}$$

which we can refer to as a "data set".

My question is, what is


I have seen multiple references that takes expectation with respect to the data set, however, I have never seen a closed form expression of $\mathbb{E}(\mathcal{D})$

Is $\mathcal{D}$ a random variable? A set of scalars? How do we make sense of $\mathbb{E}(\mathcal{D})$?


2 Answers 2


$D$ can be interpreted as a random vector. If $X$ lives in $\mathcal{X}$ and $Y$ lives in $\mathcal{Y}$ then $(X,Y)$ is a random vector that lives in $\mathcal{X}\times \mathcal{Y}$. Let $Z= \mathcal{X}\times \mathcal{Y}$. Then $D = (z_1,\ldots, z_n) \in Z\times \cdots \times Z = Z^n$ where $z_i = (x_i,y_i)$. If $P = P(x,y)$ is the joint distribution of $X,Y$ and the data points $z_i$ are iid then $D$ can be said to be a random vector in $Z^n$ with distribution $P^n$. You can then interpret $E(D)$ accordingly.

  • 1
    $\begingroup$ Calling this the expectation of the data is wrong if data refers to a specific sample for then it makes no sense. Expectation is defined for random variables or random vectors and not for an observed sample. $\endgroup$ Dec 27, 2017 at 5:56
  • 1
    $\begingroup$ The point is that the data set obtained is one of many possible data sets. The likelihood that I obtain the data set $D = \{(x_1,y_1),\ldots, (x_n,y_n)\}$ is exactly $\Pi_{i=1}^n P(x_i,y_i)$ (assuming the pairs are iid). In that regard there's nothing that precludes us from talking about the expectation of $D$ as a random vector. If you're talking about a particular data set then we'd be taking the expectation of a constant random vector which would be the vector itself. $\endgroup$
    – user189711
    Dec 27, 2017 at 6:06

$\mathcal D$ is not a random variable, so the expectation is not well defined.

EDIT: In the paper you're referring to they talk about an expectation of a loss function, which is a function of two random variables and thus a random variable itself.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.