What does it mean to take expectation with respect to data set?

Question

Let $X$ be a random variable with some distribution $f_X(x)$, and assume that we generate $X_1, \ldots, X_N$ i.i.d. samples from this distribution. For simplicity, assume $X_n \in \mathbb{R}$.

Then it is clear from elementary probability theory:

$$\mathbb{E}(X) = \int\limits_{-\infty}^\infty x f_X(x) dx$$

If we have a function $g$, then

$$\mathbb{E}(g(X)) = \int\limits_{-\infty}^\infty g(x) f_X(x) dx$$

Suppose now we associate each sample $X_i$ with a label $y_i \in \{-1,+1\}$ and form the set:

$$\mathcal{D} = \{(X_1, y_1), \ldots, (X_N, y_N)\}$$

which we can refer to as a "data set".

My question is, what is

$$\mathbb{E}(\mathcal{D})?$$

I have seen multiple references that takes expectation with respect to the data set, however, I have never seen a closed form expression of $\mathbb{E}(\mathcal{D})$

Is $\mathcal{D}$ a random variable? A set of scalars? How do we make sense of $\mathbb{E}(\mathcal{D})$?

Answer 1

$D$ can be interpreted as a random vector. If $X$ lives in $\mathcal{X}$ and $Y$ lives in $\mathcal{Y}$ then $(X,Y)$ is a random vector that lives in $\mathcal{X}\times \mathcal{Y}$. Let $Z= \mathcal{X}\times \mathcal{Y}$. Then $D = (z_1,\ldots, z_n) \in Z\times \cdots \times Z = Z^n$ where $z_i = (x_i,y_i)$. If $P = P(x,y)$ is the joint distribution of $X,Y$ and the data points $z_i$ are iid then $D$ can be said to be a random vector in $Z^n$ with distribution $P^n$. You can then interpret $E(D)$ accordingly.

Answer 2

$\mathcal D$ is not a random variable, so the expectation is not well defined.

EDIT: In the paper you're referring to they talk about an expectation of a loss function, which is a function of two random variables and thus a random variable itself.