5
$\begingroup$

I've been self-studying Introduction to Statistical Learning. From page 16 of the book:

"...suppose that we observe a quantitative response $Y$ and $p$ different predictors, $X_1$, $X_2$, $\ldots$, $X_p$. We assume that there is some relationship between $Y$ and $X = (X_1, X_2, \ldots, X_p)$, which can be written in the very general form $Y = f(X) + \epsilon$

Later on, we analyze the approximation function

$$ \hat{Y} = \hat{f}(X). $$

Question 1: What is the domain of $Y$ and $\hat{Y}$? Is it tuples of observations or tuples filled with values from the domains of the $X_1$, $\ldots$, $X_p$?

Question 2: Throughout this text, one will see references to the notation like $\hat{f}(x_i)$. For example:

$$ MSE = \frac{1}{n} \sum_{i=1}^n (y_i - \hat{f}(x_i))^2 $$

Isn't the domain of $\hat{f}$ random variables (not tuples of observations $x_i$)?. If so, isn't this an abuse of notation? Shouldn't we instead (technically) write something like

$$ (f(X_1, \ldots, X_p))(x_i)? $$

$\endgroup$
4
  • $\begingroup$ The word "domain" could mean distinct things in this context, ranging from mathematical (the domain of a function) to statistical (the range of a random variable) to computer science. In particular, it is difficult to discern the way in which $Y$ and $\hat Y$ could have "domains" in some shared sense. For us to understand your question, I think you will need to supply a clear definition of the sense in which you are using this word. $\endgroup$
    – whuber
    Apr 18, 2016 at 20:08
  • $\begingroup$ I didn't realize the term was so varied. I meant it in the mathematical sense: if $f:A \rightarrow B$, then $A$ is the domain (at least how I'm using the term). $\endgroup$
    – George
    Apr 18, 2016 at 20:09
  • $\begingroup$ Thank you; that helps. But then isn't the domain of $\hat f$ exceptionally clear from the notation? It is explicitly the set of whatever kind of thing "$x_i$" might be (most likely an element of $\mathbb{R}^p$). This suggests you aren't exactly thinking of the domain according to the usual mathematical definition. $\endgroup$
    – whuber
    Apr 18, 2016 at 20:13
  • $\begingroup$ @whuber: The notation is being used in conflicting ways. Sometimes I'm seeing $f(X_1, \ldots, X_p)$, and sometimes I'm seeing $\hat{f}(x_i)$ (even though $f$ and $\hat{f}$ are taken to have the same domains, whatever they are). I think, if I had to guess, that the second sort of usage is just loose notation that connotes $f(X_1, \ldots, X_p)(x_i)$. But I wanted to get some second opinions from this community if possible. $\endgroup$
    – George
    Apr 18, 2016 at 20:17

1 Answer 1

7
$\begingroup$

Often orthography hints at meaning. With this as a point of departure, along with an understanding of what a random variable is, we can figure out the rest. Rather than just giving an interpretation, I will first take you through the reasoning used to parse these expressions: that might help you make sense of related material elsewhere in the book.


Random variables

We might take the capitalization of $X_1, \ldots, X_n$ and $Y$ to be hints that they are intended to denote random variables. Recall that abstractly this means there is a probability space on some (abstract, typically unnamed) underlying set $\Omega$ and that each of these quantities is a measurable real-valued function with domain $\Omega$. We can write this with arrows in the form

$$X_1, \ldots, X_n, Y: \Omega \to \mathbb{R}.$$

$\Omega$ is a mathematical representation of the real-world objects in which we are interested. These objects typically are not themselves mathematical and have no numbers intrinsically associated with them. They could be people, bugs, stars, economies, production processes, or whatever. The $X_i$ and $Y$ represent numerical properties of these objects.


Vectors

The random variable $X$ is vector-valued. Formally,

$$X:\Omega\to \mathbb{R}^p$$

is defined in terms of the $p$ canonical projection functions

$$\eqalign{ \pi_i:\mathbb{R}^p \to \mathbb{R} \\ \pi_i(x_1, \ldots, x_{i-1}, x_i, x_{i+1}, \ldots, x_p) = x_i }$$

as being the unique function for which $\pi_i \circ X = X_i$. With arrows we may depict it thus:

$$X_i: \Omega \xrightarrow{X} \mathbb{R}^p \xrightarrow{\pi_i} \mathbb{R}.$$

$X$ conveniently gathers the $p$ predictors $X_i$ into a single mathematical object.


Responses, model functions, and errors

In the expression

$$Y = f(X) + \epsilon,$$

we know that the value of $Y$ is a real number, whence both $f(X)$ and $\epsilon$ are real numbers. The only reasonable meaning is that $f:\mathbb{R}^p \to \mathbb{R}$, thus:

$$f:\mathbb{R}^p \to \mathbb{R.}$$

The model function $f$ simply is a function of $p$ real numbers. If you give me the values of $p$ predictors $X_i(\omega)$, $i=1, \ldots, p$, then $f$ will give me some guess about the corresponding value of $Y(\omega)$

The quantity $\epsilon$ is unlikely just to be a fixed number: it represents the difference between $Y$ and what $f$ predicts from $X$. Therefore we should understand that

$$\epsilon:\Omega \to \mathbb{R}$$

is a random variable (the "error") and the "$+$" represents pointwise addition of functions in the usual way:

$$Y(\omega) = (f(X) + \epsilon)(\omega) = f(X(\omega)) + \epsilon(\omega)$$

for all $\omega\in\Omega$. This is the usual way to add the random variables $f\circ X$ and $\epsilon$.

If you want to be formal about it you will collect $f$ and $\epsilon$ into an ordered pair of functions and identify that with a function into the Cartesian product of their ranges, giving this picture of $Y$:

$$Y: \Omega \xrightarrow{(X, \epsilon)} \mathbb{R}^p \times \mathbb{R} \xrightarrow{(f, \operatorname{id})} \mathbb{R}\times\mathbb{R}\xrightarrow{+}\mathbb{R}.\tag{1}$$

"$\operatorname{id}$" denotes the identity function on $\mathbb{R}$, $x\to x$ for all numbers $x$.


Predictors

Since $\hat f$ is an "approximation function," we should try to understand it as being some version of $f$: it's a guess concerning what $f$ is. Consequently $\hat f$ also must be a function of $p$ variables. Let's draw the arrows for $\hat{Y}=\hat f(X)$:

$$\hat{Y}: \Omega \xrightarrow{X} \mathbb{R}^p \xrightarrow{\hat f} \mathbb{R}.\tag{2}$$

This clearly shows that $\hat Y = \hat{f}\circ X$ is the same kind of object as $Y$: namely, a random variable.


Answers

Answer 1

The arrows in $(1)$ and $(2)$ show that both $Y$ and $\hat Y$ are random variables: their domains are $\Omega$.

Answer 2

When an "observation" of $X$ is drawn, it gives us a "realization" $x_1$. Let's do this $n \ge 0$ times (often independently, but not necessarily), producing a sample $x_1, x_2, \ldots, x_n$. Each of the $x_i$ is a $p$-vector. The expressions $f(x_i)$ and $\hat f(x_i)$ require no additional parsing: by definition, $f$ and $\hat f$ are functions whose domains are $\mathbb{R}^p$. Here, they have been applied to a specific element $x_i \in \mathbb{R}^p$.

Answer 3

The expression "$f(X_1, \ldots, X_p)$" does make sense: it represents $f\circ X: \Omega\to\mathbb{R}$. As the arrow indicates, it is applied to elements $\omega\in\Omega$, not to individual vectors $x_i$. Although we could understand "$(f(X_1, \ldots, X_p))(\omega)$" in this way, "$(f(X_1, \ldots, X_p))(x_i)$" would be nonsense unless we stipulated that $\Omega=\mathbb{R}^p$. Although sometimes this is done, and many textbooks (especially of the most elementary sort) can be read in this way, conceptually it is clearer not to confuse objects in $\Omega$ with the $p$-tuples of numbers used to characterize some of their properties. Far from being "loose," this is fairly rigorous notation.

$\endgroup$
1
  • 1
    $\begingroup$ I wish I should star answers. This was incredibly informative and clear. $\endgroup$
    – George
    Apr 22, 2016 at 13:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.