3
$\begingroup$

Help interpreting this excerpt (annotations (numbers in parentheses) and bullets added) from The Elements of Statistical Learning

Matrices are represented by bold uppercase letters; for example, a set of $N$ input $p$-vectors (1) $x_i$, $i = 1, \ldots, N$ would be represented by the $N×p$ (2) matrix $X$. In general, vectors will not be bold, except when they have $N$ components; this convention distinguishes (3)

  1. a $p$-vector of inputs $x_i$ for the $i$th observation from...
  2. ...the $N$-vector $\mathbf{x}_j$ consisting of all the observations on variable $X_j$.

Since all vectors are assumed to be column vectors (4), the $i$th row of $X$ is $x^T_i$, the vector transpose of $x_i$ (5).

My questions:

  1. Is there significance/convention to "$p$" in "$p$-vector"?
  2. Is there significance/convention to "$N$" in "$N\times p$" matrix?
  3. I don't understand the distinction they are making...
  4. What is a "column vector"?
  5. In the $x_i^T$ part what operation is performed first; the "$T$" (transpose) or the "$i$" (accessing an element)?

My attempts at answers:

  1. "$p$-vector" means "vector of length $p$". Since input variables ($X$) can be vectors, "$p$-vector" used in this text to refer to that case; so "$p$" represents number of elements in an input vector.
  2. "$N$" is used in this text to refer to when input variables ($X$) is a matrix, specifically "$N$" is used to refer to the number of rows in such a matrix. Each "row" has $p$ elements. Hence we have an $N\times p$ matrix.
  3. Do I illustrate the difference here:

    enter image description here

  4. From Wikipedia

    In linear algebra, a column vector or column matrix is an $m × 1$ matrix, that is, a matrix consisting of a single column of $m$ elements,

  5. The language makes it clear "the vector transpose of $x_i$" means that the subscript is performed first, and then the vector transpose. Here, the transposing is probably used as described in the Wikipedia article above:

    To simplify writing column vectors in-line with other text, sometimes they are written as row vectors with the transpose operation applied to them.


This question addresses the same text as my question here, but since my question is a multipart question, and even on the subject of the "transpose" my question asks about order-of-operations, then I think my question is not completely duplicative. However, I will let moderators be the judge of that.

$\endgroup$
  • $\begingroup$ Possible duplicate of Concept of p-vector and matrix $\endgroup$ – The Red Pea Jul 18 '16 at 18:17
  • 1
    $\begingroup$ Do you think your own question is a duplicate? I'm not sure I follow. Why did you post it then? If you think this should be closed, you could just delete it. $\endgroup$ – gung Jul 18 '16 at 18:54
  • $\begingroup$ Thanks, I added a rationale why my question here is not a duplicate, but I accept any judgment from moderators. I should have searched better than I did, but I still ask other questions here which the suggested duplicate does not. $\endgroup$ – The Red Pea Jul 18 '16 at 19:00
  • 1
    $\begingroup$ If the potential duplicate does not cover all your questions, it isn't a duplicate & should stay open. You could delete those questions that have already been covered to your satisfaction. If something remains for those questions, you could elaborate what you learned from the other thread & clarify what you still need to know. (Note that I am not a moderator.) $\endgroup$ – gung Jul 18 '16 at 19:27
3
$\begingroup$

1 -2. Your reasoning is correct for 1 and 2, although this should be quite obvious from definition of matrix, and I am not sure what you are asking.

3-4. Your drawing I think indicates where you are getting confused. In math, vectors are typically "column" vectors that is if $\mathbf{x}$ is a vector (or $p$-vector) I mean

$$\mathbf{x} = \begin{pmatrix} x_1 \\ \vdots \\ x_p \end{pmatrix}$$

which is contrary to your illustration. The author goes through this definition because to highlight a row he uses the following notation.

$$ \mathbf{X} = \begin{pmatrix} x_{11} & x_{12} & \dots & x_{1p}\\ x_{21} & x_{22} & \dots & x_{2p}\\ \vdots & \vdots & \ddots & \vdots\\ x_{N1} & x_{N2} & \dots & x_{Np} \end{pmatrix} \qquad x_1 = \begin{pmatrix} x_{11} \\ x_{12} \\ \vdots \\ x_{1p}\end{pmatrix} \qquad x_1^\intercal = \begin{pmatrix} x_{11}& x_{12}& \dots & x_{1p}\end{pmatrix} \qquad \mathbf{x}_2 = \begin{pmatrix} x_{12} \\ x_{22} \\ \vdots \\ x_{N2}\end{pmatrix} $$

  1. The author defines $x_i$ to be the column vector composed of the $p$ elements in the $i$th row of matrix $\mathbf{X}$. $x_i^\intercal$ then is simply the transpose of the vector (see above).
$\endgroup$
  • $\begingroup$ Thanks. Since asking the question I reviewed this diagram, and realize the importance of "observation" (aka 'row'?) vs "variable" (aka 'column'?) I think I was wondering whether $p$ always represented the number of "observations" or the number of "variables". Can $p$ always be taken to represent one or the other? $\endgroup$ – The Red Pea Jul 18 '16 at 19:56
  • 1
    $\begingroup$ I don't understand your question. In this context, the rows of $\mathbf{X}$ refer to $N$ observations of $p$ variables. It is fairly common in statistics to use $p$ as the name for number of "variables" in the model or "predictors" (hence $p$). Is that what you are asking? $\endgroup$ – bdeonovic Jul 18 '16 at 19:58
  • $\begingroup$ Yes, that's exactly what I'm asking. "p" stands for "predictors". Gotta translate this stuff to Sesame-Street language for us newbs :) $\endgroup$ – The Red Pea Jul 18 '16 at 20:05
  • $\begingroup$ One last clarification; $x$ vs $\mathbf{x}$ -- what's the difference? Just that one is bold and the other is not? And that is the distinction the textbook is talking about? In other words, $x_i$, which is one observation for all variables/predictors, from $\mathbf{x}_j$ which is all observations for one variable/predictor? (I'll avoid using the words "row(s)" and "column(s)" because of the convention to always use column vectors whether we're talking about $x_i$ or $\mathbf{x}_j$) $\endgroup$ – The Red Pea Jul 18 '16 at 20:08
  • 1
    $\begingroup$ @TheRedPea yes, $i$ is conventional for indexing out of $N$ observations and $p$ is one of a few conventions for indicating the number of features each observation has $\endgroup$ – shadowtalker Jul 18 '16 at 21:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.