I have worked with vectors and matrices but the following paragraph from The Elements of Statistical Learning by Trevor Hastie et al is little confusing (online edition, page 10)

Matrices are represented by bold uppercase letters; for example, a set of N input p-vectors $x_i$, i = 1, . . . , N would be represented by the $N$ x $p$ matrix X.

This means that the matrix X has $N$ rows and $p$ columns. Then the next lines confuse me, where I need your help.

In general, vectors will not be bold, except when they have N components; this convention distinguishes a $p$-vector of inputs $x_i$ for the $i^{th}$ observation from the $N$-vector $x_j$ consisting of all the observations on variable $X_j$. Since all vectors are assumed to be column vectors, the $i^{th}$ row of $X$ is $x^{T}_i$, the vector transpose of $x_i$.

What does this mean? Why is $i^{th}$ row of $X$ is $x^T_i$? I thought $i^{th}$ row of $X$ would have dimensions of $1$x$p$.

I would appreciate if someone can please clear my doubts.

  • 1
    $\begingroup$ I would like to direct your attention to the key phrase, "all vectors are assumed to be column vectors." Among other things, this tells you that "$x_i$"--which previously was named as a "$p$-vector," is a column, whence (written in matrix form) it would have dimensions $p\times 1$, not $1\times p$. $\endgroup$
    – whuber
    Dec 22, 2014 at 20:04
  • $\begingroup$ @whuber thanks for your reply. Am I right in understanding that the matrix $X$ has $N$ rows and $p$ columns? So if a $p$-vector is a column meaning it would be something like $[x_1..x_N]^T$. But then the $i$th row will have $p$ elements, isn't it? $\endgroup$
    – cps
    Dec 22, 2014 at 20:58
  • $\begingroup$ Your first quotation answers the first question. By definition, a $p$-vector $x$ has components $x_1, \ldots, x_p$. Thus the transpose of a $p$-vector is a row with $p$ numbers in it. $\endgroup$
    – whuber
    Dec 22, 2014 at 21:19
  • 2
    $\begingroup$ That text is fairly sophisticated: it does not assume readers are completely unfamiliar with the conventions of linear algebra or statistics. Search our site, for instance, on design matrix to see how widespread this matrix setup is. Although it does cover multiple regression in an early chapter, you would be very well served to study regression independently from another text, because then you will better appreciate ESL's point of departure and you will be more comfortable with its language and notation. $\endgroup$
    – whuber
    Dec 22, 2014 at 21:46
  • 1
    $\begingroup$ Funny I was confused by the same excerpt: stats.stackexchange.com/questions/224374/… $\endgroup$ Jul 18, 2016 at 18:33

2 Answers 2


enter image description here

Hope this image clears your doubt

  • 1
    $\begingroup$ Can you add some text explaining the image & how it answers the OP's question? $\endgroup$ Apr 1, 2015 at 12:30
  • $\begingroup$ The image represents a dataset where each column is a variable and each row is an observation. For example the 1st observation has a value x11 for the variable X1, value x21 for the variable X2 and so on.. This first set is represented by the vector x1 (lowercase x). The first column represents the value of variable X1 for all observations. It is represented by the vector X1 (uppercase X). To avoid confusion, column vectors are represented in uppercase and row vectors are represented in lowercase. Hope this clarifies. $\endgroup$ Apr 7, 2015 at 7:44

This is similar to the convention often used in econometrics. For instance, we often write OLS equation as $y=X\beta+\varepsilon$, which means column vectors $y$ and $\beta$. Vector $y$ is a column with each $i\in [1,N]$ elements corresponding to an observation. Vector $\beta$ is a column with each $k\in [1,p]$ corresponding to a parameter. Here, the matrix $X_{ij}$ has $p$ columns each containing $N$ observations of the variable $j$.

So, if you need to grab $i$-th observation of all explanatory variables as a vector, you get matrix transpose $(X_{i1},\dots,X_{ip})^T$ of $i$-th row of matrix $X$, because the vectors are to be columns.

  • $\begingroup$ In an expression like "$y=X\beta+\cdots$", there is no possibility of ambiguity (once you are told $X$ is an $n$ by $p$ matrix and that $y$ and $\beta$ are vectors, anyway): the very rules of matrix multiplication force $\beta$ to be a $p$-vector and $y$ to be an $n$-vector, both in column form. But no such expression was available to the O.P. Indeed, there would be no ambiguity whatsoever in writing $y=\beta X+\cdots$, only then both vectors would have to be rows (and $n$ would represent the number of parameters and $p$ the number of cases). Both conventions are widely used. $\endgroup$
    – whuber
    Dec 23, 2014 at 17:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.