I'm confused about the matrix notation of the loss function in ordinary least squares regression. In matrix form, the expression for a linear model is:

$$\hat{y} = Xw$$

  • Where
    • $\hat{y} \rightarrow$ prediction vector for feature matrix $X$
    • $X \rightarrow$ feature matrix with each feature as a column and each row as an observation
    • $w \rightarrow$ weight vector

The above makes sense, but then I see the loss function written as

$$J(w) = \sum_{i=1}^n (y_i - x_i^Tw)^2 $$

which leads to the following questions:

  1. Are $x$ and $X$ used interchangeably here? Shouldn't we have written $X_i$ instead to show that we are indexing the feature matrix and not some other variable $x$?
  2. Does the indexing or transposition happen first in $x_i^T$ or does it not matter?
  3. From my reading, it seems an arbitrary choice whether $X_i$ indexes along the row or column dimensions, but I will assume it must be that we are returning a row from the matrix since we need to match the $w$ vector and have one value per feature. However, if we are getting a row from X, this means that we are then transposing a row vector when we do $x_i^T$ which would yield a column vector and the multiplication with the column vector $w$ would not work? Is there some magic conversion of $x_i$ to a column vector when it is yanked from $X$ or is it arbitrary if vectors are considered columns or rows when returned from a matrix?
  4. Is the loss function above the convention of writing it or are there other ways? For example, I would have thought the most intuitive way would be the following since it is in the same form and order as in the expression of the linear model and explicit about that a row vector is returned from $X$ when we index a row from it $$J(w) = \sum_{i=1}^n (y_i - X_iw)^2 $$
  • $\begingroup$ It's common for the elements of $X$ to be called $x$ and similarly for other capital/lowercase letters $\endgroup$ Commented Nov 30, 2023 at 21:04
  • 1
    $\begingroup$ You can work this out from the rules of matrix multiplication. In particular, because $y_i$ is a number, so must $x_i^\prime w$ be a number; and evidently from the formulation $\hat y = Xw,$ $w$ must be a column vector whose dimension is equal to the number of columns of $X$ -- the number of explanatory variables. Consequently $x_i$ must also be a column vector of the same dimension as $w.$ And that's strange, because the only candidate for $x_i$ would be row $i$ of $X$ -- but evidently that's what it must be. $\endgroup$
    – whuber
    Commented Nov 30, 2023 at 22:04
  • $\begingroup$ Thank you @whuber and ThomasLumley. If I understand correctly, writing $x_i$ actually means that I am indexing the matrix $X$ but we use lower case $x$ to indicate that a vector will be returned? This is somewhat confusing to me since we write $y_i$ in the same equation it means that we are indexing the vector $y$ and return a scalar (rather than indexing some matrix $Y$)? $\endgroup$ Commented Nov 30, 2023 at 22:19
  • 1
    $\begingroup$ Right: don't rely on case, bolding, or anything else: use the mathematical context to figure out what each object is. $\endgroup$
    – whuber
    Commented Nov 30, 2023 at 23:47

1 Answer 1

  1. Capital $X$ is the matrix of all feature values for all subjects, while lowercase $x$ is the vector of feature values for one subject. The lowercase $x$ is used in this notation to indicate that it is a vector in the product instead of a matrix, though $X_i^Tw$ is unlikely to cause serious confusion.

  2. Think of the $x_i^Tw$ as a dot product, $\langle x_i, w\rangle$. A reasonable way to write that as a matrix multiplication is $x_i^Tw$ (equal to $w^Tx_i$), so I suppose you take the $i$ and then transpose.

  3. There are multiple possible ways to write it. The common way to index the model matrix $X$ seems to be to have the columns indicate features and the rows indicate subjects or observations, but if you work with the transpose of that, you can be formal about the linear algebra and derive everything in this transposed world.

  4. The way you've written this deviates from the convention for $X_i$ to be a column vector, and then $X_iw$ does not make sense as a matrix multiplication (or take $w$ as a row vector and get an entire matrix as the output which doesn't work with the scalar $y_i$ value from which this product is subtracted).

To a large extent, I find it easiest to think of the loss function as $L(w) = \sum_i\left(y_i - \hat y_i\right)^2$ and then think of what the $\hat y_i$ are (determined by the $w$). If you want to be formal about it, you know how linear algebra works and can do it right unless you overthink it.

As a final point, while I do not see this done except in some special circumstances, it might be best to use a dot in the index of the capital $X$ matrix to show if the letter applies to the rows or the columns. For instance, $X_{i.}$ could denote that you mean to take the entirety of the $i^{th}$ row while $X_{.j}$ could denote that you mean to take the entirety of the $j^{th}$ column. Then a viable tweak to your final equation would be $J(w) = \sum_i\left(y_i - X_{i.}^Tw\right)^2$.

  • $\begingroup$ This is really helpful, thank you so much @Dave! There is only one part that's still unclear to me. You write that "...while lowercase ๐‘ฅ is the vector of feature values for one subject." If $x$ is already a vector representing a row in the matrix $X$, doesn't that mean that $x_i$ is a scalar? Then why do we need to transpose it with $x_i^T$ if it is already a scalar? And why do we multiply a scalar with the full vector $w$ rather than $w_i$? $\endgroup$ Commented Nov 30, 2023 at 22:06
  • $\begingroup$ Or do you mean that since $X_i$ returns a vector we can write that as $x_i$ to be explicit that the result is a vector although we're actually indexing the matrix? I find that confusing since when we write $y_i$ we're actually indexing a vector and it returns a scalar in each iteration. I like your final point, that would make things much more clear (to me at least) and I saw a similar suggestion in the answer here to use either start or dot math.stackexchange.com/questions/384153 $\endgroup$ Commented Nov 30, 2023 at 22:06
  • $\begingroup$ @another_student The $x_i$ means whatever we define it to mean. The convention is for $x_i$ to indicate the $i^{th}$ row of matrix $X$. $//$ Regarding why we don't multiply by a scalar $w_i$, an exercise might be to try that multiplication in the summation when you have a simple linear regression (just a slope and an intercept) with three observations. $//$ Again, you know how linear algebra works. Don't overthink it. The loss subtracts the fitted value. You know how to calculate the fitted value. There might be some mathematical "slang" going in sometimes, especially in less formal sources. $\endgroup$
    – Dave
    Commented Nov 30, 2023 at 22:26
  • 1
    $\begingroup$ "The convention is for ๐‘ฅ๐‘– to indicate the ๐‘–๐‘กโ„Ž row of matrix ๐‘‹" Perfect thank you, then everything makes sense! $\endgroup$ Commented Nov 30, 2023 at 22:35

Your Answer

By clicking โ€œPost Your Answerโ€, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.