0
$\begingroup$

Most books have the notation of a weight vector w and input matrix x: $$ w = \begin{bmatrix} w_1\\...\\ w_D \end{bmatrix}, x = \begin{bmatrix} x_{11}&...&x_{1D}\\ ...&...&...\\ x_{N1}&...&x_{ND} \end{bmatrix} $$ For N samples and D features/parameters. Then it goes on to say the net input, or y prediction, or whatever the book decides to call it, is $$ y=w^Tx $$ But doesn't that mean every sample of the 1st feature is multiplied with weights $w_1, w_2,...w_D$? Intuition tells me it should be each $d$-th feature should be multiplied by the correspnding $d$'th weight, done over all samples. By this reasoning it should be more like $y=xw$, which I've definitely never seen in any of the books. What am I getting wrong?

PS I realized I missed the bias; hopefully the argument still stands.

Improve this question
$\endgroup$
1
  • $\begingroup$ As written, the dimensions aren't compatible for multiplication: $w^T$ is $1 \times D$ and $x$ is $N\times D$, so the product $w^T x$ isn't defined unless $N = D$. Writing $y = x w$ is in this context is correct, and it written that way in many places $\endgroup$
    – Artem Mavrin
    Commented Feb 29, 2020 at 21:23

1 Answer 1

Reset to default
0
$\begingroup$

The formula your "most books" give cannot be right because, in general, $N \neq D$. Consequently, you cannot multiply a row vector of length $D$ with a matrix containing $N$ rows. Thats simple linear algebra.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Using this as an example (p.11, or 9/30 slide number), is the prof incorrect? cs.toronto.edu/~rgrosse/courses/csc321_2018/slides/lec02.pdf $\endgroup$
    – Five9
    Commented Feb 29, 2020 at 21:25
  • $\begingroup$ I just realized later in slide 11/30 (p. 14) it is written Xw... I dont even go to the school so I'm not sure if the prof corrected himself mid-lecture. I guess since my confusion was justified, then it's ok I understand now... $\endgroup$
    – Five9
    Commented Feb 29, 2020 at 21:28
  • $\begingroup$ I see no proof on slide 9/30. But I admit, the notation may be misleading, because it mixes vectors and one-dimensional matrices. You can consider both $w^T$ and $x$ to be vectors, so $w^T x$ is simply a dot product. If you understood them as one-dimensional matrices ($x$ a single-row matrix, $w$ a single-column matrix), you'd need to write $x w$. On the slide 11/30, where multiple observations are collected as rows in the matrix $X$ (capital X!), this is actually the notation used: $X w$. $\endgroup$
    – Igor F.
    Commented Feb 29, 2020 at 21:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.