0
$\begingroup$

Most books have the notation of a weight vector w and input matrix x: $$ w = \begin{bmatrix} w_1\\...\\ w_D \end{bmatrix}, x = \begin{bmatrix} x_{11}&...&x_{1D}\\ ...&...&...\\ x_{N1}&...&x_{ND} \end{bmatrix} $$ For N samples and D features/parameters. Then it goes on to say the net input, or y prediction, or whatever the book decides to call it, is $$ y=w^Tx $$ But doesn't that mean every sample of the 1st feature is multiplied with weights $w_1, w_2,...w_D$? Intuition tells me it should be each $d$-th feature should be multiplied by the correspnding $d$'th weight, done over all samples. By this reasoning it should be more like $y=xw$, which I've definitely never seen in any of the books. What am I getting wrong?

PS I realized I missed the bias; hopefully the argument still stands.

$\endgroup$
  • $\begingroup$ As written, the dimensions aren't compatible for multiplication: $w^T$ is $1 \times D$ and $x$ is $N\times D$, so the product $w^T x$ isn't defined unless $N = D$. Writing $y = x w$ is in this context is correct, and it written that way in many places $\endgroup$ – Artem Mavrin Feb 29 at 21:23
0
$\begingroup$

The formula your "most books" give cannot be right because, in general, $N \neq D$. Consequently, you cannot multiply a row vector of length $D$ with a matrix containing $N$ rows. Thats simple linear algebra.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Using this as an example (p.11, or 9/30 slide number), is the prof incorrect? cs.toronto.edu/~rgrosse/courses/csc321_2018/slides/lec02.pdf $\endgroup$ – Five9 Feb 29 at 21:25
  • $\begingroup$ I just realized later in slide 11/30 (p. 14) it is written Xw... I dont even go to the school so I'm not sure if the prof corrected himself mid-lecture. I guess since my confusion was justified, then it's ok I understand now... $\endgroup$ – Five9 Feb 29 at 21:28
  • $\begingroup$ I see no proof on slide 9/30. But I admit, the notation may be misleading, because it mixes vectors and one-dimensional matrices. You can consider both $w^T$ and $x$ to be vectors, so $w^T x$ is simply a dot product. If you understood them as one-dimensional matrices ($x$ a single-row matrix, $w$ a single-column matrix), you'd need to write $x w$. On the slide 11/30, where multiple observations are collected as rows in the matrix $X$ (capital X!), this is actually the notation used: $X w$. $\endgroup$ – Igor F. Feb 29 at 21:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.