1
$\begingroup$

We are observing a linear model which is supposed to predict a target $t$ of some new value $x$ given some data $D=\{(x_1,t_1),\ldots,(x_N,t_N)\}$. This model is defined by $$y(\mathbf{x}, \mathbf{w}) = \sum_{j=1}^{M-1} w_j\phi_j(\mathbf{x}) = \mathbf{w}^T\phi(\mathbf{x})$$ where $\mathbf{w}$ denotes weighting vector to be determined and $\phi(\mathbf{x})$ denotes the design matrix.

Question 1:

What is it the basis functions in $\phi_j$ really do?

If I understand correctly, really what it is does is transform the data into a new form where it can be approximated by a linear function. I.e. we linearize the data? Is this why, even though the underlying data may not be close to being linear, we call the model a linear model?

Question 2:

How can I picture the design matrix?

As far as I understand, its just a vertical stack of our (vector of) basis functions. The component wise operation makes intuitive sense, but it's hard for me to grasp the matrix notation.

I'm mostly learning with bishops PRML book, so I used his notation for it.

$\endgroup$
2
1
$\begingroup$
  1. When the basis functions are pre-computed, you're only estimating $w$, which is a linear problem. In other words, given a fixed choice of $\phi$ and fixed data $x$, you're minimizing some loss $\min_w \mathcal{L}(t, w^\top \phi(x) )$. If you're finding it confusing to think about $\phi(x)$, we can just denote our design matrix as $\phi(x)=Z$. In the case of a least-squares loss function, we have the usual linear model $\min_w \| t-w^\top Z \|_2^2$.

  2. A design matrix is a matrix just like any other matrix: a rectangular array indexed by rows and columns.

$\endgroup$
1
  • $\begingroup$ Thank you! And is it true to say that we try to transform the data into a somewhat linear form to approximate it? If not, how do I build a bridge between some underlying data und choosing good basis functions? $\endgroup$ – Marcel Braasch Mar 12 '20 at 20:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.