the kernel trick and linear regression This question is inspired by this and the answer by Hamed.
Let us say the underlying data generation process is:
 y = 3x + sin(x)

So pretty non linear. I understand that the linear regression only has to be linear in parameters. So by transforming the original IV x into 2 new IVs x1 and x2:
x1=x and x2=sin(x)

one can estimate the parameters using this linear regression formula:
y = alpha * x1 + beta * x2

or in R speak:
lm(formula = y ~ x1 + x2 - 1)

is this correct and is this the kernel trick?
 A: Is this correct? Yes, except that it's not the kernel trick. There are two equivalent approaches to a regression problem. One is to write the response variable $y$ as a linear combination of basis functions $\phi(x)$ as $y(x) = \sum_i \omega_i \phi_i(x)$, $i=1,2$ (2 in this case, you could have infinite). This is what you are doing, and what you are calling new input variables are the basis functions evaluated at the old variables, $\phi_1(x) = x$ and $\phi_2(x) = \sin(x)$. The weights in that linear combination are what you look for.
The kernel trick is to write the regression problem as $y(x) = \sum_n y_n K(x,x_n)$, $n=1,2,...,N$, where  $N$ is the number of data points and where the kernel function $K$ can be written in terms of $\phi_1$ and $\phi_2$. An alternative is to directly propose a kernel with suitable properties and forget about basis functions $\phi$. One must take into account that training a model based on the kernel approach involves estimating $N$ parameters, probably much more than the 2 you have in the basis function approach (what's the benefit then? is that you don't need to assume any functional form such as $x$ and $\sin(x)$, so the approach is more flexible. Also, while slow for training, is fast for predicting sot that's no problem.)
This (free) book https://www.microsoft.com/en-us/research/uploads/prod/2006/01/Bishop-Pattern-Recognition-and-Machine-Learning-2006.pdf talks about this in great detail (but is not very straightforward to dive in).
