# Basic Linear Regression problem

I am a total beginner in Machine Learning but I would say that I have kind of strong mathematical background. I have started learning Machine Learning algorithims and I have the following question. Thank you for your help in advance. As we know, our basic linear regression algorithm is $\hat{\beta} = (X^T X)^{-1} X^T Y.$ I have tried this algorithim for $y = 2x + 5$ For example, (the first column is ones vector and imagine that we give $1, 2, 3, 4$ numbers each for $x$ and get $7, 9, 11, 13$ for $y$ values)

x = [[1, 1]; [1, 2]; [1, 3]; [1, 4]] y = [[7]; [9]; [11]; [13]]


When I solve the algorithm, it gives $2$ and $5$ which are the parameters of $y=2x + 5$. There is no problem till here. My question starts when I use this algorithm for, lets say $y=X_1 + X_2 + 5$. When I use this algorithm for this equation, I cannot get $1, 1, 5$ parameters as a solution. The inputs I use as following: (I give $1, 2, 3, 4$ for $X_1$ and $2, 3, 4, 5$ for $X_2$ and get $8, 10, 12, 14$ for $Y$) (Again, the first column is ones vector)

X = [[1, 1, 2]; [1, 2 , 3]; [1, 3 , 4]; [1, 4 , 5]]
y = [[8]; [10]; [12]; [14]]

• Are you sure if the inverses are computed correctly? Because the rank of X*X' of your example is not 4, it is 2. This will lead to inaccuracies while inverting the matrix. And therefore the results may not be exactly what you expect. This also means that the problem has to be defined differently if you want to get a unique solution. – lekshmi dharmarajan Sep 1 '16 at 11:34
• The answers to stats.stackexchange.com/questions/69442/… are relevant – Adrian Sep 1 '16 at 12:26