Timeline for Why does this multiple linear regression fail to recover the true coefficients?
Current License: CC BY-SA 4.0
15 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| May 23, 2023 at 10:01 | vote | accept | teeeeee | ||
| May 23, 2023 at 9:17 | history | edited | teeeeee | CC BY-SA 4.0 |
added 206 characters in body
|
| May 23, 2023 at 9:02 | history | edited | teeeeee | CC BY-SA 4.0 |
added 624 characters in body
|
| May 23, 2023 at 7:52 | answer | added | Sextus Empiricus | timeline score: 2 | |
| May 22, 2023 at 23:02 | comment | added | Sycorax♦ | The columns of $A$ are linearly dependent, as I stated in my first comment, so they are not a basis. The polynomial basis is a basis because they are linearly independent (cf Vandermonde matrix). Or pick any of the other 22 properties from the invertible matrix theorem. | |
| May 22, 2023 at 22:53 | comment | added | teeeeee | @Sycorax okay I see that there is not a unique solution, and how to recognise it. What I dont understand is what it is about my test model which makes it this way. For example, if I change x2data to x1data^2 then it works. I.e if I perform a polynomial regression instead of multiple variable regression. | |
| May 22, 2023 at 22:51 | review | Close votes | |||
| May 24, 2023 at 15:03 | |||||
| May 22, 2023 at 22:36 | comment | added | Sycorax♦ | There are infinitely many solutions to this system of linear equations. en.wikipedia.org/wiki/System_of_linear_equations#Solution_set The reason is that $A$ is not full rank. en.wikipedia.org/wiki/Rank_(linear_algebra) The $QR$ decomposition is rank-revealing; the rank of $A$ is the number of pivots in $R$. Because there are not 3 pivots in $R$, you know there is not a unique solution, so there is no such thing as "true" coefficients for this problem. | |
| May 22, 2023 at 22:35 | comment | added | jbowman |
Try printing out A and Y, confirming that everything is as it ought to be entering the fitting steps.
|
|
| May 22, 2023 at 22:32 | comment | added | teeeeee | @whuber I like your example. I guess you are tactfully trying to tell me that there are multiple solutions to my setup. I thought that since I generated the y data exactly using the model equation that I should be able to recover the coefficients exactly? Is it not the case? Am I missing something fundamental. Thanks | |
| May 22, 2023 at 22:31 | comment | added | teeeeee | @Sycorax Sure. I have edited the question to try reflect your issues raised. Essentially, all 3 methods are solving Eq. (3), just with slightly different numerical approaches. I did not include an error term, because I wanted to generate some test data, so it should be easy to obtain the fit coefficients exactly (so I thought). Also, Eq.(3) does not contain an error term (even though it was present in Eq, (1) from which it was derived. | |
| May 22, 2023 at 22:28 | history | edited | teeeeee | CC BY-SA 4.0 |
added 505 characters in body
|
| May 22, 2023 at 18:03 | comment | added | whuber♦ | A minimal reproducible example of this phenomenon, which will therefore be helpful to contemplate, is $x_1=(1),$ $y=(0),$ with the model $E[y] = \alpha + \beta x_1.$ Consultant (A) tells you the solution is $\hat\alpha=\hat\beta=0$ while consultant (B) tells you no, it's $\hat\alpha=1,$ $\hat\beta=-1.$ Who is correct, if either? | |
| May 22, 2023 at 15:56 | comment | added | Sycorax♦ |
Your code is labeled with three methods to estimate coefficients, but you don't tell us what the outputs are. You are more likely to get useful answers to this question if you can articulate your process & results in a way that does not depend on users reading, understanding, and running your code. // If I understand the code correctly (I don't use MATLAB), you have a design matrix with linearly dependent columns and y_data has no error term. So ... yeah, there are multiple solutions to the linear system.
|
|
| May 22, 2023 at 15:15 | history | asked | teeeeee | CC BY-SA 4.0 |