Return to Question

An example to show that it works when fitting a cubic function with no noise (i.e. with deterministic data, as mentioned by @Gregg H):

beta0_true = 0.3;
beta1_true = 0.6;
beta2_true = -3;
beta3_true = -2;

x_data = linspace(-3,2,100).';
epsilon =  0;       % or use epsilon = randn(length(x_data),1) to test noise

y_data = beta0_true + beta1_true*x_data + beta2_true*x_data.^2 + beta3_true*x_data.^3 + epsilon;

A = [ ones(length(x_data),1)   x_data   x_data.^2   x_data.^3 ];
Y = y_data;

beta_fit = A \ Y;

beta_fit = [0.3 0.6 -3 2]

An example to show that it works when fitting a 2D parabolic function with no noise (i.e. with deterministic data, as mentioned by @Gregg H):

x1_data = linspace(-1,1,20).';      % First predictor variable
x2_data = linspace(-0.3,0.6,20).';  % Second predictor variable

beta0_true = -0.2; % Intercept
beta1_true = 0.4;  % coefficient of x1 data
beta2_true = 1.2;  % coefficient of x2 data

y_data =  beta0_true + beta1_true*x1_data.^2 + beta2_true*x2_data.^2 ; % Create test response data

A = [ ones(length(x1_data),1)   x1_data.^2   x2_data.^2 ]; % Design matrix
Y = y_data;

beta_fit2 = A \ Y;

beta_fit = [-0.2   0.4   1.2]

The issue is when trying to add a linear term (such as beta4_true * x_data1 ), i.e. a tilt to the parabola.

None of which are correct. Can anyone tell me what might be going wrong here?

Thanks

None of which match the original input beta coefficients. Can anyone tell me what might be going wrong here?

Thanks

Edit

An example to show that it works when fitting a cubic function with no noise (i.e. with deterministic data, as mentioned by @Gregg H):

beta0_true = 0.3;
beta1_true = 0.6;
beta2_true = -3;
beta3_true = -2;

x_data = linspace(-3,2,100).';
epsilon =  0;       % or use epsilon = randn(length(x_data),1) to test noise

y_data = beta0_true + beta1_true*x_data + beta2_true*x_data.^2 + beta3_true*x_data.^3 + epsilon;

A = [ ones(length(x_data),1)   x_data   x_data.^2   x_data.^3 ];
Y = y_data;

beta_fit = A \ Y;

which outputs as expected:

beta_fit = [0.3 0.6 -3 2]

I am trying to use linear least squares regression to extract the coefficients of a model. Specifically, I am looking at a model with two independent predictor variables $x_1$ and $x_2$, and an output response variable $y$, with coefficients $\beta_0,\beta_1$ and $\beta_2$. (I believe this is a case of multiple linear regression.) The model is of the form $$ y_i = \beta_0 + \beta_1x_{i1} + \beta_2 x_{i2} + \varepsilon_i $$ where $i$ denotes the observation number. Or in matrix form with $N$ observations $$ \begin{align} \mathbf{Y} &= \mathbf{A}\boldsymbol{\beta} + \boldsymbol{\varepsilon} \\ \begin{pmatrix} y_1 \\ y_2 \\ \vdots \\ y_N \end{pmatrix} &= \begin{pmatrix} 1 & x_{11} & x_{21} \\ 1 & x_{12} & x_{22}\\ \vdots & \vdots & \vdots \\ 1 & x_{1N} & x_{2N} \end{pmatrix} \begin{pmatrix} \beta_0 \\ \beta_1 \\ \beta_2 \end{pmatrix} + \begin{pmatrix} \varepsilon_1 \\ \varepsilon_2\\ \vdots \\ \varepsilon_N \end{pmatrix} \end{align} $$ I should be able to solve this using the least squares condition $$\boldsymbol{\beta} = (\mathbf{A}^\textrm{T}\mathbf{A})^{-1}\mathbf{A}^\textrm{T}\mathbf{Y} $$.

I made a simple (Matlab) script to see if I could recover the correct beta coefficients on a test case, given below, but it does not give the correct result. Can anyone tell me what might be going wrong here?

x1_data = linspace(-1,1,20).';      % First independent variable
x2_data = linspace(-0.3,0.6,20).';  % Second independent variable

beta0 = -0.2; % Intercept
beta1 = 0.4;  % coefficient of x1 data
beta2 = 1.2;  % coefficient of x2 data

y_data =  beta0 + beta1*x1_data + beta2*x2_data; % Create test response data

A = [ ones(length(x1_data),1)   x1_data   x2_data ]; % Design matrix
Y = y_data;

beta_fit1 = (A.'*A) \ (A.'*Y);  % Method #1
beta_fit2 = A \ Y;              % Method #2

[Q,R] = qr(A);
beta_fit3 = R \ (Q'*Y);         % Method #3

Thanks

I am trying to use linear least squares regression to extract the coefficients of a model. Specifically, I am looking at a model with two independent predictor variables $x_1$ and $x_2$, and an output response variable $y$, with coefficients $\beta_0,\beta_1$ and $\beta_2$. (I believe this is a case of multiple linear regression.) The model is of the form $$ y_i = \beta_0 + \beta_1x_{i1} + \beta_2 x_{i2} + \varepsilon_i \tag{1} $$ where $i$ denotes the observation number. Or in matrix form with $N$ observations $$ \begin{align} \mathbf{Y} &= \mathbf{A}\boldsymbol{\beta} + \boldsymbol{\varepsilon} \\ \begin{pmatrix} y_1 \\ y_2 \\ \vdots \\ y_N \end{pmatrix} &= \begin{pmatrix} 1 & x_{11} & x_{21} \\ 1 & x_{12} & x_{22}\\ \vdots & \vdots & \vdots \\ 1 & x_{1N} & x_{2N} \end{pmatrix} \begin{pmatrix} \beta_0 \\ \beta_1 \\ \beta_2 \end{pmatrix} + \begin{pmatrix} \varepsilon_1 \\ \varepsilon_2\\ \vdots \\ \varepsilon_N \end{pmatrix} \end{align} \tag{2} $$ I should be able to solve this using the least squares condition $$\boldsymbol{\beta} = (\mathbf{A}^\textrm{T}\mathbf{A})^{-1}\mathbf{A}^\textrm{T}\mathbf{Y} \tag{3}$$.

I made a simple (Matlab) script to see if I could recover the correct beta coefficients on a test case, given below. I used three methods: 1) directly solving Eq.(3) using Matlab's backslash operator 2) the same thing, except neglecting the transpose in Eq.(3), since it coincides with the least squares result. 3) solving Eq.(3) using a QR decomposition insead.

x1_data = linspace(-1,1,20).';      % First independent variable
x2_data = linspace(-0.3,0.6,20).';  % Second independent variable

beta0 = -0.2; % Intercept
beta1 = 0.4;  % coefficient of x1 data
beta2 = 1.2;  % coefficient of x2 data

y_data =  beta0 + beta1*x1_data + beta2*x2_data; % Create test response data

A = [ ones(length(x1_data),1)   x1_data   x2_data ]; % Design matrix
Y = y_data;

beta_fit1 = (A.'*A) \ (A.'*Y);  % Method #1
beta_fit2 = A \ Y;              % Method #2

[Q,R] = qr(A);
beta_fit3 = R \ (Q'*Y);         % Method #3

which outputs:

beta_fit1 = [-254.4 -762.3 1696.0]
beta_fit2 = [-0.02 0.94 0]
beta_fit= [-0.02 0.94 0]

None of which are correct. Can anyone tell me what might be going wrong here?

Thanks