I am studying multivariate linear regression, trying to fit by the rule of thumb a multivariate model that predicts a target dataset $T$ over three datasets $D_1,D_2,D_3$, (blue, orange, green), all of them normalized.
$T$ against $D_1$, $D_2$, and $D_3$ (blue, orange, green).">
The scatter plot shows that the data sets contain outliers, and are differently correlated with the target dataset, with lines that represent the regression lines fitting $T$ with the datasets. These are the lines I traced by eye, basically just specifying a value for the slope.
I am trying to fit a regressor by eye, not using any quantitative method as OLS, just checking the plots of $T$ and of the residuals vs. the data.
%--------------------------------------------------------------------------
1) I checked the clouds of points, and concluded that one dataset, the green one $D_3$, is strongly uncorrelated with $T$, so it must be dropped. Is it a reasonable conclusion? I checked the Pearson cross-correlation matrix, and the correlation coefficient between $T$ and $D_3$ is -0.11.
%--------------------------------------------------------------------------
2) To proceed heuristically, I think this way: $$ y=\beta_1 x_1+\beta_2 x_2+\epsilon $$ is my regression model, so I set $\beta_i$ as the slope relative to the data from dataset $D_i$. By looking at the previous figure, I establish two values $\beta_1=1.2$ and $\beta_2=1$.
To check these values I plot the residuals, knowing that they should look like Gaussian noise, and I obtain the following plot:
$D_1$ and $D_2$ (blue and orange)">
The lines represent the slope of the residuals I found w.r. to $D_1$ and $D_2$, so now I imagine that $\epsilon=\varepsilon+\alpha_1 x_1+\alpha_2 x_2$, where $\varepsilon$ is really a Gaussian noise, while the $\alpha$'s should be summed to the $\beta$'s. Always by eye, I get $\alpha_1=-0.7$, and $\alpha_2=-0.9$. After correcting the $\beta$'s, the correlation model looks like this:
$T$ against $D_1$ and $D_2$ (blue and orange), using the corrected slopes.">
At this point, I expect the residuals to look like Gaussian noise, while instead they look like depending on $D_1$ and $D_2$, as in the following picture:
$D_1$ and $D_2$ (blue and orange), using the corrected slopes. The black line has slope 1.5">
The black line has a slope of 1.5.
%--------------------------------------------------------------------------
(I apologize for the very long description of the procedure)
I notice that my reasoning is not helping me in fitting a regression model. How would you solve this problem?
Using OLS, I found out that $\beta_1=0.743$ and $\beta_2=0.266$.
