I am trying to predict an output dependent variable $Y$ for a feature set of 4 independent variables - $X_1, \dots,X_4$ using regression. Here is a scatter plot of each of the dependent variables against the dependent variable $Y$.

enter image description here

Linear regression of the features $X_1, \dots, X_4$ was not able to predict $Y$ well. The $R^2$ of the predictions was just 0.1. I am thinking that a good fit might be obtained if I used more features which are polynomial (or some other function such as log/square root) of the input features instead of directly using the features. I tried fitting Y against second order polynomials such as $X_iX_j$ and $X_i^2$. But it doesn't seem to help.

I think that the shape of the scatter plots of $X_1$ and $X_2$ might be giving a clue. Based on the scatter plots here, can you give any recommendations about the list of features I can use to predict $Y$? How I can improve the prediction of $Y$? I have tried feature scaling and regularization. They don't seem to help.

I am using scikit-learn to fit and predict. Here is the sample code.

*X_train,X_test,y_train,y_test = train_test_split(X,y,test_size=0.3, random_state=1)
lin_reg1 = LinearRegression()
print('The score is {}'.format(lin_reg1.score(X_test,y_test)))*
  • 2
    $\begingroup$ Do you have to use linear regression? How does regressing using random forests perform? Or k-nn regression? $\endgroup$
    – KirkD_CO
    Mar 11, 2018 at 1:03
  • 2
    $\begingroup$ Regarding "Based on the scatter plots here, can you give any recommendations about the list of features I can use to predict $Y$", then it is hard to see a) whether there is any marginal dependence at all (it seems like this is not the case) and b) whether you just have a lots of data points. Plot a smooth instead. $\endgroup$ Mar 11, 2018 at 8:26
  • 1
    $\begingroup$ By the way, you should not cross post. You have also posted this question on Stackexchange. $\endgroup$ Mar 11, 2018 at 8:28
  • $\begingroup$ Thanks a lot for your reply, KirkDCO. I am not restricted to use only linear regression. I will try random forest and k-nn regression and update you. Thanks a lot for your suggestions. It really helps a ML newbie like me. $\endgroup$ Mar 11, 2018 at 11:26
  • $\begingroup$ Thanks a lot for your feedback, Benjamin. I will try your suggestion and update you. And sorry about cross posting. Am new to stackexchange as well. Just trying to learn the ropes.. $\endgroup$ Mar 11, 2018 at 11:28

1 Answer 1


It looks like you have some multicolinearity issues. Have you looked at the correlation between X1 and X2, as well as between X3 and X4 ? X3 and X4 are borderline identity. I would reduce the dimensionality using PCA(loosing inference about predictors) or switch to a GAM; a GEE may be useful if the issue is a result of improper or unknown covariance structure. However, your plots look like shotgun blasts, you have likely picked poor predictors.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.