I'm running Linear Regression on a synthetic data set based on two features - $x_1$,$x_2$. The target value is $x_2 + x_1^2 + x_1x_2$

Running linear regression (over Spark with L-BFGS) with just the right features produces a very good result over the training set - (coeffs and intercept are the results returned by the algorithm)

The features are $(x_1,x_2,x_1^2,x_2^2,x_1x_2)$

coeffs: List(0.0, 1.0, 1.0, 0.0, 1.0) intercept 0.000039508668123 error: 0.0000286960471777 lambda: 0.0

However, whenever I add additional features, somehow the result gets worse! Note that the new features aren't even used at all - they all get 0.

The features are $(x_1,x_2,x_1^2,x_2^2,x_1x_2,x_1^3,x_2^3,x_1^2x_2,x_2^2x_1)$

coeffs: List(0.07, 2.02, 1.0, -0.01, 1.0, 0.0, 0.0, 0.0, 0.0) intercept -23.02563372048826 error: 5.03134487460479 lambda: 0.0

Note that lambda is 0 so no regularization is utilized.

Any idea why this might be happening?

  • $\begingroup$ What is the length of your feature vectors? i.e. what is the size of the problem? $\endgroup$
    – adunaic
    Aug 10 '17 at 12:33
  • $\begingroup$ I'm not sure how you solve the linear regression, but if it uses gradient descent then it will take more iterations to get to the same score (more parameters to optimize = slower optimization) $\endgroup$
    – Cherny
    Aug 10 '17 at 13:11

Adding more details may help others to answer your question. With current information here are my guess:

  • One possible problem is you do not have sufficient data to run the third order polynomial expansion.

  • Another possible problem is the ground truth is $x_2+x_1^2+x_1x_2$, so the ground truth coefficients for other terms are zero.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.