# Linear regression getting worse results on training set with additional parameters

I'm running Linear Regression on a synthetic data set based on two features - $x_1$,$x_2$. The target value is $x_2 + x_1^2 + x_1x_2$

Running linear regression (over Spark with L-BFGS) with just the right features produces a very good result over the training set - (coeffs and intercept are the results returned by the algorithm)

The features are $(x_1,x_2,x_1^2,x_2^2,x_1x_2)$

coeffs: List(0.0, 1.0, 1.0, 0.0, 1.0) intercept 0.000039508668123 error: 0.0000286960471777 lambda: 0.0


However, whenever I add additional features, somehow the result gets worse! Note that the new features aren't even used at all - they all get 0.

The features are $(x_1,x_2,x_1^2,x_2^2,x_1x_2,x_1^3,x_2^3,x_1^2x_2,x_2^2x_1)$

coeffs: List(0.07, 2.02, 1.0, -0.01, 1.0, 0.0, 0.0, 0.0, 0.0) intercept -23.02563372048826 error: 5.03134487460479 lambda: 0.0


Note that lambda is 0 so no regularization is utilized.

Any idea why this might be happening?

• What is the length of your feature vectors? i.e. what is the size of the problem? – adunaic Aug 10 '17 at 12:33
• I'm not sure how you solve the linear regression, but if it uses gradient descent then it will take more iterations to get to the same score (more parameters to optimize = slower optimization) – Cherny Aug 10 '17 at 13:11

• Another possible problem is the ground truth is $x_2+x_1^2+x_1x_2$, so the ground truth coefficients for other terms are zero.