I'm running Linear Regression on a synthetic data set based on two features - $x_1$,$x_2$. The target value is $x_2 + x_1^2 + x_1x_2$
Running linear regression (over Spark with L-BFGS) with just the right features produces a very good result over the training set - (coeffs and intercept are the results returned by the algorithm)
The features are $(x_1,x_2,x_1^2,x_2^2,x_1x_2)$
coeffs: List(0.0, 1.0, 1.0, 0.0, 1.0) intercept 0.000039508668123 error: 0.0000286960471777 lambda: 0.0
However, whenever I add additional features, somehow the result gets worse! Note that the new features aren't even used at all - they all get 0.
The features are $(x_1,x_2,x_1^2,x_2^2,x_1x_2,x_1^3,x_2^3,x_1^2x_2,x_2^2x_1)$
coeffs: List(0.07, 2.02, 1.0, -0.01, 1.0, 0.0, 0.0, 0.0, 0.0) intercept -23.02563372048826 error: 5.03134487460479 lambda: 0.0
Note that lambda is 0 so no regularization is utilized.
Any idea why this might be happening?