I am trying to solve a linear regression problem in an automated fashion, however am having a problem with extremely large weights.
I have several thousand datasets, and am running linear regression on each of them. I am doing this by using the apache commons math OLSMultipleLinearRegression library. In 90% of cases I am getting good results, however in the remaining 10% there appears to be overfitting, and in 0.1% that overfitting is horrendous (i.e. weights with order of magnitude 10^30). When running via gradient descent I can implement regularisation to deal with these issues, however is there a similar method when solving via QR decomposition?
Currently my best idea is to run QR decomposition, then if the weights are too high re-run with gradient descent. Is there a better way?