# Linear Regression coefficients through ANN

I am struggling to get ANN to estimate constant and coefficients of a linear regression problem. Unfortunately my results are way off from the expected. Kindy take a look at the reproducible code below.

from sklearn import datasets
from sklearn.model_selection import train_test_split

X, y = sklearn.datasets.make_regression(n_samples=500, n_features=2, n_targets=1, bias=10.0, noise=15, shuffle=True, random_state=4)

from sklearn.preprocessing import MinMaxScaler
mm = MinMaxScaler()
X = mm.fit_transform(X)

X_train, X_test, y_train, y_test = train_test_split(Xc, y, test_size=0.33, random_state=42)

## Let's evaluate linear regression results
import statsmodels.api as sm

reg = sm.OLS(y_train, X_train)
reg=reg.fit()

y_train_pred = reg.predict(X_train)
y_test_pred = reg.predict(X_test)

print(r2_score(y_train,y_train_pred))
print(r2_score(y_test,y_test_pred))
reg.summary()

####    coef    std err t   P>|t|   [0.025  0.975]
#const  -216.2056   3.672   -58.876 0.000   -223.429    -208.982
#x1      174.5213   4.913   35.520  0.000   164.856 184.186
#x2      272.8751   4.779   57.094  0.000   263.473 282.277

# Let's build an ANN and evaluate results

X_train = X_train[:,1:] ## constant dropped because bias=True
X_test = X_test[:,1:]

model = Sequential()
model.fit(X_train, y_train, epochs=500, validation_split=0.2, verbose = 0)

import matplotlib.pyplot as plt
plt.scatter(y_train, y_train,  color='black')
plt.plot(y_train, model.predict(X_train), color='blue', linewidth=3)
plt.show()

for layer in model.layers:
weights = layer.get_weights()
print(weights)

## [array([[0.9757047], [0.839206 ]], array([0.8604008]]


The weights are very different from regression coefficients. Where am I going wrong?

Although more sophisticated optimizers do well in general for solving highly non-convex problems, using vanilla gradient descent may very well suffice for a convex one such as this (i.e. linear regression):

model.compile(loss='mean_squared_error', optimizer='sgd')


A small touch on your plotting:

plt.plot(y_train, model.predict(X_train), 'x', color='blue', linewidth=3)


which gives the following fit (using 1.5K epochs):

• WOW!! Surprised that choice of optimizer made such a difference. Thanks very much. Commented Dec 28, 2019 at 3:15