I am struggling to get ANN to estimate constant and coefficients of a linear regression problem. Unfortunately my results are way off from the expected. Kindy take a look at the reproducible code below.

from sklearn import datasets
from sklearn.model_selection import train_test_split

X, y = sklearn.datasets.make_regression(n_samples=500, n_features=2, n_targets=1, bias=10.0, noise=15, shuffle=True, random_state=4)

from sklearn.preprocessing import MinMaxScaler
mm = MinMaxScaler()
X = mm.fit_transform(X)
Xc = sm.add_constant(X)

X_train, X_test, y_train, y_test = train_test_split(Xc, y, test_size=0.33, random_state=42)

## Let's evaluate linear regression results
import statsmodels.api as sm

reg = sm.OLS(y_train, X_train)

y_train_pred = reg.predict(X_train)
y_test_pred = reg.predict(X_test)


####    coef    std err t   P>|t|   [0.025  0.975]
#const  -216.2056   3.672   -58.876 0.000   -223.429    -208.982
#x1      174.5213   4.913   35.520  0.000   164.856 184.186
#x2      272.8751   4.779   57.094  0.000   263.473 282.277

# Let's build an ANN and evaluate results

X_train = X_train[:,1:] ## constant dropped because bias=True
X_test = X_test[:,1:]

model = Sequential()
model.add(Dense(1, input_dim=2, kernel_initializer='normal', activation='linear'))
model.compile(loss='mean_squared_error', optimizer='adam')
model.fit(X_train, y_train, epochs=500, validation_split=0.2, verbose = 0) 

import matplotlib.pyplot as plt
plt.scatter(y_train, y_train,  color='black')
plt.plot(y_train, model.predict(X_train), color='blue', linewidth=3)

for layer in model.layers:
    weights = layer.get_weights()

## [array([[0.9757047], [0.839206 ]], array([0.8604008]]

The weights are very different from regression coefficients. Where am I going wrong?


1 Answer 1


Although more sophisticated optimizers do well in general for solving highly non-convex problems, using vanilla gradient descent may very well suffice for a convex one such as this (i.e. linear regression):

model.compile(loss='mean_squared_error', optimizer='sgd')

A small touch on your plotting:

plt.plot(y_train, model.predict(X_train), 'x', color='blue', linewidth=3)

which gives the following fit (using 1.5K epochs):

enter image description here

  • $\begingroup$ WOW!! Surprised that choice of optimizer made such a difference. Thanks very much. $\endgroup$ Commented Dec 28, 2019 at 3:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.