Neural Net learning opposite trend in 1-variable regression problem I'm trying to fit a neural net to a pretty simple 1-variable regression problem. My output is a probability and the input is a continuous feature. The association is clearly negative. The below gives
X = np.array(impute_df['runner1_speed'])
y = np.array(impute_df['dest_1__'])
sns.scatterplot(x=X, y=y);


But with the simple net below, the fitted values are positively associated with the predictor.
inputs = keras.Input(shape=(1,))
hidden = layers.Dense(16, activation="relu")(inputs)
outputs = layers.Dense(1, activation="sigmoid")(hidden)
impute_mod = keras.Model(inputs=inputs, outputs=outputs, name="impute_mod")
impute_mod.compile(
    loss=keras.losses.MeanSquaredError(),
    optimizer=keras.optimizers.Adam(learning_rate=0.001))
history = impute_mod.fit(X, y, batch_size=4, epochs=20, verbose=0)
sns.scatterplot(x=X, y=impute_mod.predict(X)[:,0])


I must be doing something stupid but I can't figure out what. Any ideas? Thanks so much!
 A: My suspicion is that this network hasn't finished training. Quite plainly, the model fit is poor, so we should think through all parts of the modeling process to isolate factors which might contribute to this poor fit.

*

*The training or validation loss may have plateaued, but this is not necessarily evidence that the model has finished training. Indeed, the trend is opposite the prediction, so my inference is that the model has simply plateaued at a high value of the loss.


*One reason that the loss could plateau at a high value is that the sigmoid function has become saturated, so continuing training will be a slow process. This is because gradient-based updates are dependent on the value of the gradients, and for a saturated sigmoid unit, these gradients will be small.
My surmise is that continuing training for many more epochs may solve the problem. OP attempted this and commented that the model improved, finding the negative association implied by the scatter plot.
An alternative model would use an identity activation in the output layer. This changes the model and its interpretation, but because we remove the sigmoid unit and its saturating non-linearity, this model might be easier to estimate.
