# RNN starts to predict only zeros when given a bit more data

I have some economic data - so very interdependent and complicated - from worldbank. It also has a lot of missing values. I normalized it so all features are approximately normal, zero centered and with variance of 1. I then filled the missing values with zeros - I tried some different ways of imputation, but that one works almost the best and is very simple. The data is 25 dimensions, 60 timesteps, and for ~200 countries. Then, I fit an RNN which would predict all 25 variables one step ahead. Here is the problem: I first left out the years 2013-2016 as validation, and the model predicted very reasonable figures for 2013-2026, which trends and so on: . Note that the first years there is no data, and this is how the zero-fill looks. I then fit the exact same model, only with no validation data - so with data up to 2016. This time the predictions were almost exactly constants - see here:. It seems the model is predicting values that are very close to zero. The code was exactly the same:

# select train data
years_to_validate = 0
years_to_predict  = 10
years_train = generate_year_list(stop=2016-years_to_validate)
years_val = generate_year_list(start=2016-years_to_validate+1)
train_panel = normalized_panel[:, years_train, :].copy()
train_panel.fillna(0, inplace=True)

hl = [120,120]
print "ARCHITECTURE:", hl
X_train = train_panel[:,years_train,:][:,:-1,:]
y_train = train_panel[:,years_train,:][:,1:,:]
hidden_layers=hl,
d=0.2,
patience=50,
epochs=200)

# finally, predict
for start, year in enumerate(years_val+years_predict):
predictions = model.predict(train_panel[:,start+1:,:].values)[:,-1,:]
train_panel = train_panel.swapaxes(0,1)
new_year_df = pd.DataFrame(data=predictions,index=train_panel.axes[1], columns=y_train.axes[2])
train_panel[year] = new_year_df
train_panel = train_panel.swapaxes(0,1)
print "score:", rmse(normalized_panel[:,years_val,:].values,
train_panel[:,years_val,:].values)

#revert to original scale and distributions
train_panel = normalizer.renormalize(train_panel)


and here is the model:

def dense_gradient_model(X_panel, y_panel, epochs=5000, hidden_layers = [120], d=0.2, patience=30):
X = X_panel.fillna(0).values
y = y_panel.fillna(0).values
n_samples, n_timesteps, n_feat = X.shape
main_input = Input(shape=(n_timesteps, n_feat), name='main_input')
layers = [main_input]
for hl in hidden_layers:
layers.append(Bidirectional(LSTM(hl,
return_sequences=True,
dropout_W = d,
dropout_U = d)
)(layers[-1]))
# merge_l = merge([layers[-1], main_input], mode='concat')
outputs = Bidirectional(LSTM(y.shape[-1],
return_sequences=True,
dropout_W = d,
dropout_U = d),
merge_mode='sum')(layers[-1])
model = Model(input=main_input, output = outputs)
model.compile(optimizer='rmsprop', loss='mse')
early_stopping = EarlyStopping(patience=patience)
history = model.fit(X, y, nb_epoch = epochs, validation_split=0.1, callbacks=[early_stopping])
return model


Why is this happening? The model is the same, the hyperparameters and the same, and the data is more.

• I would suspect something with the actual 2016 data that you added. Is that missing? Zero-filled missing data in the most recent observation may be producing that sharp change toward the mean for the blue and yellow lines. – zbicyclist Jan 14 '17 at 22:20
• Yeah, I got there too. Turns out, in 2016 those specific features were missing, and the net had learned that zeros are typically consecutive. In a way, I'm surprised how close to the mean it got, it shows that the network has more capacity than I expected. – Hristo Buyukliev Jan 15 '17 at 15:12