I have some economic data - so very interdependent and complicated - from worldbank. It also has a lot of missing values. I normalized it so all features are approximately normal, zero centered and with variance of 1. I then filled the missing values with zeros - I tried some different ways of imputation, but that one works almost the best and is very simple. The data is 25 dimensions, 60 timesteps, and for ~200 countries. Then, I fit an RNN which would predict all 25 variables one step ahead. Here is the problem: I first left out the years 2013-2016 as validation, and the model predicted very reasonable figures for 2013-2026, which trends and so on: image here. Note that the first years there is no data, and this is how the zero-fill looks. I then fit the exact same model, only with no validation data - so with data up to 2016. This time the predictions were almost exactly constants - see here:with no validation data withholded. It seems the model is predicting values that are very close to zero. The code was exactly the same:

# select train data
years_to_validate = 0
years_to_predict  = 10
years_train = generate_year_list(stop=2016-years_to_validate)
years_val = generate_year_list(start=2016-years_to_validate+1)
train_panel = normalized_panel[:, years_train, :].copy()
train_panel.fillna(0, inplace=True)

# create 1-step-ahead model
hl = [120,120]
print "ARCHITECTURE:", hl
X_train = train_panel[:,years_train,:][:,:-1,:]
y_train = train_panel[:,years_train,:][:,1:,:]
model = dense_gradient_model(X_train, y_train, 

# finally, predict
for start, year in enumerate(years_val+years_predict):
    predictions = model.predict(train_panel[:,start+1:,:].values)[:,-1,:]
    train_panel = train_panel.swapaxes(0,1)
    new_year_df = pd.DataFrame(data=predictions,index=train_panel.axes[1], columns=y_train.axes[2])
    train_panel[year] = new_year_df
    train_panel = train_panel.swapaxes(0,1)
print "score:", rmse(normalized_panel[:,years_val,:].values, 

#revert to original scale and distributions
train_panel = normalizer.renormalize(train_panel)

and here is the model:

def dense_gradient_model(X_panel, y_panel, epochs=5000, hidden_layers = [120], d=0.2, patience=30):
    X = X_panel.fillna(0).values
    y = y_panel.fillna(0).values
    n_samples, n_timesteps, n_feat = X.shape
    main_input = Input(shape=(n_timesteps, n_feat), name='main_input')
    layers = [main_input]
    for hl in hidden_layers:
                                        dropout_W = d, 
                                        dropout_U = d)
    # merge_l = merge([layers[-1], main_input], mode='concat')
    outputs = Bidirectional(LSTM(y.shape[-1], 
                                dropout_W = d, 
                                dropout_U = d), 
    model = Model(input=main_input, output = outputs)
    model.compile(optimizer='rmsprop', loss='mse')
    early_stopping = EarlyStopping(patience=patience)
    history = model.fit(X, y, nb_epoch = epochs, validation_split=0.1, callbacks=[early_stopping])
    return model

Why is this happening? The model is the same, the hyperparameters and the same, and the data is more.

  • 1
    $\begingroup$ I would suspect something with the actual 2016 data that you added. Is that missing? Zero-filled missing data in the most recent observation may be producing that sharp change toward the mean for the blue and yellow lines. $\endgroup$ – zbicyclist Jan 14 '17 at 22:20
  • 1
    $\begingroup$ Yeah, I got there too. Turns out, in 2016 those specific features were missing, and the net had learned that zeros are typically consecutive. In a way, I'm surprised how close to the mean it got, it shows that the network has more capacity than I expected. $\endgroup$ – Hristo Buyukliev Jan 15 '17 at 15:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.