I have some economic data - so very interdependent and complicated - from worldbank. It also has a lot of missing values. I normalized it so all features are approximately normal, zero centered and with variance of 1. I then filled the missing values with zeros - I tried some different ways of imputation, but that one works almost the best and is very simple. The data is 25 dimensions, 60 timesteps, and for ~200 countries. Then, I fit an RNN which would predict all 25 variables one step ahead. Here is the problem: I first left out the years 2013-2016 as validation, and the model predicted very reasonable figures for 2013-2026, which trends and so on: . Note that the first years there is no data, and this is how the zero-fill looks. I then fit the exact same model, only with no validation data - so with data up to 2016. This time the predictions were almost exactly constants - see here:. It seems the model is predicting values that are very close to zero. The code was exactly the same:
# select train data years_to_validate = 0 years_to_predict = 10 years_train = generate_year_list(stop=2016-years_to_validate) years_val = generate_year_list(start=2016-years_to_validate+1) train_panel = normalized_panel[:, years_train, :].copy() train_panel.fillna(0, inplace=True) # create 1-step-ahead model hl = [120,120] print "ARCHITECTURE:", hl X_train = train_panel[:,years_train,:][:,:-1,:] y_train = train_panel[:,years_train,:][:,1:,:] model = dense_gradient_model(X_train, y_train, hidden_layers=hl, d=0.2, patience=50, epochs=200) # finally, predict for start, year in enumerate(years_val+years_predict): predictions = model.predict(train_panel[:,start+1:,:].values)[:,-1,:] train_panel = train_panel.swapaxes(0,1) new_year_df = pd.DataFrame(data=predictions,index=train_panel.axes, columns=y_train.axes) train_panel[year] = new_year_df train_panel = train_panel.swapaxes(0,1) print "score:", rmse(normalized_panel[:,years_val,:].values, train_panel[:,years_val,:].values) #revert to original scale and distributions train_panel = normalizer.renormalize(train_panel)
and here is the model:
def dense_gradient_model(X_panel, y_panel, epochs=5000, hidden_layers = , d=0.2, patience=30): X = X_panel.fillna(0).values y = y_panel.fillna(0).values n_samples, n_timesteps, n_feat = X.shape main_input = Input(shape=(n_timesteps, n_feat), name='main_input') layers = [main_input] for hl in hidden_layers: layers.append(Bidirectional(LSTM(hl, return_sequences=True, dropout_W = d, dropout_U = d) )(layers[-1])) # merge_l = merge([layers[-1], main_input], mode='concat') outputs = Bidirectional(LSTM(y.shape[-1], return_sequences=True, dropout_W = d, dropout_U = d), merge_mode='sum')(layers[-1]) model = Model(input=main_input, output = outputs) model.compile(optimizer='rmsprop', loss='mse') early_stopping = EarlyStopping(patience=patience) history = model.fit(X, y, nb_epoch = epochs, validation_split=0.1, callbacks=[early_stopping]) return model
Why is this happening? The model is the same, the hyperparameters and the same, and the data is more.