0
$\begingroup$

I am trying to construct a neural net.

I have several questions about the procedure. I can get results from a manual model, by switching parameters with my instincts and running each time. I am using also an Early Stopping strategy. My evaluation score is taking the interval of validation_MSE scores [best epoch -25, best epoch +25] and taking the mean of these values.

At beginning I splitted my data as train and test set, by the way.

1-) If I don't use an EarlyStopping strategy. When I predict the test data, will I use the weights related to the last epoch?

2-) If I use EarlyStopping with a checkpoint, this means I will save the best epoch and best weight set. And I will predict with this weight set, right?

3-) Actually, I am assuming each epoch as a different model. And models need to be cross validated to get a generalization performance. Don't we need to run a CrossValidation for EACH single epoch to get a generalization performance?

Let's say I want to run a CV for neural network. Rather than splitting as train-test at beginning, I will let CV do this. (Or I can use both, to get a final test score for best model)

For each CV fold, it will use different train data and test data. And results will be different. For ex, 1st fold will choose 27. epoch as best epoch and 2nd will choose 32. epoch. Their weights and network parameters are totally different. I don't think it is a generalization performance of my network. I have to predict with one epoch set, which one for example? I can't get that info from that CV.

Will I say, "OK, one of the epochs will give me a good performance for that neural-net configuration. I can use it." But same epoch may give a bad result for a different train set.

That's why I am wondering if there is a procedure to run CV for each epoch, and early stop according to that value, and choose the best epoch for prediction model.

4-) My approach of taking interval epoch means do not seem to be correct right now, again because of the same reason, each epoch is a different model. Why take the means of them?

5-) My neural net code is below. If you have any suggestions helping me to transform that code into a gridsearchcv format, it would be wonderful. ( If my above concerns are true, you an implement something to overcome)

from keras.models import Sequential
from keras.layers import Dense
from keras.wrappers.scikit_learn import KerasRegressor
from keras.optimizers import RMSprop, SGD
from keras.callbacks import EarlyStopping, ModelCheckpoint
from keras.models import load_model

start_time = timeit.default_timer()

x_train, x_test, y_train, y_test = train_test_split(df_x, df_y.values.ravel(), test_size = 0.25, random_state = 23)

numeric_pipe = make_pipeline(Normalizer())
categoric_pipe = make_pipeline(OneHotEncoder(sparse = False, handle_unknown='ignore'))
preprocessor = ColumnTransformer(transformers = [('num',numeric_pipe, num_cols), ('cat',categoric_pipe,cat_cols)])

all_pipe = make_pipeline(preprocessor)

x_train = all_pipe.fit_transform(x_train)
x_test = all_pipe.transform(x_test)

optimizer = RMSprop(0.001)

keras_param = {'dense1' : [10,30,50,80],
               'dense2' : [10,30,50,80],
               'optimizerr' : ['adam', optimizer]}

model = Sequential()
model.add(Dense(30, input_dim=x_train.shape[1], kernel_initializer='normal', activation='relu'))
model.add(Dense(10, activation='relu'))
model.add(Dense(1, activation='linear'))
model.summary()


model.compile(loss='mse', optimizer='adam', metrics=['mse','mae'])

patience = 100

callbacks = [EarlyStopping(monitor='val_mean_squared_error', patience=patience),
             ModelCheckpoint(filepath='best_model.h5', monitor='val_mean_squared_error', save_best_only=True)]

history = model.fit(x_train, y_train, epochs=10000, batch_size=30,  verbose=1, validation_split=0.2, callbacks = callbacks)

print("Average of last +-25 best epochs of best epoch:")
print(np.mean(history.history['val_mean_squared_error'][(np.max(history.epoch)-patience-25):(np.max(history.epoch)-patience+25)]))
print("............................")

saved_model = load_model('best_model.h5')

loss, mse, mae = saved_model.evaluate(x_test, y_test, verbose=0)
print("Test set MSE: {}".format(mse))
print("Test set MAE: {}".format(mae))
print("Test set LOSS: {}".format(loss))
print("---%0.1f minutes---" %((timeit.default_timer()-start_time)/60))
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.