I've trained a simple Neural net (scikit learn's MLPClassifier) in order to perform binary classification on some data (the titanic dummy problem on kaggle).
I know that standardizing data prior to using it to train neural nets is supposed to make training faster but I didn't expect it to change the results (it scores around 65% accuracy on non-standardized data vs around 80% for standardized data (local test set) )
# comment the following code to see the difference scaler = StandardScaler() scaler.fit(X_train) X_train = scaler.transform(X_train) X_test = scaler.transform(X_test)
I noticed that the results of both approaches seem to converge when I train the neural net using full batch gradient descent (the default is to use mini batches with stochastic gradient descent).
see the notebook here (look at parapgraph )
Question: Are these discrepancies solely due to the stochastic nature of SGD?