3
$\begingroup$

I've trained a simple Neural net (scikit learn's MLPClassifier) in order to perform binary classification on some data (the titanic dummy problem on kaggle).

I know that standardizing data prior to using it to train neural nets is supposed to make training faster but I didn't expect it to change the results (it scores around 65% accuracy on non-standardized data vs around 80% for standardized data (local test set) )

# comment the following code to see the difference
scaler = StandardScaler()
scaler.fit(X_train)

X_train = scaler.transform(X_train)
X_test = scaler.transform(X_test)

I noticed that the results of both approaches seem to converge when I train the neural net using full batch gradient descent (the default is to use mini batches with stochastic gradient descent).

see the notebook here (look at parapgraph [8])

Question: Are these discrepancies solely due to the stochastic nature of SGD?

$\endgroup$
  • $\begingroup$ Trying to understand the situation. You have 2 variables: scaled vs. non-scaled, and SGD vs. batch gradient descent, so 4 possible combinations. Can you tell us the training and test set error for each of these configurations? Or, if you have time, plot learning rate curves for each. Also, what kind of hyperparameters do you have, and what procedure are you using to set them in each case? $\endgroup$ – user20160 Mar 13 '17 at 10:42
1
$\begingroup$

Refer to this link for better understanding.

standardizing the inputs can make training faster and reduce the chances of getting stuck in local optima.

Hence, SGD is sensitive to feature scaling.

$\endgroup$
  • $\begingroup$ The first paragraph is not relevant to this situation, while the second basically reiterates the OP's initial understanding.... $\endgroup$ – Dougal Mar 31 '17 at 10:31

Not the answer you're looking for? Browse other questions tagged or ask your own question.