3
$\begingroup$

I've trained a simple Neural net (scikit learn's MLPClassifier) in order to perform binary classification on some data (the titanic dummy problem on kaggle).

I know that standardizing data prior to using it to train neural nets is supposed to make training faster but I didn't expect it to change the results (it scores around 65% accuracy on non-standardized data vs around 80% for standardized data (local test set) )

# comment the following code to see the difference
scaler = StandardScaler()
scaler.fit(X_train)

X_train = scaler.transform(X_train)
X_test = scaler.transform(X_test)

I noticed that the results of both approaches seem to converge when I train the neural net using full batch gradient descent (the default is to use mini batches with stochastic gradient descent).

see the notebook here (look at parapgraph [8])

Question: Are these discrepancies solely due to the stochastic nature of SGD?

Improve this question
$\endgroup$
1
  • $\begingroup$ Trying to understand the situation. You have 2 variables: scaled vs. non-scaled, and SGD vs. batch gradient descent, so 4 possible combinations. Can you tell us the training and test set error for each of these configurations? Or, if you have time, plot learning rate curves for each. Also, what kind of hyperparameters do you have, and what procedure are you using to set them in each case? $\endgroup$
    – user20160
    Commented Mar 13, 2017 at 10:42

1 Answer 1

Reset to default
1
$\begingroup$

Refer to this link for better understanding.

standardizing the inputs can make training faster and reduce the chances of getting stuck in local optima.

Hence, SGD is sensitive to feature scaling.

Improve this answer
$\endgroup$
1
  • $\begingroup$ The first paragraph is not relevant to this situation, while the second basically reiterates the OP's initial understanding.... $\endgroup$
    – Danica
    Commented Mar 31, 2017 at 10:31

Not the answer you're looking for? Browse other questions tagged or ask your own question.