0
$\begingroup$

I have a binary classification problem(benign/malicious) and I have applied simple neural network with one hidden layer for solving the problem. I have 46 features in my dataset and for the hidden layer I am using 46/2. Also my data is not scaled so for it I am using standardscaler. I am getting accuracy of around 99.79% with the code. However the accuracy was unexpected, I was expecting around 93-94%, I am afraid that I am leaking some data or making some silly mistake.

def create_baseline():
    model = Sequential()
    model.add(Dense(23, input_dim=46, kernel_initializer='normal', activation='relu'))
    model.add(Dense(1, kernel_initializer='normal', activation='sigmoid'))
    model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])
    return model


estimators = []
estimators.append(('standardize', StandardScaler()))
estimators.append(('mlp', KerasClassifier(build_fn=create_baseline, epochs=1000, batch_size=len(X), validation_split=0.15, verbose=0)))
pipeline = Pipeline(estimators)
kfold = StratifiedKFold(n_splits=5, shuffle=True, random_state=seed)
scores = cross_val_score(pipeline, X, y, cv=kfold)
print("%.2f%%" % (scores[1]*100))

I am using around 5871 malicious samples and 3488 benign samples.

Just to cross check I have implemented without cross validation, and I am getting ~99% accuracy in that also. Train test split is (70% - 30%)

scale = StandardScaler(with_mean=0, with_std=1)
new_X_train = scale.fit_transform(X_train)
new_X_test = scale.transform(X_test)


model = Sequential()

model.add(Dense(23, input_dim=46, kernel_initializer="normal"))
model.add(PReLU(alpha_initializer='zero', weights=None))

model.add(Dense(1, kernel_initializer='normal'))
model.add(Activation('sigmoid'))

model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])

history = model.fit(new_X_train, y_train, epochs=1000, batch_size=len(X_train), validation_split=0.15)

scores = model.evaluate(new_X_test, y_test)
print("%s: %.2f%%" % (model.metrics_names[1], scores[1]*100))

print(history.history.keys())

plt.plot(history.history['acc'])
plt.plot(history.history['val_acc'])
plt.title('model accuracy')
plt.ylabel('accuracy')
plt.xlabel('epoch')
plt.legend(['train', 'test'], loc='upper left')
plt.show()

plt.plot(history.history['loss'])
plt.plot(history.history['val_loss'])
plt.title('model loss')
plt.ylabel('loss')
plt.xlabel('epoch')
plt.legend(['train', 'test'], loc='upper left')
plt.show()

The plots I am getting are: lossaccuracy

$\endgroup$
  • $\begingroup$ I don't see any obvious errors, but just in case, try smaller batch size. I am not sure how keras handles when batch size is larger than the training set (which is your case, batch_size=len(X_train) but 15% of the training set are reserved for validation) $\endgroup$ – Jan Kukacka Feb 17 '18 at 9:41
  • $\begingroup$ Now I am getting accuracy around 99.96% with batch_size = len(X_train)/4. Are there any implementation mistakes? Or this is because of the small datasize? $\endgroup$ – mg9893 Feb 17 '18 at 9:49
  • $\begingroup$ Your dataset size seems okay for such a small network. Are you sure there is no problem with your data, i.e. some of the features is very highly correlated with the output? $\endgroup$ – Jan Kukacka Feb 17 '18 at 10:12
  • $\begingroup$ Can you suggest few methods to check that? $\endgroup$ – mg9893 Feb 17 '18 at 10:16
  • 1
    $\begingroup$ Well, computing the correlation between individual features and the output would be a good start. $\endgroup$ – Jan Kukacka Feb 17 '18 at 10:17
2
$\begingroup$

The code looks fine. Without access to your data, at least in standardized/anonymized form, it's impossible to give a definitive answer. However, when in this kind of predicament there is a standard set of checks worth going through1: some of these steps don't apply to you, but I'll include them anyway to make the list more comprehensive. As a best practice these checks should be done before training the DNN, but they can be of help also when you have already used $k-$fold CV and/or training/validation/test set split to estimate correctly the generalization error, and you found a suspiciously low value.

  1. check that all data transforms (or estimators, in Tensorflow lingo), are fit on the training set and applied to the test set, for each fold of $k-$fold CV. Using pipelines, as you did, is a simple way to make sure of this. Just check that all operations on data are performed inside the pipeline.
  2. check that you're using the right metric for classification. In your case, accuracy makes sense because the two classes are reasonably balanced. But on a medical data set, where the incidence of the disease in the population is $0.1\%$, you will get $99.9\%$ accuracy on a representative data set, just by classifying each point to the majority class. In this case, $99.9\%$ accuracy is nothing to be surprised of, and actually accuracy is not the right metric to use here.
  3. check that the problem is not overly simple. For example, it could be linearly separable. Some folks think this can be seen with PCA: PCA has nothing to do with linearly separable classification problems. Instead, use linear discriminant analysis or linear SVM: both will fit a linearly separable problem perfectly (or nearly perfectly), and get excellent out-of-sample performance. Alternatively, you can use regularized logistic regression (unregularized LR estimated with MLE is ill-posed for linearly separable problems). Plotting decision regions would also help a lot: mlxtend's plot_decision_regions is a simple and convenient way to do this. Of course when you have more than 2 predictors you can only visualize 2D projections of the decision boundary, on the planes defined by the various pairs of input features.
  4. create a correlation matrix including inputs and the class label: of course, computing the Pearson correlation coefficient between a continuous and a categorical variable doesn't make sense, but most good visualization tools (i.e., not matplotlib) will automatically switch to a visualization which makes sense. If there are a few inputs which are very correlated with the class label, then, even if there is significant correlation among inputs, a neural network is likely to do well. Note: after performing check 3, this check may seem redundant, but I believe visualization never hurts.
  5. an heuristic I've found useful sometimes is to refit the NN on shuffled (permutated) labels. In this case the neural network has only one way to get good training set error: memorize the training set. This will usually manifest in much longer training time. Also, it has no way to get good test set error. If you still get excellent performance on the test set, then either there is something seriously wrong with your code (consider writing unit tests for each method you defined) or with the framework you're using. Consider opening a GitHub issue at this point.
  6. Go to the extreme opposite: train on a single (or very few) data points. You should get $\sim 100\%$ accuracy on the training set (and extremely low on the test set). If you don't, again there's something seriously flawed with your code or framework.
  7. Finally, try to obtain a few new test points, which you've never personally seen (let alone used in training) until now. The new test points must be representative of your real life usage of the NN. If you still get excellent accuracy, then get used to it, you're just very good at Deep Learning ;-)

1 though I frankly would love to be in this kind of predicament more often, when my NN does greatly both under $k-$fold CV and on a well-designed validation set, without having to touch a single line of my architecture ;-)

$\endgroup$
  • $\begingroup$ ...not matplotlib but what? $\endgroup$ – amoeba Feb 20 '18 at 13:40
  • $\begingroup$ @amoeba DataExplorer, GGpairs or even pairs, with customized panel functions :-D $\endgroup$ – DeltaIV Feb 20 '18 at 14:11
  • $\begingroup$ Wait, that's for R, I thought we are talking Python (since the question was in Python)... $\endgroup$ – amoeba Feb 20 '18 at 14:12
  • $\begingroup$ @amoeba I know, I know, I was just messin' with you ;-) (I thought the smiley would make it clear). For Python, to the best of my knowledge, there is nothing which is completely automatic. But combining the excellent seaborn.pairplot (especially with the hue setting) with factorplot or in general checking seaborn [capabilities for plotting of pairwise relationships]( seaborn.pydata.org/tutorial/…) should 1/ $\endgroup$ – DeltaIV Feb 20 '18 at 14:28
  • $\begingroup$ 2/ help. At least, it's a vast improvement over all the manual tinkering you would have to do in matplotlib. Finally, for simpler cases than what I was talking about, pandas plotting.scatter_matrix can help, but you miss the possibility to combine it with other seaborn bonanza. By the way, any suggestions for learning Python, coming from R/matlab? Courses, books, etc. $\endgroup$ – DeltaIV Feb 20 '18 at 14:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.