4
$\begingroup$

I've been learning about neural networks with the idea of applying them to analyze a psychology experiment. I thought I had a handle on them when I got reasonable performance on previous toy datasets, but now I have another toy dataset for which I can get 100% accuracy with logistic regression, but only about 67% accuracy with a neural network. I don't know whether the issue is more a matter of statistics or of programming, but I've adjusted lots of little things like the network structure and the batch size and I haven't made much progress. I can at least tell that I have an issue of underfitting rather than overfitting, because the training error is no better than the test error. Why is the model doing so badly?

The dataset consists of 100 cases, each with 3 binary predictors. (These values are actually real data from the aforementioned psychology experiment.) The outcome is just a majority vote of the predictors. I assess models with tenfold cross-validation.

All the code is in Python. The logistic-regression model is fit like this:

import sklearn.linear_model

model = sklearn.linear_model.LogisticRegression(
    penalty = "none", solver = "lbfgs")
model.fit(X[folds != fold_i], y[folds != fold_i])

The neural network is fit like this:

import os; os.environ["TF_CPP_MIN_LOG_LEVEL"] = "3"
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense, Activation

model = Sequential([
    Dense(10, activation = "relu"),
    Dense(1, activation = "softmax")])
model.compile(
    optimizer = "rmsprop",
    loss = "binary_crossentropy")
model.fit(
    X[folds != fold_i],
    y[folds != fold_i],
    verbose = False,
    epochs = 100,
    batch_size = np.sum(folds != fold_i) // 5 + 1)

print("Training accuracy:", np.mean(
   y[folds != fold_i] == model.predict(X[folds != fold_i])[:,0]))

Here's the overall program, including the data. Replace ... with the model-fitting code of your choice.

import numpy as np

X = np.array([[c == '1' for c in line] for line in [
    '1101111111011111111111111111111100011010111110010011111111111111011111110111111111111101111111111110',
    '0011111001000001011010001011000100010001110011011101111111110000111111001001001001110011011101111000',
    '0111101111010000000000111110001101000010110000101111010111100110010110000110011000001010000000000110']])
X = X.T
n_cases = X.shape[0]

y = np.sum(X, axis = 1) > 1

np.random.seed(123)
n_folds = 10
folds = np.repeat(range(n_folds), np.ceil(n_cases / n_folds))[
    range(n_cases)]
np.random.shuffle(folds)

y_pred = np.zeros_like(y)

for fold_i in range(n_folds):
    print("Fold", fold_i)

    ...

    y_pred[folds == fold_i] = model.predict(X[folds == fold_i])[:,0]

print(np.mean(y_pred == y)) 
$\endgroup$
1
  • 6
    $\begingroup$ You are using a softmax activation for the final layer, which is meant for multi-class classification, not binary classification. In multi-class classification, the probabilities of all the classes must sum to one for each prediction. I believe the model thinks you are doing a multi-class classification over one class, so it's just predicting $1$ for all samples. Just type print(model.predict(X)) to verify this. If it's all ones, swap "softmax" with "sigmoid". $\endgroup$ Jan 13, 2020 at 20:51

1 Answer 1

7
$\begingroup$

Bridgeburners's comment is correct: in this model, softmax always predicts 1 (because there's only one output node). I changed to sigmoid activation as I had used originally, then found another problem: under sigmoid, model.predict returns probabilities, not classes, so the predictions need to thresholded first. With these bugs fixed, and epochs cranked up to 300, I'm now getting 99% accuracy.

import numpy as np

X = np.array([[c == '1' for c in line] for line in [
    '1101111111011111111111111111111100011010111110010011111111111111011111110111111111111101111111111110',
    '0011111001000001011010001011000100010001110011011101111111110000111111001001001001110011011101111000',
    '0111101111010000000000111110001101000010110000101111010111100110010110000110011000001010000000000110']])
X = X.T
n_cases = X.shape[0]

y = np.sum(X, axis = 1) > 1

np.random.seed(123)
n_folds = 10
folds = np.repeat(range(n_folds), np.ceil(n_cases / n_folds))[
    range(n_cases)]
np.random.shuffle(folds)

y_pred = np.zeros_like(y)

for fold_i in range(n_folds):
    print("Fold", fold_i)

    import os; os.environ["TF_CPP_MIN_LOG_LEVEL"] = "3"
    from tensorflow.keras.models import Sequential
    from tensorflow.keras.layers import Dense, Activation

    model = Sequential([
        Dense(10, activation = "relu"),
        Dense(1, activation = "sigmoid")])
    model.compile(
        optimizer = "rmsprop",
        loss = "binary_crossentropy")
    model.fit(
        X[folds != fold_i],
        y[folds != fold_i],
        verbose = False,
        epochs = 300,
        batch_size = np.sum(folds != fold_i) // 5 + 1)

    print("Training accuracy:", np.mean(
       y[folds != fold_i] == (model.predict(X[folds != fold_i])[:,0] > .5)))

    y_pred[folds == fold_i] = (model.predict(X[folds == fold_i])[:,0] > .5)

print(np.mean(y_pred == y))
$\endgroup$
0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.