3
$\begingroup$

I am fairly new to neural networks. I am trying to empirically show that a neural network can work better than logistic regression when the underlying function is non-linear. In my simulation study, the true probability of assignment to the treatment group is a function of $X_1$, $X_2^2$, $X_3$, and $X_4$ - however, I am acting as if I don't know that $X_2$ is squared, and just using the non-squared term. Logistic regression performs poorly.

I think that I can get a neural network to estimate the function better since it's a (rather simple) non-linear function. However, my first attempt was not fruitful and I'm not sure if I need to increase "hidden," the number of hidden neurons, or change the activation function (or something else). Here's what I tried in R:

nn = neuralnet(t ~ X1+X2+X3+X4, data=df, hidden=3, act.fct = "logistic",
             linear.output = FALSE)

Any thoughts?

Thanks!

Improve this question
$\endgroup$
0

1 Answer 1

Reset to default
1
$\begingroup$

Yes, increasing the number of hidden neurons should improve the capacity of the model to learn the best function. There's a couple a things you could do. First, I would recommend using one of the more common tools for deep learning, like PyTorch. This will give you more control over what happens inside the neural network. Also, the deep learning performs best when you have multiple layers, as one of the points of deep learning is that the composition of functions leads to better representations of the data from which it can extract information from. Third, try using ReLu activations instead; these are more common in industry now because they give computationally easier gradients and reduces vanishing/exploding gradients, which works better for deeper layers. Try some regularization too, such as dropout layers.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thanks! So even in my simple situation where the underlying function is X1 + X2^2 + X3 + X4, would you recommend more layers and/or neurons? I have tried 3 neurons in the first layer, and 2 neurons in the second layer along with a logistic activation function, and I don't think it recognized the non-linearity very well. I'm a little wary of blindly increasing the complexity of the network when I only have four features. $\endgroup$
    – Alex
    Commented Jan 29, 2020 at 18:31
  • 1
    $\begingroup$ Don't be too wary. It's not unusual to see deep neural networks that have on the scale of thousands to millions of parameters. The full benefit of deep learning though does require multiple layer, so just try 5-7 layers of like 10-20 neurons each just to see what happens. Just make sure that your dataset is sufficiently large. Also, use Rectified Linear activations instead of sigmoid when you have deeper networks because those make gradient calculations easier. $\endgroup$
    – Anon
    Commented Jan 30, 2020 at 4:55

Not the answer you're looking for? Browse other questions tagged or ask your own question.