A neural network doesn't really care about the activation functions, and if we choose any activation function and a compatible loss, the model will converge into something. In a way, any model will be as good as another.

From this point of view, the only reason to pick a model over another would be the time spend to fit the data.

At the same time, say we have just one neuron, and we are fitting the classic of cat v non-cat with either linear or binary regression. But why would you choose any of them, if we can't really see the shape of the data, like we do in a plot?

Then, I assume the function itself is quite irrelevant.

I'm doing tests, but theoretically, would this be the same?

  • $\begingroup$ Yes, plenty of reasons. Related Is linear regression obsolete? $\endgroup$ Jan 15, 2021 at 13:21
  • $\begingroup$ I think you missed the word "theoretically". @user2974951 $\endgroup$
    – Minsky
    Jan 15, 2021 at 15:17
  • 1
    $\begingroup$ Maybe you are looking for the No free lunch theorem or Occam's razor. $\endgroup$ Jan 15, 2021 at 16:25
  • $\begingroup$ sure, but simplicity isn't always the same for all of us @displayname $\endgroup$
    – Minsky
    Jan 16, 2021 at 4:12

1 Answer 1


In a way, any model will be as good as another.

But models aren't as good as each other even when "kept the same".


  • Neural networks and some other learners give a different result each time with training.
  • Choice of network structure and activation functions can have a mathematically proven limitation on what functions the network is capable of learning.
  • Often you try an arbitrary bunch of models and just pick the best one. And the best one could have been the best one just because of blind luck. But the model still "matters" since it got the best result, unless you would say it doesn't matter whether you won lotto or not because every ticket, winning or not, just contains lines of numbers.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.