3
$\begingroup$

I've heard neural networks be defined as computing a function but if the activation functions are already chosen for the hidden layers and output layers, and the cost function is also chosen, then isn't the overall function already determined and the neural network is just trying to find the parameters of that function that determine the best decision boundary?

In Goodfellow's DL book it says:

"If we use a sufficiently powerful neural network, we can think of the neuural network as being able to represent any function from a wide classs of functions, with this class being limited only by features such as continuity and boundedness rather than by having a specific parametric form. From this point of view we can view the cost function as being a functional rather than just a function."

Is what he means here that the function is essentially limitless and is so only really determined by its parameters, ergo, finding the parameters is defining the function?

$\endgroup$
1
$\begingroup$

You are on the right track: The network architecture determines the function between input and output up to its parameters. Since in deep learning, we are talking about millions of parameters, the function is so extremely flexible that it can well approximate any relationship between in- and output. As long as you got enough data to estimate all the parameters in a reliable way... but that is another story.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.