Can someone explain why neural networks are highly parameterized? I understand that neural networks by definition, are a parametric model. 
If I am correct, Parametric methods make an assumption about the functional form, or shape, of f. For a neural network, what is the assumption of its form? I have interpreted them (Albeit, only one introductory data mining course) as a Blackbox model. 
What further pushed me to ask this question is the greater necessity for a large sample size, which is consistent with a nonparametric method!
 A: Neural networks have their parameters (called weights in the Neural Network lingo) organized in matrices and vectors. This is not so strange, the parameters in a linear or logistic regression are placed in vectors, so this is just a generalization of how we store the parameters in simpler models.
Let's take a two layer neural network as a simple example, then we can call our matrices of weights $W_1$ and $W_2$, and our vectors of bias weights $b_1$ and $b_2$. To get predictions from out network we:


*

*Multiply our input data matrix by the first set of weights: $W_1 X$

*Add on a vector of weights (the first layer biases in the lingo): $W_1 X + b_1$

*Pass the results through a non-linear function $a$, the activation function for our layer: $a(W_1 X + b_1)$.

*Multiply the results by the matrix of weights in the second layer: $W_2 a(W_1 X + b_1)$

*Add the vector of biases for the second layer: $W_2 a(W_1 X + b_1) + b_2$

*This is our last layer, so we need predictions. This means passing this final result through an output function (often a soft-max to get some probabilities): $o( W_2 a(W_1 X + b_1) + b_2 )$
So, if you want a formula for the parametric form of a neural network, it's something like this:
$$ \hat y = o( W_2 a(W_1 X + b_1) + b_2 ) $$
If you really, really wanted to, you could unwrap all these matrix multiplications and write the whole thing in terms of the individual real-number weights. You'd end up with a bunch of summation signs, a whole lot of indices to keep track of, and the resulting formula would not be so illuminating or useful (besides proving that, yes, it can be done).
More complicated networks, with more layers, or more structure on their layers, lead to generalizations of this basic form, but the core idea is the same, and the formulas would be constructed is a similar manner.
