Multilayer Perceptron | Likelihood of parameters I am referring to a simple multiplayer perceptron (e.g. only one hidden layer and one output layer).
Notation:
Assume we have $K$ dimensional output. The number of samples we have is $N$. And we consider a classification problem, i.e. the output is one-hot encoded.
Let $\hat y$ be the output determined by our network i.e. $\hat y = (\hat y_1, ..., \hat y_k)$. Let $y$ represent the actual class label (also one-hot encoded).
In a script, the following is stated:
Given $K$-dimensional output, our likelihood is given by:
$$ l(\theta) = \sum_1^N \sum_1^K (\hat y_{ik} (\theta) - y_{ik})^2$$
My question:
Why is this the likelihood? For me, it looks more like an empirical cost function. I know what a likelihood in the sense of statistics is (https://en.wikipedia.org/wiki/Likelihood_function) and I am used to Maximum Likelihood estimation. But the previous stated likelihood $l(\theta)$ is for me not a likelihood... or am I missing something?
 A: Consider the simpler case, where you have one target, $y$. The input-output relationship in neural networks is, in general,
$$y=f(\theta, x)+\epsilon$$
where, $y$ is the target, $x$ is the feature vector, $\theta$ is the set of parameters, and $\epsilon$ is the random error. It's typical to assume that the random error is distributed normally with zero-mean and variance $\sigma^2$ for some $\sigma>0$. This means the output variable is also normally distributed: $$y|x,\theta \sim \mathcal N(f(x,\theta),\sigma^2)$$
The likelihood of $\theta$ will be
$$\mathcal{L}(\theta)=\prod_{i=1}^N p(y_i|x_i,\theta)\propto \exp\left(-\sum_{i=1}^N\frac{(y_i-f(x_i,\theta))^2}{2\sigma^2}\right)$$
This expression is to be maximized. Typically, we take negative log-likelihood and minimize it:
$$\text{NLL}=\frac{1}{2\sigma^2}\sum_{i=1}^N (y_i-f(x_i,\theta))^2\propto \sum_{i=1}^N (y_i-f(x_i,\theta))^2$$
which is MSE (ignored $\sigma$ because it doesn't affect the optimisation).
In case of $K$ targets, the equations will be in the form of multivariate normals under some independence assumptions, i.e. you assume $y_{ik}$ and $y_{il}$ are independent given $x,\theta$. This still makes sense, because if you have the data sample, $x$ and the parameters, other neurons' outputs don't give you extra information. This converts our formula to:
$$\text{NLL}\propto\sum_{i=1}^N\sum_{k=1}^K (y_{ik}-f(x_i,\theta)_k)^2$$
In multi class classification problems, it's typical to use cross-entropy loss function (instead of MSE) together with a softmax layer in the end, where the above arguments may slightly change.
