My overall question is: the universal approximation theorems can provide a good heuristics on defining the loss function for supervised regression problems, i.e., because universal approximation functions (say, F(W, x)) in theory can approximate any ground truth function (say, f(x)) with arbitrary precision, in practice when we have finitely many (N) data, we can aim to minimize the empirical risk

$$\sum_{i=1}^{N} (F(W, x_i) - f(x_i))^2$$

with respect to the parameters W's. The above loss function is not even strictly dependent on the universal approximation theorems, but is used to determine coefficients for any model function from the available data.

When it comes to classification tasks, we usually minimize the cross-entropy loss function. I have seen derivations of the cross-entropy loss function starting from two probability distributions. But eventually we are still trying to approximate some underlying ground truth function. Can you provide any derivation/explanation of the cross-entropy loss from the function approximation point of view?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.