Suppose you had a classification problem where you are trying to predict a class label (e.g., $[0 \: 1 \: 0]^T$) with a model. One way to do this is to use log loss:
$\Large \ell_{\log} = -\sum_i[y_i\log \hat{y}_i + (1-y_i)\log (1-\hat{y}_i)]$
This is attractive because it does the right thing: it pushes $\hat{y}$ to $\inf$ when $y_i$ is $1$, and to $- \inf$ when $y_i$ is zero. But another way to do this is with elementwise division:
$\Large \ell_{\text{div}} = \sum_i[\frac{\hat{y}_i}{\max (y_i, \epsilon)} + \frac{y_i}{\max (\hat{y_i}, \epsilon)}]$
Here, the minimum of the function is attained when $y$ matches $x$ on all dimensions. Isn't this a preferable cost function? Why isn't it used?
Note: $\epsilon$ is a small positive constant to prevent division by zero.
EDIT: My question isn't only about $0$ vs $1$ classification. I am also interested in situations where $y$ has real-valued entries, and we are interested in generating a vector $\hat{y}$ that matches it, in the sense that the two are "close together" in some sense. For classification problems, $\log$ is the way to go; but what about real-valued vectors?
