I had a interview on deep learning internship job yesterday, and the interviewer showed a loss function formula(below) and asked me 2 questions:

\begin{equation} J(\theta)=-\mathbb{E}_{x,y\sim\hat{p}_{data}}\log{p_{model}(y|x)} \end{equation}

  1. How to understand $p(y|x)$ in the formula above?
  2. On what task does this loss function used for?

Since I have no idea on this loss function, how to answer these 2 questions?

  • $\begingroup$ Not exactly the same, but I think my answer should be good for you. stats.stackexchange.com/questions/241380/… $\endgroup$
    – Haitao Du
    Dec 26, 2016 at 4:13
  • 1
    $\begingroup$ As reproduced the loss function is meaningless since there is no $\theta$ on the right hand side. $\endgroup$
    – Xi'an
    Dec 26, 2016 at 10:24

1 Answer 1

  1. Since $J(\theta)$ is a function of $\theta$, $p_{model}(y|x)$ should be understood as the likelihood function of $\theta$, where $\theta$ is the model parameters, IMO.

  2. Therefore this loss is used for maximum likelihood estimation (MLE).

Say we have a neural network that takes in $x$ and outputs $f_\theta(x)$, and we assume a Gaussian noise for $y\sim N(f_\theta(x), \sigma^2)$, then $p_{model}(y|x)=\frac{1}{\sqrt{2\sigma^2\pi}}\exp(-\frac{(y-f_\theta(x))^2}{2\sigma^2})$, then $$J(\theta)=\frac{1}{2\sigma^2}\frac{1}{m}\sum (y-f_\theta(x))^2+constant$$ minimizing $J(\theta)$ in this case equals to minimizing our favorite mean-squared-error.

  • $\begingroup$ Where is $\theta$ in your likelihood? $\endgroup$
    – Xi'an
    Dec 26, 2016 at 10:25
  • $\begingroup$ @Xi'an right, maybe it's left out intentionally for the question? $\endgroup$
    – dontloo
    Dec 26, 2016 at 11:06
  • $\begingroup$ A question that is intentionally making no mathematical sense?! In your answer, would you consider $\theta$ as implicitly contained in $p(y|x)$ or anything else? For instance, what does $\theta$ stand for in the Normal example? $\endgroup$
    – Xi'an
    Dec 26, 2016 at 11:14
  • $\begingroup$ @Xi'an ... oops I wasn't prepared for all these questions, but surely I can improve the notation a bit. $\endgroup$
    – dontloo
    Dec 26, 2016 at 11:22

Not the answer you're looking for? Browse other questions tagged or ask your own question.