Let $\mathbb{E}_x[g(X_t)]$ be the expected value of a random variable $X_t$ with known probability density $f_t(x)$ then for the continuous case

$$\mathbb{E}[g(X_t)] = \int_{-\infty}^{\infty} g(x)f_t(x)dx$$

where $g(x)$ or the distribution of $g(x)$ is not known. Hence, one possible pair of the training set in a supervised setting would be $(f_t(x), \mathbb{E}[g(X_t)])$ evaluated at a fixed time $t$. Note that for most distribution the useful bounds are finite.

Can neural networks learn the function $g(x)$ or its probability distribution?

Reference: Law of the unconscious statistician

I have added a time dependency but the problem is fundamentally the same. Please explain in as much detail as possible. This seems extremely useful if possible.

*images removed

  • 1
    $\begingroup$ I'm completely confused. What is $p_t(x)$? You mentioned $f_t(x)$ once but then you never use it... What is that that you observe in each data point? Thanks. $\endgroup$ – stans Aug 19 '18 at 19:47
  • $\begingroup$ @stans I apologize I had a typo. You observe the expected value and the distribution of $X_t$ $\endgroup$ – Edv Beq Aug 19 '18 at 19:52
  • $\begingroup$ Thank you for the clarification... My solution is very raw, pretty formal. But I had to post it as a solution, not a comment, because it is too long. $\endgroup$ – stans Aug 19 '18 at 20:07
  • $\begingroup$ Is the problem well-defined? There may be infinitely many $g(x)$ that have the same expectation under $f$. $\endgroup$ – Moss Murderer Aug 19 '18 at 20:25
  • $\begingroup$ @MossMurderer We only care to find one of them. $\endgroup$ – Edv Beq Aug 19 '18 at 20:27

Thanks. This is an interesting problem......

Formally, speaking, the problem can be solved with neural networks if we discretize the range of the data into M bins. Then every input is a 1-by-M vector. In the intermediate layers, we build the NN in the usual way. The second to last layer has M neurons, each implementing $g(x_i)$. The last layer is non-typical but perfectly allowed. Not only does it connect to the previous layer but also to the input nodes. It takes all of information available and calculates

$ \sum g(x_i)f(x_i)\Delta_i $

This is a roundabout approach though. I'm sure better approaches exist, after some thinking.

  • $\begingroup$ Can you please explain in your answer what do you mean by 1. "each implementing $g(x)$" as the function is not known, and 2. what is "non-typical"? Any references would also be tremendous $\endgroup$ – Edv Beq Aug 19 '18 at 20:12
  • $\begingroup$ @Edv Beq, sure $g()$ is unknown. But that is what neural networks do. Each neuron implements something which is unknown, through a combination of prespecified activation functions and unknown weights $W$ in the previous layers. Weights $W$ are estimated from the data using, for example, backpropagation. $\endgroup$ – stans Aug 19 '18 at 20:17
  • $\begingroup$ By "non-typical" I mean that, usually, intermediate layers do not connect to the input layer. This is pretty informal. $\endgroup$ – stans Aug 19 '18 at 20:18
  • $\begingroup$ Is the input layer fully connected to the second to last layer with weights in between - meaning each input node is connected to all second to last layer nodes with a weight in between? Or, are we just scaling the second to last layer with the corresponding input node? $\endgroup$ – Edv Beq Aug 19 '18 at 21:14
  • $\begingroup$ Suppose the architecture is: { Inputs ---> Layer $1$ ---> .... ---> Layer $K-1$ ---> Layer $K$ ---> Output }. The inputs are connected to layers $1$ and $K$ only. Generally speaking, they do not have to be connected to any other intermediate layers... Neuron $i$ in layer $K - 1$ implements $g(x_i)$... Weights $W$ entering each neuron in each of the layers $1$ through $K-1$ are unknown and are estimated in the usual way (e.g. backpropagation)... The only neuron in layer $K$ implements $\sum g(x_i) f(x_i) \Delta_i$. The formula is perfectly clear and does not require any extra weights. $\endgroup$ – stans Aug 19 '18 at 21:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.