# Can neural networks learn $g(x)$ from $\mathbb{E}[g(X_t)] = \int_{-\infty}^{\infty} g(x)p_t(x)dx$

Let $\mathbb{E}_x[g(X_t)]$ be the expected value of a random variable $X_t$ with known probability density $f_t(x)$ then for the continuous case

$$\mathbb{E}[g(X_t)] = \int_{-\infty}^{\infty} g(x)f_t(x)dx$$

where $g(x)$ or the distribution of $g(x)$ is not known. Hence, one possible pair of the training set in a supervised setting would be $(f_t(x), \mathbb{E}[g(X_t)])$ evaluated at a fixed time $t$. Note that for most distribution the useful bounds are finite.

Can neural networks learn the function $g(x)$ or its probability distribution?

Reference: Law of the unconscious statistician

I have added a time dependency but the problem is fundamentally the same. Please explain in as much detail as possible. This seems extremely useful if possible.

*images removed

• I'm completely confused. What is $p_t(x)$? You mentioned $f_t(x)$ once but then you never use it... What is that that you observe in each data point? Thanks. – stans Aug 19 '18 at 19:47
• @stans I apologize I had a typo. You observe the expected value and the distribution of $X_t$ – Edv Beq Aug 19 '18 at 19:52
• Thank you for the clarification... My solution is very raw, pretty formal. But I had to post it as a solution, not a comment, because it is too long. – stans Aug 19 '18 at 20:07
• Is the problem well-defined? There may be infinitely many $g(x)$ that have the same expectation under $f$. – Moss Murderer Aug 19 '18 at 20:25
• @MossMurderer We only care to find one of them. – Edv Beq Aug 19 '18 at 20:27

Thanks. This is an interesting problem......

Formally, speaking, the problem can be solved with neural networks if we discretize the range of the data into M bins. Then every input is a 1-by-M vector. In the intermediate layers, we build the NN in the usual way. The second to last layer has M neurons, each implementing $g(x_i)$. The last layer is non-typical but perfectly allowed. Not only does it connect to the previous layer but also to the input nodes. It takes all of information available and calculates

$\sum g(x_i)f(x_i)\Delta_i$

This is a roundabout approach though. I'm sure better approaches exist, after some thinking.

• Can you please explain in your answer what do you mean by 1. "each implementing $g(x)$" as the function is not known, and 2. what is "non-typical"? Any references would also be tremendous – Edv Beq Aug 19 '18 at 20:12
• @Edv Beq, sure $g()$ is unknown. But that is what neural networks do. Each neuron implements something which is unknown, through a combination of prespecified activation functions and unknown weights $W$ in the previous layers. Weights $W$ are estimated from the data using, for example, backpropagation. – stans Aug 19 '18 at 20:17
• By "non-typical" I mean that, usually, intermediate layers do not connect to the input layer. This is pretty informal. – stans Aug 19 '18 at 20:18
• Is the input layer fully connected to the second to last layer with weights in between - meaning each input node is connected to all second to last layer nodes with a weight in between? Or, are we just scaling the second to last layer with the corresponding input node? – Edv Beq Aug 19 '18 at 21:14
• Suppose the architecture is: { Inputs ---> Layer $1$ ---> .... ---> Layer $K-1$ ---> Layer $K$ ---> Output }. The inputs are connected to layers $1$ and $K$ only. Generally speaking, they do not have to be connected to any other intermediate layers... Neuron $i$ in layer $K - 1$ implements $g(x_i)$... Weights $W$ entering each neuron in each of the layers $1$ through $K-1$ are unknown and are estimated in the usual way (e.g. backpropagation)... The only neuron in layer $K$ implements $\sum g(x_i) f(x_i) \Delta_i$. The formula is perfectly clear and does not require any extra weights. – stans Aug 19 '18 at 21:27