1
$\begingroup$

I have a question on how to use ML (machine learning) methods for the following task. I have a m-dimensional (field is always over real numbers $R$) input vector $\vec{x}$, and I want to learn the output function $f$ which is associated with the input vector as follows:

$\vec{x} = (x_1, ..., x_m)^T \in R^m \rightarrow f_{\vec{x}}(\cdot) $.

$f_{\vec{x}}$ takes a scalar as input (from a finite range) and outputs a scalar (for notational convenience, I'll omit the subscript):

$f:[a,b] \rightarrow R,$

$\zeta \mapsto f(\zeta)$

So my question is, how can I do this in the best way possible? I'd like to use (possibly deep) neural networks, but I am also open for other architectures. My first approach was to use a fully-connected neural net with ReLU activation function, and simply discretize uniformly $f(\zeta) \approx f_i := f(\zeta_i)$, but that did not bring good results. Presumably, due to the following reasons:

  • The output vector $\vec{y} = (y_1, ..., y_i, ..., y_n)^T := (f_1, ...,f_i, ..., f_n)^T$ becomes very high-dimensional if I want to have a good resolution, but this increases the complexity of the task.
  • Neighboring values (e.g. $f_i$ and $f_{i+1}$, and also $f_i$ and $f_{i+2}$, but probably not $f_i$ and $f_{i+100}$) are strongly correlated, and I do not utilize this "correlation length" in my architecture.

Thus, I was thinking that I should rather learn the coefficients in a different space. I anticipate that e.g. Fourier space would work well, since the basis is orthonormal (thus not correlated) and I can specify my domain knowledge (highest harmonic order / correlation length).

Would a different approach, e.g. a different representation or a different network architecture, work better? Or should I use something else than (standard feed-forward) neural networks? I'd be very happy to hear your feedback. Thank you.

Edit:

Since my original question unfortunately caused confusion, I want to point out the following:

  • My question was also of conceptual nature: Is there a representation for functions within the ML community, which is an established "standard" choice (as already mentioned, I assume that all orthogonal bases work particularly well, but maybe some better than others)? However, I also asked if there is possibly a particular network architecture or other ML approach which is predestined for this task, since I anticipate that this is case is a standard problem and I simply did not stumble across it in my research.
  • The notation $f_{\vec{x}}(\cdot)$ describes a function associated with the vector $\vec{x}$ (of a variable which I called $\zeta$).
  • From a practical perspective:
    • $m$ is typically between 1 and 10 in my application, so rather low-dimensional
    • If I discretize $f(\cdot)$ uniformly, then $n$ will be rather high, typically $n>>100$
    • I know that $f(\cdot)$ has some structure / correlation length $\ell$, but I do not which exact value (I could estimate an upper boundary though). On the other hand, this means I know that $f(\cdot)$ is not, e.g., white noise.

As a toy example and for demonstrational purposes (to show you, that a naive approach does not work), I did the following:

$m:=1$, thus input is just a scalar $x_1$.

Desired output to learn is a shifted unit box: $f_{x_1}(\zeta) = \begin{cases} 1 &\mbox{if } \lvert \zeta - x_1 \rvert < 1/2 \\ 0 & \mbox{else} \end{cases}$, with $\zeta \in [a, b] = [-10, 10]$. This results in the following function which I'd like to learn:

Shifted unit box ground truth

I discretized $f_{x_1}(\cdot)$ into $n=1001$ values ($\zeta = -10, -9.98, -9.96, ..., 10$), and I generated 101 data records ($x_1 = -5, -4.9, ..., 5$), from which I chose 50 random samples for training. Furthermore, I used several hyperparameter configurations, but in this run specifically I used 2 hidden layers with width = 10 and ReLU activation function.

Shifted unit box prediction

As you can see, the network learns something similar, but it has terrible accuracy considering that the output function is trivial. A different choice of the hyperparameters does not increase accuracy significantly. Of course, it also helped that the sampling was extremely narrow (this is not possible in my application), and that the input dimensionality was the lowest possible.

I hope that this clarifies my question and demonstrates my motivation for a different approach for this type of problem.

$\endgroup$
7
  • $\begingroup$ Could you give more details? This sounds like a generic description of any machine learning problem that can be solved using any algorithm. $\endgroup$ – Tim Feb 12 at 21:47
  • $\begingroup$ I don't understand the setup. Just add an extra dimension to your X,z, and build a model to predict the output $f_x(z)$ $\endgroup$ – seanv507 Feb 13 at 0:35
  • $\begingroup$ @Tim Ok, I will try to give more information. $\endgroup$ – user311204 Feb 13 at 10:27
  • $\begingroup$ @seanv507 This does not appear applicable. E.g., what would be the value of $z$ for each input vector $\vec{x}$ (I guess you mean that I should include it in the training process)? Also, if I specify $z$, how should this be the independent variable in my function? This would imply that I learn a function $f_{\vec{x},z}(\zeta)$ $\endgroup$ – user311204 Feb 13 at 10:30
  • $\begingroup$ You have multiple values of z for each x,. And yes you would learn f(x,z)(z) $\endgroup$ – seanv507 Feb 14 at 7:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy