1
$\begingroup$

I am working on a regression problem to predict 3 outputs from 5 inputs, The inputs range from -30 to 30 except for one input that ranges from 20000 to -2e7. The 3 outputs range from 0 to 2e6, I am using Keras API and my network is simple 3 hidden layers (32169),

I am using leaky relu and Adam optimizer and training over 500 epochs with a batch size= 64. I use sklearn standardscaler() for standardizing the data.

My problem is that the network doesn't learn and the prediction I am getting are not accurate at all!! I tried complicating the network by adding layers and units but it doesn't work at all, I also tried using different normalization methods like minmax() and tanh estimator but no improvements were noticed!!

I tried many combinations of learning rates (0.1 to 0.000001) also epochs=(100 to 1000000), I tried changing batch size (10 to 256) no luck at all.

I tried different activation functions (relu,elu...etc) also tried different optimizers(RMSprop, SGD, adagrad, adam ...etc) no improvement at all!!!

My validation loss typically goes from around 1 to 0.3 and stops improving, I tried dividing the network into 3 networks where each predicts only one output but it didn't improve anything!!

this is my model: enter image description here

and this is my learning curve:

enter image description here

These are the output data distributions

enter image description here

and here are the input data distributions:

enter image description here

There is no relation between the inputs and the outputs!! Can anyone help me with this problem?! thank you!

Improve this question
$\endgroup$
8
  • 2
    $\begingroup$ Have you had success with simpler models like linear regression? Do you have reason to believe that your variables will be predictive of the outcome? $\endgroup$
    – Dave
    Commented Oct 29, 2020 at 10:30
  • $\begingroup$ nope! I tried simpler ones and no luck , the data I use is generated through simulations I basically use genetic algorithms to optimize a control system under variable conditions so the data itself is random, so the inputs and outputs are not related $\endgroup$
    – Yassine Kebbati
    Commented Oct 29, 2020 at 10:35
  • $\begingroup$ Please edit that important information into your original question. Not everyone reads comments. $\endgroup$
    – Dave
    Commented Oct 29, 2020 at 10:38
  • 1
    $\begingroup$ I’m super curious why you want to write a regression between independent variables, though. $\endgroup$
    – Dave
    Commented Oct 29, 2020 at 10:48
  • $\begingroup$ well I am trying to make an adaptive PID controller, where I have to adapt the three gains Kp, Ki, Kd based on the 5 inputs that affect the system to be controlled so I try many combinations of Kp, Ki, Kd using GA and optimize on that to create a dataset so that when applying the PID controller I can adapt the gains based those 5 inputs. So if there is no function that relates inputs to outputs the NN will not be able to learn right? $\endgroup$
    – Yassine Kebbati
    Commented Oct 29, 2020 at 10:59

2 Answers 2

Reset to default
1
$\begingroup$

Your learning curve looks good, mean squared error is decreasing for both test- and training set, and after a while the network has learned what it can apparently. "Not enough" is subjective, but this looks like it's learning, no obvious problems.

That said, I have the following suggestions

  • Scale your inputs to resemble a normal distribution. For input 1 to 3, the effect will be limited, you should just divide by standard deviation (be sure to estimate it from the test distribution). For the other two, apply a log-scaling first, then scale it.
  • Also scale the output variable using a logarithm and scale it
  • Your network seems a bit heavy, reduce the number of layers to two, and reduce the number of nodes in the hidden layers as well. For example, use

5 input - 5 hidden layer - 3 output 5 input - 3 hidden layer - 3 output

EDIT: if it is true that inputs and outputs are independent, this whole exercise is pointless. You cannot make good predictions if your inputs don't say anything about the outputs. Good on @Dave for catching that fundamental issue.

EDIT: Then the question is; how can the MSE still decrease? I'm guessing then this is because the network is learning the average outputs. If you initialize a network, especially with such unscaled predictors and outcomes, it will do much worse than predicting the average of the outputs. It will adapt the parameters so that it predicts the average for all outputs, which is the best you can do given independent (in other words, useless) information.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Yes, the inputs and outputs are independent, so I guess a NN won't help here, thank you for the answer. $\endgroup$
    – Yassine Kebbati
    Commented Oct 29, 2020 at 11:12
0
$\begingroup$

You said that the inputs and outputs are independent. In that case, you’re getting the right answer. You should not be able to use the inputs to predict the outputs.

That’s what independence means: knowing something about one gives no insight into the other.

Improve this answer
$\endgroup$
4
  • $\begingroup$ I mean there is no function that relates the inputs to the outputs, the outputs are PID gains, the inputs are specific to the system I am trying to control, so in other words when I use the outputs(gains) on the system with the corresponding inputs (system-specific), I obtain optimal results so I am trying to predict the best gains for varying system-specific parameters which are the inputs to my network. The is no direct relation $\endgroup$
    – Yassine Kebbati
    Commented Oct 29, 2020 at 11:06
  • $\begingroup$ Thank you for your answer! $\endgroup$
    – Yassine Kebbati
    Commented Oct 29, 2020 at 11:13
  • $\begingroup$ @YassineKebbati You’re welcome! The typical way to thank people on Stack Exchange is by upvoting an accepting answers, both of which guide other people with similar questions to answers that might be helpful. $\endgroup$
    – Dave
    Commented Oct 30, 2020 at 21:09
  • $\begingroup$ Yes I am new here lol I have done that but it won't show because I have less than 15 reputation !! $\endgroup$
    – Yassine Kebbati
    Commented Nov 1, 2020 at 14:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.