I want to construct a model like this:

$$y = g(X, Z)$$ $$X = \alpha_1 x_1 + \alpha_2 x_2 + \dots + \alpha_n x_n$$ $$Z = \beta_1 z_1 + \beta_2 z_2 + \dots + \beta_n z_n$$

Where $x_i \text{ and } z_i$ are input features and $\alpha_i \text{ and } \beta_i$ are coefficients to be estimated. $g()$ is a non-linear function to be learned.

Some motivations behind my design of the model:

  1. I have observed that the target (say sales) could be impacted by several "dimensions" such as weather, income, etc. The relationship however, cannot be captured by simple linear model. Let's say Gradient Boosting is a viable option for non-linear modelling here.
  2. In addition, for each dimension, I have input from different regions (Temperature of New York City, LA, etc) and that's why I have not just one $x$ but $x_1, x_2, \dots, x_n$. I believe that I should aggregate them linearly because how temperature impacts the sales should be consistent across regions.
  3. The challenge is that I have no idea beforehand what weights ($\alpha_1, \alpha_2, \dots, \alpha_n$) would be ideal to use.

I hope there is a way that I can jointly learn the non-linear transformation as well as the coefficients. However the only way I can think about is neural network but I'm afraid with very small sample size (less than 50 data points) I cannot afford to do NN. I tried but the performance of the model would fluctuate violently with random initialization and it definitely cannot outperform my benchmark which is gradient boosting with only takes $x_1 \text{ and } z_1$.

Am I missing anything here. Sorry for being verbose. Many thanks in advance!

  • $\begingroup$ check out generalised additive models $\endgroup$ – Alex Oct 18 '19 at 2:44
  • $\begingroup$ @Alex, Thanks for your comments. I will give it a try! $\endgroup$ – Tlin Oct 18 '19 at 17:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.