3
$\begingroup$

I have been googling a lot and somehow cannot find a good answer.

Lets say we have deep neural net model, arbitrary topology. Also we have 10 features and for each features we got 1000 observations for each time step.

What is the best way to pass the data into our network? Perhaps some initial central moments, like mean, variance, skewness, kurtozis...? Or should we find the distribution that best explains our features (relying somehow on expert judgment), like lognormal and pass only the two parameters obtained with MLE?

Any ideas or experiences?

EDIT (on answer):

The goal is binary classification.

If we input whole 10*1000 as features we obtain the curse of dimensioanlity. Maybe I wasn't clear. We have 10 000 different observations at each time step, while we know that they come from 10 separate distributions. Learning a neural net with 10K input features is at least for my hardware currently impossible.

We obtain those samples at each time step as an output of some meta model, and we have 10 different meta models. Sorry for unclear question before.

SECOND EDIT:

Perhaps we can teach 10 variational autoencoders, that will map our distributions to lets say 3 parameters and then manage to decode the information and we minimize KL loss. Has anyone tried that with unknown distributions?

Best JJ

Improve this question
$\endgroup$
7
  • 1
    $\begingroup$ what are you trying to achieve as an end-goal? $\endgroup$
    – user3235916
    Commented Jul 17, 2019 at 7:57
  • $\begingroup$ @user3235916 I have edited the question. $\endgroup$
    – JacobJacox
    Commented Jul 17, 2019 at 10:21
  • $\begingroup$ Ah. I think you might get better answers if you could explain a little more what you are trying to do. I have a hard time understanding why you would run ten different meta models that each output 1000 data points, which you then want to combine into a binary classification. Why do you do this? What do the meta models do? Why do you think this pipeline yields better classifications than a simpler model? $\endgroup$
    – Stephan Kolassa
    Commented Jul 17, 2019 at 10:41
  • $\begingroup$ @StephanKolassa Yeah sorry. Meta models are very complex by themselves, but basically each operate on specific field (some with text, some with one hot encoded data, some with numeric features) and return possible numeric outputs. Those outputs form distribution of possible changes in target variable. $\endgroup$
    – JacobJacox
    Commented Jul 17, 2019 at 10:53
  • $\begingroup$ @StephanKolassa One way for prediction would be to calculate mean and just count how many are >0 and we could sell that as probabilities of classification. But I am sure that is not an optimal solution right... perhaps you can tell more if you understand your distributions better... That's why I want to feed them somehow in a model. $\endgroup$
    – JacobJacox
    Commented Jul 17, 2019 at 10:53

2 Answers 2

Reset to default
1
$\begingroup$

I think the approaches you have suggested are all good ideas (calculating moments, seeing if the outputs of the meta models follow standard distributions etc). I would try approximating the outputs of the meta-models with simple distributions first (e.g. normal, exponential...) The problem with this as you have identified is that they may not capture the structure of the output of the meta models very well.

If this is the case, you could take a more non-parametric approach. You could feed the values at certain percentiles of the output of your meta-models in and do this for each meta-model. Assuming they are continuously distributed, you could for example feed the values at each 2% percentile into your data. This way you could reduce the dimensionality of of the output of each meta-models to 50, but still capture the shape of the distribution of outputs. You can obviously smooth the distribution over meta-model outputs prior to doing this etc etc.

Improve this answer
$\endgroup$
3
  • $\begingroup$ I like it, what abount second edit? $\endgroup$
    – JacobJacox
    Commented Jul 17, 2019 at 16:21
  • $\begingroup$ its not clear to me how a VAE is the easiest or best first choice here. I would always start simple. How doe VAEs get around the problem of training a large number of parameters? They'll also need lots of required cross validation for choosing the dimensionality of the latent space, VAE architecture etc. How well VAEs perform will also really depends on how the outputs of the meta models are distributed and what parts and structure of that distribution are important. For example, if you are using an MSE metric to optimise VAEs and and the tails are important you could run into trouble... $\endgroup$
    – user3235916
    Commented Jul 18, 2019 at 11:36
  • $\begingroup$ definetly not the easiest but it removes this prior knowledge problem. I would use KL loss.. so you would sample from decoding part with learned parameters then minimize the difference between prior and sampled? $\endgroup$
    – JacobJacox
    Commented Jul 18, 2019 at 11:46
3
$\begingroup$

Feed in the observations themselves.

Feeding in the distribution of your observations will not be helpful. What is it that drives your target variable? It's actual observations, not their distribution. If the value of feature A correlates with outcome X, then you want to detect this, and it doesn't matter (primarily) how prevalent A is, or what shape it has.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Thank you, but I wasn't clear enough. I cannot afford 10K features. I edited my question, to explain better what I wanted to ask. $\endgroup$
    – JacobJacox
    Commented Jul 17, 2019 at 10:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.