2
$\begingroup$

I have a dataset of multiple continuous variables associated with a target value. I would like to estimate the probability distribution of the output given new data. (I have around 50-120 features with 2.5 million samples.)

In an MLP classification approach, I would bin the output into 1000 bins(classes) then train it with the data and obtain the probability for each class. If I go this way, what hyperparameters (loss function, activation function, layers) are suitable for this task?

If an MLP classifier isn't a good idea, what other approaches can I use for probability distribution estimation?

$\endgroup$
6
  • $\begingroup$ Welcome to Cross Validated! I don’t really understand what you’re trying to do, but it’s generalized advised go to bin continuous variables. Why not work with the continuum? $\endgroup$
    – Dave
    Aug 20, 2022 at 17:06
  • $\begingroup$ @Apple Your autocorrect could be improved… $\endgroup$
    – Dave
    Aug 20, 2022 at 18:37
  • $\begingroup$ What do you mean by "the probability distribution of the output given new data"? $\endgroup$
    – dipetkov
    Aug 20, 2022 at 20:39
  • $\begingroup$ @Dave Thanks for the reply! Yes I'm trying to bin my continuous output and treat it as a classification problem. I'm not sure what you mean by working with the continuum tho. Do you mean like first assuming the type of distribution and estimate it's parameters? (eg. Mean and standard deviation for a normal distribution) $\endgroup$
    – Andrew
    Aug 21, 2022 at 3:27
  • 1
    $\begingroup$ @dipetkov Apologies for my wording, I'll try my best. By new data I mean data that isn't used during the training of the network. For example if I have the data of turtle sizes with their speed, I would like to estimate the probability distribution of the speed of the turtle (probabilities of the turtle moving at different speed) given it's size. Then I can apply the same model to a new turtle to find out it's possible speeds given it's size $\endgroup$
    – Andrew
    Aug 21, 2022 at 3:27

1 Answer 1

2
$\begingroup$

You explain your objective better in a comment than in the body of the question:

For example if I have the data of turtle sizes with their speed, I would like to estimate the probability distribution of the speed of the turtle (probabilities of the turtle moving at different speed) given its size. Then I can apply the same model to a new turtle to find out its possible speeds given its size.

The turtle analogy is fun, so I take it at face value in my answer. In your actual application you have 50+ features which only exacerbates the problems with your approach.


Effectively you propose to discretize the conditional distribution of speed given size into a histogram. As @Dave points out, this results in a huge loss of information: 1) to cut a continuous variable into intervals (bins) assumes that there is no meaningful variability inside each bin and a discontinuous jump between bins; 2) to treat the histogram bins as classes ignores their natural ordering.

Not discussed in your proposal but implied nevertheless is the fact that you would have to discretize size as well. I assume that size is a continuous variable*, so it's unlikely that there are two or more turtles with exactly the same size in your data. Another round of information loss.

Choosing a complex model — Multi-layer Perceptron classifier — will not help you recover the information you lose from ironing out the details in your data.

You could estimate the distribution of speeds given weight with quantile regression, by fitting a model for a choice of quantiles, eg., {0.1, 0.25, 0.5, 0.75, 0.9}. The more quantiles, the finer approximation to the conditional distribution of speed given weight. Keep in mind that it is more difficult to estimate the tails of a (unimodal) distribution than its center. This means you'll need to observe a lot of turtles.

Or you could specify the form of $f_\theta(\text{speed} | \text{size})$ as a function of a few parameters $\theta$ and optimize.

The best approach, however, is to collect more data. There is unexplained variability in speed after conditioning of size. What additional information can help model this variability: turtle species? gender? temperature? terrain (if turtles are observed in the wild) or stimuli (if turtles are observed in a lab), ...

Once you have more features that are predictive of turtle speed, you can build a more sophisticated model (withou discretization) $\text{speed} \sim f(\text{size},x_2,\ldots,x_k)$. Then you can get an idea of the distribution of speeds given size by marginalizing out predictors $x_2,\ldots,x_k$.

٭ Instructions for Taking Turtle Measurements. Alabama Outdoor Classroom Box Turtle Research Program by the Alabama Wildlife Federation.

$\endgroup$
3
  • $\begingroup$ I will not be binning size since using an MLP allows continuous variables to be used as inputs. Still, I am now made aware that binning the output would result in information loss. Would you say that increasing the number of bins, or smoothing the distribution using something like kernel density estimation can solve the problem? I am going to check out the two other methods you mentioned: quantile regression and function estimation. Thanks a lot! $\endgroup$
    – Andrew
    Aug 21, 2022 at 10:32
  • 1
    $\begingroup$ MLP allows for continuous variables. The problem is that for any one observed combination of your 50+ predictors you'll observe one data point (most likely). So the classifier won't be able to estimate meaningful (conditional) probabilities for the 1000 bins based one 1 observation in 1 out of 1000 bins. $\endgroup$
    – dipetkov
    Aug 21, 2022 at 10:36
  • 1
    $\begingroup$ You know that (most) supervised algorithms are deterministic: they make the same prediction given the same values for the predictors $x$: $f(y|x)$. You'd like to go around this fact by estimating a histogram for each combination of $x$. But if the conditional distribution $Y|X=x$ has signal not explained by the $x$s, you are missing important predictors (more likely) or you fitted a model that's too simple/has bias (unlikely with an MLP). $\endgroup$
    – dipetkov
    Aug 21, 2022 at 10:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.