# Estimate a probability distribution of target values using features

In my particular problem, I have $$t \in \{1,...N\}$$ time periods, and feature vectors $$x_t \in R^m$$ which I hypothesize predict something about the probability distribution that the targets $$y_t \in R$$ come from:

$$y_t \sim P(x_t)$$

Where $$P$$ is the probability distribution that is (at least partly) determined by $$x_t$$.

For instance I could hypothesize that $$P$$ is the normal distribution, or some other parameterized distribution, where the variance, mean, and any other parameters that specify the distribution are determined by $$x_t$$.

The question is, how do I estimate what $$P(x_t)%$$ is?

I want a predictive model that estimates (out of sample) what $$P(x)$$ is.

My first idea was just to fit the parameterized probability distribution to the $$y_t$$ "near" a given point $$x$$ using maximum likelihood, and say that is my estimate of $$P(x)$$. However this seems primitive.

Any better ideas?

In my problem, $$m$$, the number of features, can be small, 3-5ish. Currently, these are the outputs of sophisticated quantile* estimators (estimators which themselves use dozens of features), so there is a lot of prior belief about how these should relate to the distribution.

*an $$\alpha$$ quantile estimate is $$\hat{y_t}(x_t)$$ such that $$P(y_t \leq \hat{y_t} | x_t) < \alpha \in [0,1]$$

Edit: Now I am fitting a 3-parameter distribution to the quantile estimates using least-squares criterion. Still open to other suggestions!

The conditional distribution of $$y_t$$ given $$x_t=x$$ is given by $$F(y_t\vert x_t)=P(y_t\le y\vert x_t=x)=\mathbb{E}(1_{\{y_t\le y\}}\vert x_t=x)$$ Therefore, you can estimate the conditional CDF using regression methods, with the caveat that $$1_{\{y_t\le y\}}$$ is a function of $$y$$ and therefore, estimation of the conditional CDF is essentially a set of regressions. Common choices for the estimation of the conditional CDF include Nadaraya-Watson, local linear estimation, or adaptive weighted Nadaraya-Watson.
• In other words, perform a set $i \in \{1,...I\}$ of classification problems each estimating $P(y_t \leq y_i | x=x_t)$. This is basically what I already have, except inverted - I have quantile estimates $\hat{y_t}(x_t) : P(y_t \leq \hat{y_t} | x_t) < \alpha \in [0,1]$. I estimate quantiles using "sophisticated" off-the-shelf ML models, but the outputs are noisy and I just consider them "features" for estimating the "true" distribution. Maybe I'll try your idea with features being these quantiles. Dec 6 '18 at 23:51