In my particular problem, I have $$t \in \{1,...N\}$$ time periods, and feature vectors $$x_t \in R^m $$ which I hypothesize predict something about the probability distribution that the targets $y_t \in R$ come from:

$$ y_t \sim P(x_t) $$

Where $P$ is the probability distribution that is (at least partly) determined by $x_t$.

For instance I could hypothesize that $P$ is the normal distribution, or some other parameterized distribution, where the variance, mean, and any other parameters that specify the distribution are determined by $x_t$.

The question is, how do I estimate what $P(x_t)%$ is?

I want a predictive model that estimates (out of sample) what $P(x)$ is.

My first idea was just to fit the parameterized probability distribution to the $y_t$ "near" a given point $x$ using maximum likelihood, and say that is my estimate of $P(x)$. However this seems primitive.

Any better ideas?

In my problem, $m$, the number of features, can be small, 3-5ish. Currently, these are the outputs of sophisticated quantile* estimators (estimators which themselves use dozens of features), so there is a lot of prior belief about how these should relate to the distribution.

*an $\alpha$ quantile estimate is $\hat{y_t}(x_t)$ such that $P(y_t \leq \hat{y_t} | x_t) < \alpha \in [0,1]$

Edit: Now I am fitting a 3-parameter distribution to the quantile estimates using least-squares criterion. Still open to other suggestions!


Here's my take - model this problem as a conditional probability function and estimate the conditional CDF.

The conditional distribution of $y_t$ given $x_t=x$ is given by $$ F(y_t\vert x_t)=P(y_t\le y\vert x_t=x)=\mathbb{E}(1_{\{y_t\le y\}}\vert x_t=x) $$ Therefore, you can estimate the conditional CDF using regression methods, with the caveat that $1_{\{y_t\le y\}}$ is a function of $y$ and therefore, estimation of the conditional CDF is essentially a set of regressions. Common choices for the estimation of the conditional CDF include Nadaraya-Watson, local linear estimation, or adaptive weighted Nadaraya-Watson.

Hope this helps.

  • $\begingroup$ In other words, perform a set $i \in \{1,...I\}$ of classification problems each estimating $P(y_t \leq y_i | x=x_t)$. This is basically what I already have, except inverted - I have quantile estimates $\hat{y_t}(x_t) : P(y_t \leq \hat{y_t} | x_t) < \alpha \in [0,1]$. I estimate quantiles using "sophisticated" off-the-shelf ML models, but the outputs are noisy and I just consider them "features" for estimating the "true" distribution. Maybe I'll try your idea with features being these quantiles. $\endgroup$ Dec 6 '18 at 23:51
  • $\begingroup$ Update: I ended up least-squares fitting a three-parameter distribution upon an ensemble of quantile estimators. $\endgroup$ Feb 6 '19 at 8:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.