I am confused about the estimation of conditional probabilities. Suppose I want to predict a binary outcome variable $Y = 0,1$ given $n$ categorical features $X = (X_1, \ldots, X_n)$, i.e. to estimate the probability $P(Y=1|X=x)$, given data sample pairs $\mathcal D$.

I can train a parametric regression model (like logistic regression), and use it to predict $P(Y=y|X=x)$ over new data. Now let $X=x^*$ be a particular value of $X$ for which I have a lot of samples $\mathcal D^* = \{(y_1, x^*), \ldots, (y_N, x^*)\}$. In this case I can compute the empirical frequency $f_1 = n_1/N$ where $n_1$ is the number of $y_i = 1$ in $\mathcal D^*$.

It seems to me that $f_1 \sim P(Y=1|X=x^*)$ is going to be a precise estimate in this case (because I have a lot of data, and it I'm not making assumptions on the density, essentially by the LLN). Can I expect this to be true for any choice of parametric regressor: that

  • the frequency will give a better estimation than any other regression model, on those $x^*$ of which I have a large sample?
  • In that case, should I modify the regressor and substitute its prediction with $f_1$ to get a better one?

The problems that we usually face, are that (a) we don't have enough data to estimate the conditional distribution in multivariate scenario (curse of dimensionality) and (b) we want to extrapolate beyond the data we've seen.

Regression models solve both problems by approximating the distribution (it's expectation) with a linear model. So assuming that you have enough data for precise estimate of the empirical distribution, it could work better then logistic regression if the relation is far from linear. On another hand, it will not let you extrapolate.

  • $\begingroup$ Thank you Tim. Is it possible to give a mathematical proof of this? That empirical estimation is going to be "better" than any other (potentially biased) regression model, at least for those $X=x^*$ of which we have many samples? $\endgroup$
    – user227451
    Nov 26 '18 at 7:49
  • $\begingroup$ @user227451 what kind of proof would you expect? That there exists conditional probability distribution that can have different form then logistic regression model? I'm not claiming that empirical distribution would work better. I said assuming that you have enough data, this would mean large number of cases for each of the combinations of the possible values of all the variables, what in most cases would mean impossibly large samples... As I said in the beginning, usually we don't have big enough samples. $\endgroup$
    – Tim
    Nov 26 '18 at 8:10
  • $\begingroup$ I am interested to know if it is possible to formalize and prove in a mathematical way this general statement: that for any regression model, if at some combination of features $X=x^*$ I have a lot of samples, then the empirical estimates (for this value $x^*$) is going to be better than the model's estimate. It is a statement about fitness of models. I understand this is almost obvious, but I'm looking for a mathematical proof. I expect the proof to rely on the definition of bias and convergence results, but not being an expert I can not formalize it further. $\endgroup$
    – user227451
    Nov 26 '18 at 9:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.