# Estimating conditional probability with many samples

I am confused about the estimation of conditional probabilities. Suppose I want to predict a binary outcome variable $$Y = 0,1$$ given $$n$$ categorical features $$X = (X_1, \ldots, X_n)$$, i.e. to estimate the probability $$P(Y=1|X=x)$$, given data sample pairs $$\mathcal D$$.

I can train a parametric regression model (like logistic regression), and use it to predict $$P(Y=y|X=x)$$ over new data. Now let $$X=x^*$$ be a particular value of $$X$$ for which I have a lot of samples $$\mathcal D^* = \{(y_1, x^*), \ldots, (y_N, x^*)\}$$. In this case I can compute the empirical frequency $$f_1 = n_1/N$$ where $$n_1$$ is the number of $$y_i = 1$$ in $$\mathcal D^*$$.

It seems to me that $$f_1 \sim P(Y=1|X=x^*)$$ is going to be a precise estimate in this case (because I have a lot of data, and it I'm not making assumptions on the density, essentially by the LLN). Can I expect this to be true for any choice of parametric regressor: that

• the frequency will give a better estimation than any other regression model, on those $$x^*$$ of which I have a large sample?
• In that case, should I modify the regressor and substitute its prediction with $$f_1$$ to get a better one?

• Thank you Tim. Is it possible to give a mathematical proof of this? That empirical estimation is going to be "better" than any other (potentially biased) regression model, at least for those $X=x^*$ of which we have many samples? Nov 26 '18 at 7:49
• I am interested to know if it is possible to formalize and prove in a mathematical way this general statement: that for any regression model, if at some combination of features $X=x^*$ I have a lot of samples, then the empirical estimates (for this value $x^*$) is going to be better than the model's estimate. It is a statement about fitness of models. I understand this is almost obvious, but I'm looking for a mathematical proof. I expect the proof to rely on the definition of bias and convergence results, but not being an expert I can not formalize it further. Nov 26 '18 at 9:24