Conditional Expected Value of the Response Variable and the Prediction with the fitted model in Linear Regression I am running the following code to check my concepts in simple linear regression. As we know from theory, $E[Y|X] = \beta_0 + \beta_1X$ or put in another way, $Y=\beta_0 + \beta_1X + \epsilon$, I was expecting that lm with R I shall have $E[Y|X=x]= \beta_0 + \beta_1x$, but for $X=4$ the values obtained are different. Can anyone explain what's happening? The following is the code I used.
set.seed(1234)
x <- sample(1:10, 100, replace=TRUE)
y <- sample(1:10, 100, replace=TRUE)
m <- lm(y~x)
summary(m)
# Coefficients:
#        Estimate Std. Error t value Pr(>|t|)    
#(Intercept)  6.16881    0.59312  10.401   <2e-16 ***
#x           -0.06643    0.10460  -0.635    0.527    
c(1, 4) %*% coefficients(m)
#[1,] 5.903106
predict(m, newdata=data.frame(x=4))
#[1,] 5.903106
mean(y[x==4]) # E[Y|X=4]
#[1] 5.928571

 A: Consider
set.seed(1234)
x <- sample(1:10, 100, replace=TRUE)
y <- sample(1:10, 100, replace=TRUE)
m <- lm(y~x)
summary(m)

x.jitter <- jitter(x)
y.jitter <- jitter(y)

plot(x.jitter,y.jitter)
abline(m)
points(x.jitter[x==4],y.jitter[x==4],col="red", pch=19)
points(4,mean(y[x==4]),col="blue",pch=19)


With mean you only use the observations for which $x$ equals 4 to produce an estimate of the conditional expectation (the points highlighted in red in the plot - all points are jittered to be able to see if there are several points for a given $(x,y)$ combination). This conditional mean is the blue dot. With the regression (the straight line), the entire sample is used to produce estimated conditional expectations for any value of $x$. 
As it turns out (and will generally be the case), the blue dot is, while close, not exactly on the fitted regression line.
Generally, the estimates of the regression will be more stable, as more data is used to produce these. In the limit, consider, for example, what would have happened if you had desired an estimate at $x=3.5$. With mean, that would not have been possible as you do not have observations to average over at that $x$. 
Conversely, if a linear model does not describe the conditional expectation, it may be better to only use local observations to produce an estimate rather than using the entire sample. 
What you basically did with mean was a nonparametric regression with a kernel equal to a point mass at $x=4$. More conventional nonparametric regression also uses "neighboring" observations, from which we can also draw information even if the conditional expectation is nonlinear.
