Question about least squares estimator explanation with a parabola I just completed a Linear Regression course in college, and I recall that the TA during office hours would explain the least squares estimator using the drawing of a parabola, and indicate that the minimum was at the bottom. I've seen this concept come up again however with straight lines due to the equation involving absolute values. I have a basic of idea of what the least squares estimators are, but can someone explain by including an explanation of the parabola?
 A: Perhaps this explanation and walkthrough will be a bit more to your liking. Additional information and relevant formulas for OLS estimation can be found here. 
As mentioned in @whuber's comment (and as you seem to be aware), the idea in OLS is to create a function based on possible $\beta$s that can identify parameter estimates for which the sum of squared residuals is minimized (in operator form here): $\sum_{i=1}^N{(y_i-\hat{y_i})^2}$. Note that in this equation $\hat{y_i}=\hat{\beta_0}+\hat{\beta_1}x_i$, which ties in to some of the formulas @whuber included in his comments above. 
As we presumably have $x$ values for each observation in the data set, the real unknowns in the second equation then are our estimates for $\beta_0$ and $\beta_1$. 
Let's say I wanted to take a brute force approach to the problem. I could ignore linear algebra and calculus altogether, which provide me with analytic answers to the question at hand and simply try out a bunch of values for my $\hat{\beta}$s. 
So I simulate some data: 
set.seed(123)
X<-rnorm(100, mean=50, sd=10)
Y<-X+rnorm(100, mean=0, sd=15)

Now I have an $x$ variable and a $y$ variable that should probably be linearly related (though I did add a noise term in there). 
Thank goodness for computers because I cannot actually imagine iterating through plausible values for my $\hat{\beta}$s by hand. So instead I ask R to try out 100 reasonable values for my parameter estimates. 
B1<-runif(n=100, min=0, max=2)

And since I know that for each $\hat{\beta_1}$ I test the following is true $\hat{\beta_0}=\bar{y}-\hat{\beta_1}\bar{x}$. This means I can write a function that 1) calculates $\hat{\beta_0}$ from my possible $\hat{\beta_1}$'s, 2) calculates $\hat{y_i}$ for each "test" regression equation, 3) calculates $\sum_{i=1}^N{(y_i-\hat{y_i})^2}$, and 4) stores the relevant values for future use: 
my.regress<-function(X, Y, B1){
    DF<-data.frame()
    for(i in 1:length(B1)){
        #1)
        B0<-mean(Y)-mean(X)*B1[i]
        #2)
        Y.hat<-B0+B1[i]*X
        #3)
        SSe<-sum((Y-Y.hat)^2)
        #4)
        temp.vec<-c(B0, B1[i], SSe)
        DF<-rbind(DF, temp.vec)
    }
    colnames(DF)<-c('B0', 'B1', 'SSe')
    return(DF)
}

All I have to do is tell my program where to find the relevant information for my function: 
My.results<-my.regress(X=X, Y=Y, B1=B1)

And voila... or well almost, as you may be asking where is your parabola. One presentation involves plotting my estimates for $\hat{\beta_1}$ on the x-axis and the sum of squared residuals on the y-axis:

I can also find the values for my $\hat{\beta}$s that produce the smallest sum of squared residuals (at least among the 100 possible values I examined). 
> My.results[My.results$SSe==min(My.results$SSe),]
         B0        B1      SSe
93 2.480992 0.9195704 20777.45

And... they are not too far off from the "correct" values: 
> fit<-lm(Y~X)
> summary(fit)

Call:
lm(formula = Y ~ X)

Residuals:
    Min      1Q  Median      3Q     Max 
-28.610 -10.253  -1.312   8.710  49.356 

Coefficients:
            Estimate Std. Error t value Pr(>|t|)    
(Intercept)   2.3933     8.2897   0.289    0.773    
X             0.9213     0.1603   5.747 1.03e-07 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 14.56 on 98 degrees of freedom
Multiple R-squared:  0.252,     Adjusted R-squared:  0.2444 
F-statistic: 33.02 on 1 and 98 DF,  p-value: 1.029e-07

I hope that helps provide a little more insight about what is going on with OLS estimation at a more conceptual level. I tried to shy away from the linear algebra, but despite the headscratching it often produces it is worth understanding if you are going to find yourself working with these models often in the future. 
