I think the ordinary least squares are the sum of the vertical distance between the observed data to the model line(regression). And residual is calculated by add up all vertical distance between each observed data and the model line(regression), not by sum of squares? I am still not clear about these concept. Thanks!
You have the terms associated with the wrong concepts.
The residual for an observation is the difference between the observation (the y-value) and the fitted line. Some residuals are positive and some are negative.
The usual regression line is fitted by minimizing the sum of squares of the residuals (hence 'least squares').
[The sum of the distances (the absolute values of the residuals) is not used in ordinary least squares -- but they are used in least absolute values regression (also called $L_1$ regression). Least squares is easier to fit, but many statistics packages will offer both.]
Least square is an estimating method by minimizing the sum of squares of vertical distances. Residual, a quantitative measurement, is the vertical distance between the observed and the estimated using estimated parameters. Each observation has one residual (NO summing up, No square) and the residual could be negative or positive. We usually check the scatter plot of residuals to learn the fitting of a model.