I'm completely blind and come from a programming background.

What I'm trying to do is to learn machine learning, and to do this, I first need to learn about linear regression. All the explanations on the Internet I am finding about this subject plot the data first. I'm looking for a practical explanation of linear regression that is not dependent on graphs and plots.

Here is my understanding of the aim of simple linear regression:

Simple linear regression is trying to find the formula that once you give X to it, would provide you with the closest estimation of Y.

So, as I understand it, what needs to be done is to compare the predictor (for example the area of a house in square feet) with the independent variable (the price). In my example, you probably can create a non-visual way of getting the best formula to calculate the price of a house from its area. For example, maybe you would get the area and price of 1000 houses in a neighborhood, and divide the price to the area? The result (at least in Iran which is where I live) would have a very negligible variance. So you'd probably get something like this:

Price = 2333 Rials * Area of the house

Of course, you would then need to go through all the 1000 houses in your data set, put the area in the formula above, compare the estimate with the real price, square the results (I guess to prevent variances from canceling each other out) and then get a number, then keep playing around with the 2333 to decrease the errors.

Of course, this is the brute force option where it will probably take ages to compute the errors and arrive at the best option, but you see what I'm saying? I didn't say anything about a graph, or a line, or points on a plot, or the best way of fitting a line to your existing data.

So, why would you need a scatter plot and linear algebra for this? Isn't there a non-visual way?

First, am I right in my assumptions? If not, I'd love to be corrected. Whether or not I am, though, is there a way to come up with the formula without playing around with linear algebra?

I would really appreciate it if I could get an example with the explanation, so that I can do it along with the text to test my understanding.

  • 2
    But do you have spacial imagination that can take over the vision? If yes, I suppose a scatterplot can be imagined some way. I doubt that the essense of regression can be captured by propositional thinking (such as verbal) solely. – ttnphns Apr 1 '16 at 13:27
  • 3
    What's your math background? The Wikipedia page called Simple Linear Regression is mostly text, and has what I think is a reasonably clear description in the first paragraph. How does that article compare to the level of detail you're looking for? – shadowtalker Apr 1 '16 at 13:30
  • 3
    I'll keep on thinking of it, see if I can come up, but right off the bat, think about regression as solving an equation that has no solution. All your data points will be incorrectly predicted by your regressor (the area of the house). You are looking for an equation that makes your errors as tolerable as possible. – Antoni Parellada Apr 1 '16 at 14:02
  • 8
    excellent question, we need to think more about explaining our concepts to people with disabilities – Aksakal Apr 1 '16 at 14:09
  • 4
    You don't need to use a plot. Indeed, for multiple linear regression (regression with many predictors) you can't plot a $p+1$ dimensional space. However, the linear algebra still works. All linear algebra formulae involved in linear regression can be reduced to operations on simple scalar numbers. You just wouldn't want to do it that way by hand if you value your sanity. – conjectures Apr 1 '16 at 14:24

11 Answers 11

up vote 17 down vote accepted

Yes your onto it. You have to keep playing around with the 2333 until you find the right one which minimizes the error. But there's a mathematical way to find the "right" one. Let's call that number $\beta$. $E$, the sum of the squared errors (SSE) is a function of $\beta$ since for each choice of $\beta$ can calculate the amount each estimate is off, square it, and sum them together.

What $\beta$ minimizes the total sum of the squared errors? This is just a calculus problem. Take the derivative of $E$ by $\beta$ and set it equal to zero. This gives an equation for $\beta$. Check the second derivative is positive to know that it's a minimium. Thus you get an equation for $\beta$ which minimizes the error.

If you derive it this way, you will get $\beta$ as a summation. If you write out the linear algebra form of the estimate you will see that this is the same thing.

Edit: Here's a link to some notes with this type of derivation. The math gets a little messy, but at it's core it's just a calculus problem.

  • OMG. Finally! A non-linear-algebra way to calculate this. The concepts you are talking about in your answer are over my head, but I'll definitely look into derivatives in an effort to understand this line of thinking better. – Parham Doustdar Apr 2 '16 at 7:37
  • 1
    I linked to some notes that explain it at a pretty elementary level. I think any answer will need calculus because the way you solve problems like "find the minimum of $E(\beta)$" is to take a derivative and set it equal to zero. Intuitively, this is just saying that the minimum (or maximum) of a hill will be where the hill is flat (since the slope is highest along the side of the hill!). Derivative = slope. So in areas changing $\beta$ starts causing little change in $E$ you're near the minimum (or maximum. You need to make sure it's not a maximum!). – Chris Rackauckas Apr 2 '16 at 7:42
  • 4
    This idea then brings you to machine learning. One of the basic methods in machine learning is gradient decent. That basically translates to "follow the slope". if you keep on letting the ball roll in the direction where the hill is steepest, you'll hit a minimum. So the gradient decent method is to do precisely this: find out which way of changing $\beta$ causes the error to decrease the most and go that way! – Chris Rackauckas Apr 2 '16 at 7:49
  • 2
    For least squares regression you don't need to do gradient decent since you can solve for an equation which is the answer, but this gives a good way of understanding what machine learning is. It boils down to choosing a way of measuring error, and then finding some way to minimize the error equation. The result is the "best" estimating equation learned via the data. I hope that helps you on your path to machine learning! – Chris Rackauckas Apr 2 '16 at 7:50

Your understanding is close, but needs some extension: Simple linear regression is trying to find the formula that once you give X to it, would provide you with the closest estimation of Y based on a linear relation between X and Y.

Your example of house prices, when extended a bit, shows why you end up with scatter plots and the like. First, simply dividing the price by the area doesn't work in other cases, like land prices in my home town, where regulations on construction mean that simply owning a parcel of land upon which you can build a house has a high value. So land prices aren't simply proportional to areas. Each increase of parcel area might give the same increase in parcel value, but if you went all the way down to a (mythical) parcel of 0 area there would still be an associated apparent price that represents the value of just owning a parcel of land that's approved for building.

That's still a linear relation between area and value, but there is an intercept in the relation, representing the value of just owning a parcel. What makes this nevertheless a linear relation is that the change in value per unit change in area, the slope or the regression coefficient, is always the same regardless of the magnitudes of area or value.

So say that you already know somehow both the intercept and the slope that relate parcel areas to value, and you compare the values from that linear relation to the actual values represented by recent sales. You will find that the predicted and actual values seldom if ever coincide. These discrepancies represent the errors in your model, and result in a scatter of values around the predicted relation. You get a scatter plot of points clustered around your predicted straight-line relation between area and value.

In most practical examples you don't already know the intercept and the slope, so you have to try to estimate them from the data. That's what linear regression tries to do.

You may be better off thinking about linear regression and related modeling from the perspective of maximum-likelihood estimation, which is a search for the particular parameter values in your model that make the data the most probable. It's similar to the "brute-force" approach you propose in your question, but with a somewhat different measure of what you are trying to optimize. With modern computing methods and intelligent design of the search pattern, it can be done quite quickly.

Maximum-likelihood estimation can be conceptualized in ways that don't require a graphical plot and is similar to the way you already seem to be thinking. In the case of linear regression, both standard least-squares regression and maximum likelihood provide the same estimates of intercept and slope.

Thinking in terms of maximum likelihood has the additional advantage that it extends better to other situations where there aren't strictly linear relations. A good example is logistic regression in which you try to estimate the probability of an event occurring based on predictor variables. That can be accomplished by maximum likelihood, but unlike standard linear regression there is no simple equation that produces the intercept and slopes in logistic regression.

  • 1
    I thought that ''linear'' in ''linear regression'' meant ''linear in the parameters'', so you may have $x^2$ as an independent variable, but the coefficient of each independent variable must appear in a linear way ? – user83346 Apr 1 '16 at 15:31
  • @fcop you are correct. I was starting from the example provided by the OP, which posited a proportionality between values and areas. I tend to think about the transformed values of original predictor variables as the actual independent variables in the regression when transformations like powers or logs are used. I think that ends up in practice as mostly a difference in terminology, although there are differences in implied error models. – EdM Apr 1 '16 at 15:49
  • I see your point, anyhow, it was a good answer (+1) – user83346 Apr 1 '16 at 15:59

First of all, my compliments. It is difficult for everyone to struggle with statistics (I am a physician, so you can guess how hard it is for me)...

I can propose not a visual explanation to linear regression, but something very close: a tactile explanation to linear regression.

Imagine you are entering a room from a door. The room is more or less a square in shape, and the door is in the lower left corner. You wish to get to the next room, whose door you expect is going to be in the upper right corner, more or less. Imagine that you cannot tell exactly where the next door is (ever!), but there are some people scattered in the room, and they can tell you which were to go. They can't see either, but they can tell you what is there close to them. The final path you will take to reach the next door, guided by this people, is analogous to a regression line, which minimizes the distance between these people, and brings you toward the door, close to (if not on) the correct path.

  • 1
    (+1) I like your example very much and it is funny that by pure coincidence we used very similar illustration for this problem! – Tim Apr 1 '16 at 15:05
  • "The room is more or less a square in shape" - what is square to blind people? With this sentence you got us back to where we were to start with. – Aksakal Apr 1 '16 at 15:57
  • 4
    I don't agree. Let them walk 10 feet in one direction, then let them turn 90° (such as an armspan) and let them walk again 10 feet. That's a square if you can't see properly. – Joe_74 Apr 1 '16 at 16:02
  • @GiuseppeBiondi-Zoccai, if I'm building a model of pressure in the chamber on the temperature, why would I need to bring up squares and lines and other spatial concepts? It's surely convenient if you're not blind, but for a blind person these spatial analogies do not bring anything to the table for the problem at hand, they only complicate the exposition – Aksakal Apr 1 '16 at 17:23
  • 2
    Again, I politely disagree... my assumption has always been that blind people have particularly developed tactile spatial skills. Anyway, any example which work is fine, and the more the merrier. – Joe_74 Apr 1 '16 at 18:59

Nice example that can help for your question was provided by Andrew Gelman and David K. Park (2012). Let's stick to your example of predicting the price of house $Y$ given it's area $X$. For this we use simple linear regression model

$$ Y = \beta_0 + \beta_1 X + \varepsilon $$

For sake of simplicity, let's forget about the intercept $\beta_0$, you can check this thread to learn why is it important. This data can be visualized on a scatterplot. What is scatterplot? Imagine two-dimensional space (it could be a room), the datapoints are "scattered" around the place, where values of both variables mark their $y$-axis and $x$-axis positions. What you already know, is that it somehow translates to the linear regression model.

To make it clear, lets simplify this example even more -- as Gelman and Park did. The simplification that they proposed is to divide the $X$ variable, i.e. area of the house, into three groups: "small", "medium", and "big" houses (they describe how to optimally make such decision, but this is of lesser importance). Next, calculate the average size of "small" house and average size of "big" house. Calculate also average price of "small" house and of "big" one. Now, reduce your data to two points -- the centers of the clouds of datapoints for small and big houses scattered in the space and remove all the datapoints about "medium" houses. You are left with two points in two-dimensional space. Regression line is the line that connects the points -- you can think of it as a direction from one point to the another. The slope $\beta_1$ of this line tells us about amount of change between small and big houses in their prices.

The same happens when we have more points, scattered around the space: regression line finds her way by minimizing it's square distance to every point. So the line is going exactly through the center of the cloud of points scattered in the space. Instead of connecting two points, you can think of it as connecting unlimited number of such central points.


Gelman, A., & Park, D. K. (2012). Splitting a predictor at the upper quarter or third and the lower quarter or third. The American Statistician, 62(4), 1-8.

The short answer is, yes. What line goes best through the middle of all points that comprise the entirety or just the surface of an airplane or javelin? Draw it; in your head or on a picture. You are looking for and at that solitary line from which every point (of interest, whether you plot them or not) that would contribute to total least (among points) deviation from that line. If you do it by eye, implicitly by common sense, you will approximate (remarkably well) a mathematically calculated result. For that there are formulae which bother the eye and may not make common sense. In similar formalized problems in engineering and science, the scatters still invite a preliminary appraisal by eye, but in those arenas one is supposed to come up with a "test" probability that a line is the line. It goes downhill from there. However, you are apparently trying to teach a machine to size up (in effect) the metes and bounds of (a) a sizeable barnyard and (b) scattered livestock inside it. If you give your machine what amounts to a picture (graphical, algebraic) of the real estate and occupants, it should be able to figure out (midline neatly dividing blob in two, calculated descatter into a line) what you want it to do. Any decent statistics textbook (ask teachers or professors to name more than one) should spell out both the whole point of linear regression in the first place, and how to do it in the simplest cases (ranging to cases that are not simple). A number of pretzels later, you'll have it down pat.


In re: Silverfish's comment to my post supra (there seems no simple way other than this to add comment to that comment), yes, the OP is blind, is learning machine learning, and requested practicality without plots or graphs, but I assume that he is able to distinguish "visualizing" from "vision", visualizes and has veritable pictures in his head, and has a basic idea of all manner of physical in objects the world around him (houses, among others), so he can still "draw" both mathematically as well as otherwise in his head, and can probably put a good semblance of 2D and 3D to paper. A wide array of books and other texts nowadays is available in physical Braille as well as in electronic voice on one's own computer (such as for forums, dictionaries, etc.), and many schools for the blind have fairly complete curricula. Rather than airplane or javelin, sofa or cane would not necessarily be the more appropriate, and statistics texts are probably available. He is less concerned for how machines might learn to plot and graph or calculate regression, then for how machines might learn to do something equivalent (and more basic) in order to grasp regression (whether a machine might display it, react to it, follow it, avoid it, or whatever). The essential thrust (as to blind as well as to sighted students) is still how to visualize what can be non-visual (such as concept of linearity rather than instance of drawn line, since before Euclid and Pythagoras), and how to visualize the basic purpose of a special kind of linearity (regression, whose basic point is best fit to least deviation, since early in mathematics and statistics). A lineprinter's Fortran output of regression is scarcely "visual" till mentally assimilated, but even the basic point of regression is imaginary (a line that isn't there till it is made for a purpose).

  • 2
    Perhaps I'm misunderstanding this answer, but "draw it, in your head or on a picture" seems to somewhat miss the point of the question: the original question is posed by someone who is completely blind, and therefore looking for a non-visual way of approaching regression. – Silverfish Apr 1 '16 at 22:25
  • @Silverfish Response (too long for a comment) has been edited into the answer above – user110711 Apr 1 '16 at 23:16
  • Thanks. I thought the downvote was a bit harsh (it wasn't me) but some of the language choices in this answer were unfortunate (e.g. there are several references to doing things "by eye"). Nevertheless, I can understand why you would want to distinguish between visual perception and what can be visualised through the "mind's eye". – Silverfish Apr 2 '16 at 1:33
  • 2
    I can visualize things in my mind. It's just that I don't use the same ways of visualization. It's not a matter of not using draw or visualize. It's just a matter of using the concept to derive the visualization, rather than the other way around. I have found that this happens in a lot of places in mathematics. To explain a difficult subject, usually shapes and images are used, rather than relating the calculation to concepts that the learner would know from real life. – Parham Doustdar Apr 2 '16 at 6:28

The reason why plots are universally used to introduce simple regression - a response predicted by a single predictor - is that they aid understanding.

However, I believe I can give something of the flavor that might aid in understanding what's going on. In this I'll mostly focus on trying to convey some of the understanding they give, which may help with some of the other aspects you'll typically encounter in reading about regression. So this answer will mainly deal with a particular aspect of your post.

Imagine you are seated before a large rectangular table such as a plain office desk, one a full arm-span long (perhaps 1.8 meters), by perhaps half that wide.

You are seated before the table in the usual position, in the middle of one long side. On this table a large number of nails (with fairly smooth heads) have been hammered into the top surface such that each pokes up a little way (enough to feel where they are, and enough to tie a string to them or attach a rubber band).

These nails are at varying distances from your edge of the desk, in such a way that toward one end (say the left end) they typically are closer to your edge of the desk and then as you move toward the other end the nail-heads tend to be further away from your edge.

Further imagine that it would be useful to have a sense of how far on average the nails are from your edge at any give position along your edge.

Choose some place along your edge of the desk and place your hand there, then reach forward directly across the table, gently dragging your hand directly back toward you, then away again, moving your hand back and forth over the nail heads. You encounter several dozen bumps from these nails - the ones within that narrow breadth of your hand (as it moves directly away from your edge, at constant distance from the left end of the desk), a section, or strip, roughly ten centimeters wide.

The idea is to figure out some average distance to a nail from your edge of the desk in that small section. Intuitively it's just the middle of the bumps we hit but if we measured each distance-to-a-nail in that hand-breadth-wide section of desk, we could compute those averages easily.

For example, we could make use of a T-square whose head slides along the edge of the desk and whose shaft runs toward the other side of the desk, but just above the desk so we don't hit the nails as it slides left or right - as we pass a given nail we can get its distance along the shaft of the T-square.

So at a progression of places along our edge we repeat this exercise of finding all the nails in a hand-width strip running toward and away from us and finding their average distance away. Perhaps we divide the desk up into hand-width strips along our edge (so every nail is encountered in exactly one strip).

Now imagine there were say 21 such strips, the first at the left edge and the last at the right edge. The means get further away from our desk-edge as we progress across the strips.

These means form a simple nonparametric regression estimator of the expectation of y (our distance-away) given x (distance along our edge from the left end), that is, E(y|x). Specifically, this is a binned nonparametric regression estimator, also called a regressogram

If those strip means increased regularly - that is, the mean was typically increasing by about the same amount-per-strip as we moved across the strips - then we could better estimate our regression function by assuming that the expected value of y was a linear function of x - i.e. that the expected value of y given x was a constant plus a multiple of x. Here the constant represents where the nails tend to be when we at x is zero (often we might place this at the extreme left edge but it doesn't have to be), and the particular multiple of x being how fast on average the mean changes as we move by one centimeter (say) to the right.

But how to find such a linear function?

Imagine that we loop one rubber band over each nail-head, and attach each to a long thin stick that lays just above the desk, on top of the nails, so that it lays somewhere near the "middle" of each strip we had be for.

We attach the bands in such a way that they only stretch in the direction toward and away from us (not left or right) - left to themselves they would pull so as to make their direction of stretch at a right-angle with the stick, but here we prevent that, so that their direction of stretch remains only in the directions toward or away from our edge of the desk. Now we let the stick settle as the bands pull it toward each nail, with more distant nails (with more stretched rubber bands) pulling correspondingly harder than nails close to the stick.

Then the combined result of all the bands pulling on the stick would be (ideally, at least) to pull the stick to minimize the sum of squared lengths of the stretched rubber bands; in that direction directly across the table the distance from our edge of the table to the stick at any given x position would be our estimate of the expected value of y given x.

This is essentially a linear regression estimate.

Now, imagine that instead of nails, we have many fruits (like small apples perhaps) hanging from a large tree and we wish to find the average distance of fruits above the ground as it varies with position on the ground. Imagine that in this case the heights above the ground get larger as we go forward and slightly larger as we move right, again in a regular fashion, so each step forward typically changes the mean height by about the same amount, and each step to the right will also change the mean by a roughly constant amount (but this stepping-right amount of change in mean is different to the stepping-forward amount of change).

If we minimize the sum of squared vertical distances from the fruits to a thin flat sheet (perhaps a thin sheet of very stiff plastic) in order to figure out how the mean height changes as we move forward or step to the right, that would be a linear regression with two predictors - a multiple regression.

These are the only two cases that plots can help understand (they can show rapidly what I just described at length, but hopefully you know have a basis on which to conceptualize the same ideas). Beyond those simplest two cases, we're left with the mathematics only.

Now take your house price example; you can represent every house's area by a distance along your edge of the desk - represent the largest house size as a position near the right edge, every other house size will be some position further to the left where a certain number of centimeters will represent some number of square meters. Now the distance away represents sale price. Represent the most expensive house as some particular distance near the furthest edge of the desk (as always, the edge furthest from your chair), and every centimeter shifted away will represent some number of Rials.

For the present imagine that we chose the representation so that the left edge of the desk corresponds to a house area of zero and the near edge to a house price of 0. We then put in a nail for each house.

We probably won't have any nails near the left end of our edge (they might be mostly toward the right and away from us) because this isn't necessarily a good choice of scale but your choice of a no-intercept model makes this a better way to discuss it.

Now in your model you force the stick to pass through a loop of string at the left corner of the near edge of the desk - thus forcing the fitted model to have price zero for area zero, which might seem natural - but imagine if there are some fairly constant components of price which affected every sale. Then it would make sense to have the intercept different from zero.

In any case, with the addition of that loop, the same rubber-band exercise as before will find our least squares estimate of the line.

  • Wow, thank you for this lengthy spatial answer. It explained a lot. Thanks. – Parham Doustdar Apr 5 '16 at 4:33

Have you encountered the sort of toaster you often get in hotels. You put bread on a conveyor belt at one end and it comes out as toast at the other. Unfortunately, in the toaster at this cheap hotel, the heaters have all got moved to random heights and distances from the entrance to the toaster. You cannot move the heaters or bend the path of the belt (which is straight, by the way (this is where the linear bit comes in), but you can alter the HEIGHT and TILT of the belt .

Given the positions of all the heaters, linear regression will tell you the correct height and angle to place the belt to get the most heat overall. This is because linear regression will minimise the average distance between the toast and the heaters.

My first holiday job was doing linear regressions by hand. The guy who said you don't want to do that is RIGHT!!!

My favorite explanation of linear regression is geometric, but not visual. It treats the data set as a single point in a high-dimensional space, rather than breaking it up into a cloud of points in two-dimensional space.

The area $a$ and price $p$ of a house are a pair of numbers, which you can think of as the coordinates of a point $(a, p)$ in two-dimensional space. The areas $a_1, \ldots, a_{1000}$ and prices $p_1, \ldots, p_{1000}$ of a thousand houses are a thousand pairs of numbers, which you can think of as the coordinates of a point $$D = (a_1, \ldots, a_{1000}, p_1, \ldots, p_{1000})$$ in two-thousand-dimensional space. For convenience, I'll call two-thousand-dimensional space "data space." Your data set $D$ is a single point in data space.

If the relationship between area and price were perfectly linear, the point $D$ would sit in a very special region of data space, which I'll call the "linear sheet." It consists of the points $$M(\rho, \beta) = (a_1, \ldots, a_{1000}, \rho a_1 + \beta, \ldots, \rho a_{1000} + \beta).$$ The numbers $\rho$ and $\beta$ are allowed to vary, but $a_1, \ldots, a_{1000}$ are fixed to be the same areas that appear in your data set. I'm calling the linear sheet a "sheet" because it's two-dimensional: a point on it is specified by the two coordinates $\rho$ and $\beta$. If you want to get a sense of how the linear sheet is shaped, imagine a thin, straight wire stretched across three-dimensional space. The linear sheet is like that: it's perfectly flat, and its dimension is very low compared to the dimension of the space it sits inside.

In a real neighborhood, the relationship between area and price won't be perfectly linear, so the point $D$ won't sit exactly on the linear sheet. However, it might sit very close to the linear sheet. The goal of linear regression is to find the point $M(\rho, \beta)$ on the linear sheet which sits the closest to the data point $D$. That point is the best linear model for the data.

Using the Pythagorean theorem, you can figure out that the square of the distance between $D$ and $M(\rho, \beta)$ is $$[p_1 - (\rho a_1 + \beta)]^2 + \ldots + [p_{1000} - (\rho a_{1000} + \beta)]^2.$$ In other words, the distance between the data point and the model point is the total squared error of the model! Minimizing the total squared error of a model is the same thing as minimizing the distance between the model and the data in data space.

As Chris Rackauckas pointed out, calculus gives a very practical way to find the coordinates $\rho$ and $\beta$ that minimize the distance between $D$ and $M(\rho, \beta)$.

@Chris Rackauckas and @EDM's answers are spot on. There are many ways to approach simple linear regression that don't require plotting or visual explanations of ordinary least squares estimation, and they give very solid explanations of what actually happens when you're running OLS.

I might add that using scatterplots as an instruction tool to learn any kind of new modeling procedure, whether it's old school parametric model, advanced machine learning stuff, or bayesian algorithms, graphing can help cut down on the time it takes to learn what a particular algorithm does.

Graphing is also very important for exploratory data analysis when you are first beginning to work with a new dataset. I have had situations where I collected lots of data, worked out the theory, carefully planned out my model, and then ran it, only to end up with results that essentially had no predictive power. Plotting bivariate relationships can take out some of the guesswork: in your example, it's possible that home price is linearly related to area, but maybe the relationship isn't linear. Scatterplots help you decide if you need higher order terms in your regression, or if you want to use a different method than linear regression, or if you want to use some sort of nonparametric method.

Google for Anscombe Quartet.

It shows 4 sets of data which on inspecting numerically do not show much difference.

However, on creating a visual scatter plot, the differences become dramatically visible.

It gives a pretty clear view why you should always plot your data, regression or no regression :-)

We want to have a solution that minimizes the difference between the predicted and actual values.

We assume that the $y=bx+a$ i.e. there is a linear relationship.

We don't care whether the difference between predicted and actual $y$ is positive or negative assume that distribution of errors of $y$ posses certain properties.

If we assume that the distribution of errors is normally distributed it turns out that there is an analytical solution to this minimization problem. The the sum of squares of differences is the best value to minimize for a best fit. But normality is not required in general case.

There isn't much more to it really.

The geometrical interpretation comes handy because sum of squares has the interpretation in the form of sum of distances of the dots on the scatter plot from the $y=bx+a$ line. And human eye is very good at approximating the line that corresponds to the best fit. So it was handy before we had computers to find the fit quickly.

Nowadays it is left more as a comprehension help but is not necessary to have to understand linear regression really.

EDIT:replaced the normality of errors assumption with a correct but less concise list. Normality was required to have an analytical solution and can be assumed for many practical cases and in that case sum of squares is optimal not only for the linear estimator and maximizes likelihood as well.

If further the assumption of normality of error distribution holds then the Sum of Squares is optimal among both linear and non-linear estimators and is maximizing the likelihood.

  • 1
    Normal distribution assumption is not required for anything you described – Aksakal Apr 2 '16 at 2:27
  • Pls check this explanation stats.stackexchange.com/a/1516/98469 – Diego Apr 4 '16 at 19:57
  • The link has nothing to do with your answer. If you expanded into small sample properties or MLE, then you could bring in the normal distribution assumption, but as it stands the OLS description in your answer doesn't need normal distribution. In fact tTo minimize sum of squares you don't need any distribution or statistics at all. It's pure algebra. – Aksakal Apr 4 '16 at 20:07
  • The point is about why are we minimizing sum of squares and not some other metric. Not about how to minimize sum of squares. – Diego Apr 5 '16 at 14:39
  • Minimizing sum of squares has nothing to do with normal distribution. It's just your loss function. Any other error distribution can be used with this loss function. You need the distributions in certain cases, e.g. if you want to make inferences about parameter values in small samples etc. Even in this case you could use other distributions, I'm not sure why you're stuck on normal. – Aksakal Apr 5 '16 at 15:58

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.