Can simple linear regression be done without using plots and linear algebra?

Question

I'm completely blind and come from a programming background.

What I'm trying to do is to learn machine learning, and to do this, I first need to learn about linear regression. All the explanations on the Internet I am finding about this subject plot the data first. I'm looking for a practical explanation of linear regression that is not dependent on graphs and plots.

Here is my understanding of the aim of simple linear regression:

Simple linear regression is trying to find the formula that once you give X to it, would provide you with the closest estimation of Y.

So, as I understand it, what needs to be done is to compare the predictor (for example the area of a house in square feet) with the independent variable (the price). In my example, you probably can create a non-visual way of getting the best formula to calculate the price of a house from its area. For example, maybe you would get the area and price of 1000 houses in a neighborhood, and divide the price to the area? The result (at least in Iran which is where I live) would have a very negligible variance. So you'd probably get something like this:

Price = 2333 Rials * Area of the house

Of course, you would then need to go through all the 1000 houses in your data set, put the area in the formula above, compare the estimate with the real price, square the results (I guess to prevent variances from canceling each other out) and then get a number, then keep playing around with the 2333 to decrease the errors.

Of course, this is the brute force option where it will probably take ages to compute the errors and arrive at the best option, but you see what I'm saying? I didn't say anything about a graph, or a line, or points on a plot, or the best way of fitting a line to your existing data.

So, why would you need a scatter plot and linear algebra for this? Isn't there a non-visual way?

First, am I right in my assumptions? If not, I'd love to be corrected. Whether or not I am, though, is there a way to come up with the formula without playing around with linear algebra?

I would really appreciate it if I could get an example with the explanation, so that I can do it along with the text to test my understanding.

Answer 1

Yes your onto it. You have to keep playing around with the 2333 until you find the right one which minimizes the error. But there's a mathematical way to find the "right" one. Let's call that number $\beta$. $E$, the sum of the squared errors (SSE) is a function of $\beta$ since for each choice of $\beta$ can calculate the amount each estimate is off, square it, and sum them together.

What $\beta$ minimizes the total sum of the squared errors? This is just a calculus problem. Take the derivative of $E$ by $\beta$ and set it equal to zero. This gives an equation for $\beta$. Check the second derivative is positive to know that it's a minimium. Thus you get an equation for $\beta$ which minimizes the error.

If you derive it this way, you will get $\beta$ as a summation. If you write out the linear algebra form of the estimate you will see that this is the same thing.

Edit: Here's a link to some notes with this type of derivation. The math gets a little messy, but at it's core it's just a calculus problem.

Answer 2

Your understanding is close, but needs some extension: Simple linear regression is trying to find the formula that once you give X to it, would provide you with the closest estimation of Y based on a linear relation between X and Y.

Your example of house prices, when extended a bit, shows why you end up with scatter plots and the like. First, simply dividing the price by the area doesn't work in other cases, like land prices in my home town, where regulations on construction mean that simply owning a parcel of land upon which you can build a house has a high value. So land prices aren't simply proportional to areas. Each increase of parcel area might give the same increase in parcel value, but if you went all the way down to a (mythical) parcel of 0 area there would still be an associated apparent price that represents the value of just owning a parcel of land that's approved for building.

That's still a linear relation between area and value, but there is an intercept in the relation, representing the value of just owning a parcel. What makes this nevertheless a linear relation is that the change in value per unit change in area, the slope or the regression coefficient, is always the same regardless of the magnitudes of area or value.

So say that you already know somehow both the intercept and the slope that relate parcel areas to value, and you compare the values from that linear relation to the actual values represented by recent sales. You will find that the predicted and actual values seldom if ever coincide. These discrepancies represent the errors in your model, and result in a scatter of values around the predicted relation. You get a scatter plot of points clustered around your predicted straight-line relation between area and value.

In most practical examples you don't already know the intercept and the slope, so you have to try to estimate them from the data. That's what linear regression tries to do.

You may be better off thinking about linear regression and related modeling from the perspective of maximum-likelihood estimation, which is a search for the particular parameter values in your model that make the data the most probable. It's similar to the "brute-force" approach you propose in your question, but with a somewhat different measure of what you are trying to optimize. With modern computing methods and intelligent design of the search pattern, it can be done quite quickly.

Maximum-likelihood estimation can be conceptualized in ways that don't require a graphical plot and is similar to the way you already seem to be thinking. In the case of linear regression, both standard least-squares regression and maximum likelihood provide the same estimates of intercept and slope.

Thinking in terms of maximum likelihood has the additional advantage that it extends better to other situations where there aren't strictly linear relations. A good example is logistic regression in which you try to estimate the probability of an event occurring based on predictor variables. That can be accomplished by maximum likelihood, but unlike standard linear regression there is no simple equation that produces the intercept and slopes in logistic regression.

Answer 3

First of all, my compliments. It is difficult for everyone to struggle with statistics (I am a physician, so you can guess how hard it is for me)...

I can propose not a visual explanation to linear regression, but something very close: a tactile explanation to linear regression.

Imagine you are entering a room from a door. The room is more or less a square in shape, and the door is in the lower left corner. You wish to get to the next room, whose door you expect is going to be in the upper right corner, more or less. Imagine that you cannot tell exactly where the next door is (ever!), but there are some people scattered in the room, and they can tell you which were to go. They can't see either, but they can tell you what is there close to them. The final path you will take to reach the next door, guided by this people, is analogous to a regression line, which minimizes the distance between these people, and brings you toward the door, close to (if not on) the correct path.

Answer 4

The reason why plots are universally used to introduce simple regression - a response predicted by a single predictor - is that they aid understanding.

However, I believe I can give something of the flavor that might aid in understanding what's going on. In this I'll mostly focus on trying to convey some of the understanding they give, which may help with some of the other aspects you'll typically encounter in reading about regression. So this answer will mainly deal with a particular aspect of your post.

Imagine you are seated before a large rectangular table such as a plain office desk, one a full arm-span long (perhaps 1.8 meters), by perhaps half that wide.

You are seated before the table in the usual position, in the middle of one long side. On this table a large number of nails (with fairly smooth heads) have been hammered into the top surface such that each pokes up a little way (enough to feel where they are, and enough to tie a string to them or attach a rubber band).

These nails are at varying distances from your edge of the desk, in such a way that toward one end (say the left end) they typically are closer to your edge of the desk and then as you move toward the other end the nail-heads tend to be further away from your edge.

Further imagine that it would be useful to have a sense of how far on average the nails are from your edge at any give position along your edge.

Choose some place along your edge of the desk and place your hand there, then reach forward directly across the table, gently dragging your hand directly back toward you, then away again, moving your hand back and forth over the nail heads. You encounter several dozen bumps from these nails - the ones within that narrow breadth of your hand (as it moves directly away from your edge, at constant distance from the left end of the desk), a section, or strip, roughly ten centimeters wide.

The idea is to figure out some average distance to a nail from your edge of the desk in that small section. Intuitively it's just the middle of the bumps we hit but if we measured each distance-to-a-nail in that hand-breadth-wide section of desk, we could compute those averages easily.

For example, we could make use of a T-square whose head slides along the edge of the desk and whose shaft runs toward the other side of the desk, but just above the desk so we don't hit the nails as it slides left or right - as we pass a given nail we can get its distance along the shaft of the T-square.

So at a progression of places along our edge we repeat this exercise of finding all the nails in a hand-width strip running toward and away from us and finding their average distance away. Perhaps we divide the desk up into hand-width strips along our edge (so every nail is encountered in exactly one strip).

Now imagine there were say 21 such strips, the first at the left edge and the last at the right edge. The means get further away from our desk-edge as we progress across the strips.

These means form a simple nonparametric regression estimator of the expectation of y (our distance-away) given x (distance along our edge from the left end), that is, E(y|x). Specifically, this is a binned nonparametric regression estimator, also called a regressogram

If those strip means increased regularly - that is, the mean was typically increasing by about the same amount-per-strip as we moved across the strips - then we could better estimate our regression function by assuming that the expected value of y was a linear function of x - i.e. that the expected value of y given x was a constant plus a multiple of x. Here the constant represents where the nails tend to be when we at x is zero (often we might place this at the extreme left edge but it doesn't have to be), and the particular multiple of x being how fast on average the mean changes as we move by one centimeter (say) to the right.

But how to find such a linear function?

Imagine that we loop one rubber band over each nail-head, and attach each to a long thin stick that lays just above the desk, on top of the nails, so that it lays somewhere near the "middle" of each strip we had be for.

We attach the bands in such a way that they only stretch in the direction toward and away from us (not left or right) - left to themselves they would pull so as to make their direction of stretch at a right-angle with the stick, but here we prevent that, so that their direction of stretch remains only in the directions toward or away from our edge of the desk. Now we let the stick settle as the bands pull it toward each nail, with more distant nails (with more stretched rubber bands) pulling correspondingly harder than nails close to the stick.

Then the combined result of all the bands pulling on the stick would be (ideally, at least) to pull the stick to minimize the sum of squared lengths of the stretched rubber bands; in that direction directly across the table the distance from our edge of the table to the stick at any given x position would be our estimate of the expected value of y given x.

This is essentially a linear regression estimate.

Now, imagine that instead of nails, we have many fruits (like small apples perhaps) hanging from a large tree and we wish to find the average distance of fruits above the ground as it varies with position on the ground. Imagine that in this case the heights above the ground get larger as we go forward and slightly larger as we move right, again in a regular fashion, so each step forward typically changes the mean height by about the same amount, and each step to the right will also change the mean by a roughly constant amount (but this stepping-right amount of change in mean is different to the stepping-forward amount of change).

If we minimize the sum of squared vertical distances from the fruits to a thin flat sheet (perhaps a thin sheet of very stiff plastic) in order to figure out how the mean height changes as we move forward or step to the right, that would be a linear regression with two predictors - a multiple regression.

These are the only two cases that plots can help understand (they can show rapidly what I just described at length, but hopefully you know have a basis on which to conceptualize the same ideas). Beyond those simplest two cases, we're left with the mathematics only.

Now take your house price example; you can represent every house's area by a distance along your edge of the desk - represent the largest house size as a position near the right edge, every other house size will be some position further to the left where a certain number of centimeters will represent some number of square meters. Now the distance away represents sale price. Represent the most expensive house as some particular distance near the furthest edge of the desk (as always, the edge furthest from your chair), and every centimeter shifted away will represent some number of Rials.

For the present imagine that we chose the representation so that the left edge of the desk corresponds to a house area of zero and the near edge to a house price of 0. We then put in a nail for each house.

We probably won't have any nails near the left end of our edge (they might be mostly toward the right and away from us) because this isn't necessarily a good choice of scale but your choice of a no-intercept model makes this a better way to discuss it.

Now in your model you force the stick to pass through a loop of string at the left corner of the near edge of the desk - thus forcing the fitted model to have price zero for area zero, which might seem natural - but imagine if there are some fairly constant components of price which affected every sale. Then it would make sense to have the intercept different from zero.

In any case, with the addition of that loop, the same rubber-band exercise as before will find our least squares estimate of the line.

Answer 5

Nice example that can help for your question was provided by Andrew Gelman and David K. Park (2012). Let's stick to your example of predicting the price of house $Y$ given it's area $X$. For this we use simple linear regression model

$$ Y = \beta_0 + \beta_1 X + \varepsilon $$

For sake of simplicity, let's forget about the intercept $\beta_0$, you can check this thread to learn why is it important. This data can be visualized on a scatterplot. What is scatterplot? Imagine two-dimensional space (it could be a room), the datapoints are "scattered" around the place, where values of both variables mark their $y$-axis and $x$-axis positions. What you already know, is that it somehow translates to the linear regression model.

To make it clear, lets simplify this example even more -- as Gelman and Park did. The simplification that they proposed is to divide the $X$ variable, i.e. area of the house, into three groups: "small", "medium", and "big" houses (they describe how to optimally make such decision, but this is of lesser importance). Next, calculate the average size of "small" house and average size of "big" house. Calculate also average price of "small" house and of "big" one. Now, reduce your data to two points -- the centers of the clouds of datapoints for small and big houses scattered in the space and remove all the datapoints about "medium" houses. You are left with two points in two-dimensional space. Regression line is the line that connects the points -- you can think of it as a direction from one point to the another. The slope $\beta_1$ of this line tells us about amount of change between small and big houses in their prices.

The same happens when we have more points, scattered around the space: regression line finds her way by minimizing it's square distance to every point. So the line is going exactly through the center of the cloud of points scattered in the space. Instead of connecting two points, you can think of it as connecting unlimited number of such central points.

---

Gelman, A., & Park, D. K. (2012). Splitting a predictor at the upper quarter or third and the lower quarter or third. The American Statistician, 62(4), 1-8.

Answer 6

The short answer is, yes. What line goes best through the middle of all points that comprise the entirety or just the surface of an airplane or javelin? Draw it; in your head or on a picture. You are looking for and at that solitary line from which every point (of interest, whether you plot them or not) that would contribute to total least (among points) deviation from that line. If you do it by eye, implicitly by common sense, you will approximate (remarkably well) a mathematically calculated result. For that there are formulae which bother the eye and may not make common sense. In similar formalized problems in engineering and science, the scatters still invite a preliminary appraisal by eye, but in those arenas one is supposed to come up with a "test" probability that a line is the line. It goes downhill from there. However, you are apparently trying to teach a machine to size up (in effect) the metes and bounds of (a) a sizeable barnyard and (b) scattered livestock inside it. If you give your machine what amounts to a picture (graphical, algebraic) of the real estate and occupants, it should be able to figure out (midline neatly dividing blob in two, calculated descatter into a line) what you want it to do. Any decent statistics textbook (ask teachers or professors to name more than one) should spell out both the whole point of linear regression in the first place, and how to do it in the simplest cases (ranging to cases that are not simple). A number of pretzels later, you'll have it down pat.

---

In re: Silverfish's comment to my post supra (there seems no simple way other than this to add comment to that comment), yes, the OP is blind, is learning machine learning, and requested practicality without plots or graphs, but I assume that he is able to distinguish "visualizing" from "vision", visualizes and has veritable pictures in his head, and has a basic idea of all manner of physical in objects the world around him (houses, among others), so he can still "draw" both mathematically as well as otherwise in his head, and can probably put a good semblance of 2D and 3D to paper. A wide array of books and other texts nowadays is available in physical Braille as well as in electronic voice on one's own computer (such as for forums, dictionaries, etc.), and many schools for the blind have fairly complete curricula. Rather than airplane or javelin, sofa or cane would not necessarily be the more appropriate, and statistics texts are probably available. He is less concerned for how machines might learn to plot and graph or calculate regression, then for how machines might learn to do something equivalent (and more basic) in order to grasp regression (whether a machine might display it, react to it, follow it, avoid it, or whatever). The essential thrust (as to blind as well as to sighted students) is still how to visualize what can be non-visual (such as concept of linearity rather than instance of drawn line, since before Euclid and Pythagoras), and how to visualize the basic purpose of a special kind of linearity (regression, whose basic point is best fit to least deviation, since early in mathematics and statistics). A lineprinter's Fortran output of regression is scarcely "visual" till mentally assimilated, but even the basic point of regression is imaginary (a line that isn't there till it is made for a purpose).

Answer 7

Have you encountered the sort of toaster you often get in hotels. You put bread on a conveyor belt at one end and it comes out as toast at the other. Unfortunately, in the toaster at this cheap hotel, the heaters have all got moved to random heights and distances from the entrance to the toaster. You cannot move the heaters or bend the path of the belt (which is straight, by the way (this is where the linear bit comes in), but you can alter the HEIGHT and TILT of the belt .

Given the positions of all the heaters, linear regression will tell you the correct height and angle to place the belt to get the most heat overall. This is because linear regression will minimise the average distance between the toast and the heaters.

My first holiday job was doing linear regressions by hand. The guy who said you don't want to do that is RIGHT!!!

Answer 8

My favorite explanation of linear regression is geometric, but not visual. It treats the data set as a single point in a high-dimensional space, rather than breaking it up into a cloud of points in two-dimensional space.

The area $a$ and price $p$ of a house are a pair of numbers, which you can think of as the coordinates of a point $(a, p)$ in two-dimensional space. The areas $a_1, \ldots, a_{1000}$ and prices $p_1, \ldots, p_{1000}$ of a thousand houses are a thousand pairs of numbers, which you can think of as the coordinates of a point $$D = (a_1, \ldots, a_{1000}, p_1, \ldots, p_{1000})$$ in two-thousand-dimensional space. For convenience, I'll call two-thousand-dimensional space "data space." Your data set $D$ is a single point in data space.

If the relationship between area and price were perfectly linear, the point $D$ would sit in a very special region of data space, which I'll call the "linear sheet." It consists of the points $$M(\rho, \beta) = (a_1, \ldots, a_{1000}, \rho a_1 + \beta, \ldots, \rho a_{1000} + \beta).$$ The numbers $\rho$ and $\beta$ are allowed to vary, but $a_1, \ldots, a_{1000}$ are fixed to be the same areas that appear in your data set. I'm calling the linear sheet a "sheet" because it's two-dimensional: a point on it is specified by the two coordinates $\rho$ and $\beta$. If you want to get a sense of how the linear sheet is shaped, imagine a thin, straight wire stretched across three-dimensional space. The linear sheet is like that: it's perfectly flat, and its dimension is very low compared to the dimension of the space it sits inside.

In a real neighborhood, the relationship between area and price won't be perfectly linear, so the point $D$ won't sit exactly on the linear sheet. However, it might sit very close to the linear sheet. The goal of linear regression is to find the point $M(\rho, \beta)$ on the linear sheet which sits the closest to the data point $D$. That point is the best linear model for the data.

Using the Pythagorean theorem, you can figure out that the square of the distance between $D$ and $M(\rho, \beta)$ is $$[p_1 - (\rho a_1 + \beta)]^2 + \ldots + [p_{1000} - (\rho a_{1000} + \beta)]^2.$$ In other words, the distance between the data point and the model point is the total squared error of the model! Minimizing the total squared error of a model is the same thing as minimizing the distance between the model and the data in data space.

As Chris Rackauckas pointed out, calculus gives a very practical way to find the coordinates $\rho$ and $\beta$ that minimize the distance between $D$ and $M(\rho, \beta)$.

Answer 9

@Chris Rackauckas and @EDM's answers are spot on. There are many ways to approach simple linear regression that don't require plotting or visual explanations of ordinary least squares estimation, and they give very solid explanations of what actually happens when you're running OLS.

I might add that using scatterplots as an instruction tool to learn any kind of new modeling procedure, whether it's old school parametric model, advanced machine learning stuff, or bayesian algorithms, graphing can help cut down on the time it takes to learn what a particular algorithm does.

Graphing is also very important for exploratory data analysis when you are first beginning to work with a new dataset. I have had situations where I collected lots of data, worked out the theory, carefully planned out my model, and then ran it, only to end up with results that essentially had no predictive power. Plotting bivariate relationships can take out some of the guesswork: in your example, it's possible that home price is linearly related to area, but maybe the relationship isn't linear. Scatterplots help you decide if you need higher order terms in your regression, or if you want to use a different method than linear regression, or if you want to use some sort of nonparametric method.

Answer 10

Google for Anscombe Quartet.

It shows 4 sets of data which on inspecting numerically do not show much difference.

However, on creating a visual scatter plot, the differences become dramatically visible.

It gives a pretty clear view why you should always plot your data, regression or no regression :-)

Answer 11

We want to have a solution that minimizes the difference between the predicted and actual values.

We assume that the $y=bx+a$ i.e. there is a linear relationship.

We don't care whether the difference between predicted and actual $y$ is positive or negative assume that distribution of errors of $y$ posses certain properties.

If we assume that the distribution of errors is normally distributed it turns out that there is an analytical solution to this minimization problem. The the sum of squares of differences is the best value to minimize for a best fit. But normality is not required in general case.

There isn't much more to it really.

The geometrical interpretation comes handy because sum of squares has the interpretation in the form of sum of distances of the dots on the scatter plot from the $y=bx+a$ line. And human eye is very good at approximating the line that corresponds to the best fit. So it was handy before we had computers to find the fit quickly.

Nowadays it is left more as a comprehension help but is not necessary to have to understand linear regression really.

EDIT:replaced the normality of errors assumption with a correct but less concise list. Normality was required to have an analytical solution and can be assumed for many practical cases and in that case sum of squares is optimal not only for the linear estimator and maximizes likelihood as well.

If further the assumption of normality of error distribution holds then the Sum of Squares is optimal among both linear and non-linear estimators and is maximizing the likelihood.

Answer 12

I'm coming late to this conversation but I just want to add something that I think might aid understanding.

The computation for finding the OLS estimator relies on math from linear algebra involving matrices. It's my understanding there are a few different ways to do this. A = QR shows how to create an orthonormal matrix and an upper triangular matrix.

However, in the case of simple linear regression for an ordinary least squares estimate, meaning regression with only one independent and one dependent variable, you can use a shortcut equation in order to figure out the slope and the intercept of the line instead of leaning on the more complicated linear algebra math.

In the case of simple linear regression, the slope and intercept follow these neat closed-form equations: the slope can be calculated by multiplying the correlation r by the quotient of standard deviation of y over standard deviation of x. In this below equation, a refers to the slope and sy and sx refer to the standard deviation of y and the standard deviation of x, respectively.

a = r * ( sy / sx )

The intercept of the line of best fit for ordinary least squares simple linear regression can be calculated easily after you calculate the slope of the line of best fit. You do this by subtracting the slope of the line of best fit from mean of y, then multiplying the result by the mean of x. In the equation below, i refers to y-intercept and the straight line over the x and y values is a way of referring to the mean of x and y respectively; we refer to these terms as x-bar and y-bar.

i = y-bar - r * (sy / sx) * x-bar

Our R code looks like this:

r_slope <- ( sd(mtcars$mpg) / sd(mtcars$wt) ) * cor(mtcars$mpg, mtcars$wt)
r_intercept <- mean(mtcars$mpg) - r_slope * mean(mtcars$wt) 

I know this only covers the case of simple linear regression with one x and one y but I hope it's helpful in visualizing this case. In simple linear regression, you can definitely find the slope and intercept of the line without knowing linear algebra.