2
$\begingroup$

I'm doing some work for a summer school project and I've been asked to model some data using a 'linear least squares' model. I've done all that and analysed the results and the summary statistics look awful.

What I'm curious about is whether a linear least squares model is only intended to apply to the typical model always taught in stats classes with normal errors and the like, or whether I could use a GLM to model the data? All they said was a 'linear least squares' model. I've tried to contact my prof a few days ago but he hasn't got back to me since and this is due in a day or two.

Thanks for any advice.

Edit: I'm modelling stock data, so looking at features of the stock on each day and then the quantity the price moved over a day of trading.

$\endgroup$
3
  • 1
    $\begingroup$ What are you trying to model? OLS can be pretty robust in many situations, but often other generalized linear models may be more appropriate. Please edit your question to include a bit more detail about your analysis project. It's hard to give "decent" advice without details $\endgroup$ – Marquis de Carabas Apr 19 '16 at 19:44
  • $\begingroup$ Based on your edit the road you want to go down is not GLM, but rather (1) think about what covariates are you including in your model (e.g. do I need interactions, or perhaps transformations of them), (2) does the outcome actually look normally distributed (e.g. perhaps you need transform of outcome), (3) is my data independent? (if not, as I suspect is the case for time series data like stocks you should consider time series models like ARMA or GARCH, etc) $\endgroup$ – bdeonovic Apr 19 '16 at 20:28
  • $\begingroup$ By GLM do you mean generalized linear model or general linear model? If the first then not in general, but if the second then yes, as long as you add an appropriate adjective (such as "weighted", perhaps) to "linear least squares", or you're in a situation where that's understood. $\endgroup$ – Glen_b Apr 20 '16 at 0:04
5
$\begingroup$

"Ordinary linear regression" has perhaps a thousand different formulations. In the optimization world, it is called least squares since it minimizes the squared error of the residuals. It also maximizes the likelihood of bivariate normal data. Intuitively, this gives a pretty good "line of best fit" on a XY scatter plot of your data.

In the generalized linear model world, a OLS model is called a GLM with Gaussian error structure and identity link. GLMs use probability models for the outcome to formulate a maximum likelihood based approach to estimation. There are other GLMs to maximize likelihood for binomially distributed outcomes, Poisson distributed outcomes, etc. These do not minimize the squared error since they take account of a mean-variance relationship.

In short, GLMs are a family of regression models that depend on the type of probability model for the outcome, inclusive of ordinary linear regression. If the outcome is a count or a proportion, you should consider a GLM specially designed to that purpose.

The choice of whether to use a GLM depends on the nature of the data, not the results obtained from a model. In my work, when I fit a bad model, I consider the data and whether I "missed" something, but most often I resolve to simply report the "bad" results, such as "We did not find evidence of association between X and Y, a scatter plot revealed no clear trend between the values, and the r^2 for the fit was very low (r^22=0.15). A larger sample may reveal a more significant trend, but confidence intervals for the regression slope indicate that this would be at most X.X-X.X which is not meaningful.".

$\endgroup$
1
$\begingroup$

If your task is to use a "linear least squares" model (for whatever purpose), then the only generalized linear model that qualifies is the normal linear model. So you are back to square one.

The linear least squares model is quite flexible though: You can add interactions and non-linear terms (squares, splines, ...), you can (log-)transform the response variable and/or the covariables etc. If you are still skeptical, then you can try to compare such a model with your favorite GLM and investigate the differences (in predictions, in p-values, ...).

$\endgroup$
1
$\begingroup$

It appears you have a time-series data (i.e. for a single dependent variable in each period), and it is stock data.

Comments
a) There are enough indications that in the basic regression modelling framework, the error term related to stock data is better modeled as following the Laplace distribution and not the Normal.

b) With time series data, possible autocorrelation and even non-stationarity of the data will create various issues for the least-squares estimator, bias being the milder one, but also possible inconsistency and even spurious results.

Now, either you were given the task to just practice the estimation routine (in which case results don't really matter), or it is expected of you to see what you can do, in light of the above possible issues, so that it will become acceptable to still apply a "linear least-squares" estimation framework. This second possibility is of course the one where learning resides.

Note that "linear least-squares" does not imply any distributional assumption.

$\endgroup$
1
$\begingroup$

Least squares is a way of defining the objective function of a model, which refers to the core of the problem that models are trying the solve: to have the least amount of errors in making predictions. You can define errors in different ways: sum of absolute error, sum of square of error, weighted errors, or other definitions.

You can use the good old method that Gauss used a few hundreds years ago, OLS, or its many modern variations, such as penalized (or regularized) linear regression with a loss function that is akin to least squares.

To give a context, you can even use different types of gradient descent to solve a least squares problem.

GLM refers to the model specification, not the definition of loss. You can certainly try to use least squares. The original GLM formulators John Nelder and Robert Wedderburn proposed an iteratively reweighted least squares method for maximum likelihood estimation of the model parameters. Maximum-likelihood is the default method on many statistical computing packages such as SAS and R.

$\endgroup$
0
$\begingroup$

GLM is the generalization of linear regression when distribution function is normal. From the perspective of normal distribution GLM has closed form expression for maximum likelihood. According to Gauss Markov Theorem, Linear Least square are Best Linear Unbiased estimator which does not assume distribution is normal.

Wikipedia does explain the difference.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.