Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [least-squares]

Refers to a general estimation technique that selects the parameter value to minimize the squared difference between two quantities, such as the observed value of a variable, and the expected value of that observation conditioned on the parameter value. Gaussian linear models are fit by least squares and least squares is the idea underlying the use of mean-squared-error (MSE) as a way of evaluating an estimator.

0
votes
0answers
334 views

The statement of homoscedasticity of variance when describing the OLS model

In an applied econometrics paper, the author states the model to be estimated as: Why does the author claim homoscedasticity? This isn't making sense to me; can't the population variance-covariance ...
4
votes
0answers
295 views

Estimating parameters of an unknown PID controller

Say that I have your standard PID controller at work. To keep it extremely simple imagine I have a target $x^*$ on the variable $x$. Then the controller is: $y(t) = K_p ( x^* - x_t) + K_i \int_0^t (x^...
8
votes
3answers
286 views

OLS: $E[\epsilon_{it}^T\epsilon_{it}] \not= 0$ in 1st equation biases standard errors in 2nd equation?

Suppose ${X_{it}},{Y_{it}}$ are time series with $X_{it}\sim N(0.1,1)$, ($\sigma^2(Y_{it}) = 1$ and $mean(Y_{it})$ is similar to that for $X_{it}$, but changes when the dummy = 1). and $t \in \{1,2,......
5
votes
3answers
2k views

How to control for market return in an (SPSS) OLS?

Please consider the following panel dataset: ...
2
votes
2answers
2k views

“Least square root” fitting? A fitting method with multiple minima

I am looking for the name of a fitting method that will work even if points from multiple dataseries are meshed together. As far as I understand there are two major methods, least squares and least ...
1
vote
2answers
2k views

SPSS dummy variables in OLS

I have a timeseries dataset holding stock data for a large set of companies. Assume the following subset, where obsDay is the observation day (148 days in reality) ...
1
vote
1answer
4k views

SAS for regression with categorical and quantitative explanatory variables

I am analyzing growth over time for 5 different cultivated forms (cultivars) of maize. Graphing the data reveals a clear linear pattern for all the cultivars in the time interval I am interested in. ...
4
votes
2answers
2k views

Find residual sum of squares

Let $Y_1, Y_2,Y_3$ be uncorrelated random variables with common variance $\sigma^2>0$ such that $$ \begin{aligned} E[Y_1]&=\beta_1+\beta_2, \\ E[Y_2]&=2\beta_1 \\ E[Y_3]&=\beta_1-\...
23
votes
1answer
5k views

Linear regression prediction interval

If the best linear approximation (using least squares) of my data points is the line $y=mx+b$, how can I calculate the approximation error? If I compute standard deviation of differences between ...
3
votes
1answer
2k views

“Studentized” bootstrap confidence interval for variance of OLS error terms

Suppose that one has the usual regression model $\mathbf{y} = \mathbf{X} \boldsymbol{\beta} + \boldsymbol{\varepsilon}$, where each $\varepsilon_t$ is iid distributed with $\mathbb{E}(\varepsilon_t) = ...
5
votes
1answer
176 views

Aggregating all measurements per x-value in least-square fitting

We want to select one out of a given set of (continuous) functions that best matches a set of observations $\{(s_i,c_i) \mid 1 \leq i \leq N\} \subseteq\mathbb{N} \times \mathbb{N}$ (input size and ...
15
votes
2answers
4k views

Measures of residuals heteroscedasticity

This wikipedia link lists a number of techniques to detect OLS residuals heteroscedasticity. I would like to learn which hands-on technique is more efficient in detecting regions affected by ...
2
votes
1answer
106 views

When can we add a statistical touch to least square optimization problems?

I am trying to connect the dots between statistics and linear algebra/optimization. As you know, Least Square problems are linear algebra and optimization problems. But they also can be connected to ...
3
votes
1answer
384 views

How can you relate the coefficients of a multivariate regression of ${\bf Y} \sim {\bf X}$ to the coefficients of ${\bf X} \sim {\bf Y}$?

Is anyone aware of a orthogonal multiple regression library that is implemented in say R, Scipy, Matlab, Octave, etc.? (Or even fortran/C...) If I'm not mistaken, it would not be difficult to write ...
3
votes
0answers
188 views

Is is correct to compare t-statistics of different pairs of cointegrated timeseries?

I am testing for cointegration all the pairs from a set of 100 stocks. I run an Ordinary Least Square Regression on each pair and then I test for the existence of unit roots in the residuals. I am ...
4
votes
1answer
2k views

Does a regression tree strictly dominate OLS in prediction?

Since OLS tries to measure E[Y|X], and regression trees try to partition the data into different branches, then take means of the response under different branches, is it reasonable to say that ...
1
vote
1answer
167 views

Confusion regarding least squares method

I am having some confusion regarding least squares method. Actually, least squares method is for minimizing the square of the $L_2$ norm of $Ax-b$ as given in this video lecture. However I am confused ...
6
votes
0answers
166 views

Scope of non-linear least squares

edit: tl;dr: I can coerce a lot of optimization problems to take the form of a non-linear least squares problem, but does it make sense to do so? Suppose we have some empirical data $P=\{(x_i', y_i')\...
41
votes
5answers
67k views

Regression when the OLS residuals are not normally distributed

There are several threads on this site discussing how to determine if the OLS residuals are asymptotically normally distributed. Another way to evaluate the normality of the residuals with R code is ...
8
votes
1answer
5k views

OLS vs. logistic regression for exploratory analysis with a binary outcome

In the idealized logistic model, we obtain an S-shaped curve linking each continuous IV to the DV. But in practice this S-shape infrequently occurs, making the logistic approach seem a little less ...
6
votes
4answers
151 views

Could logistic regression be used to detect large errors in least squares regression?

I have the following linear model: $$w^*=\text{arg min}_w\sum_{i=1}^N \bigg(Y_i-\sum_{j=1}^M X_{i,j}\times w_j\bigg)^2$$ Let $T \in N^*$ and $e_i=|Y_i-\sum_{j=1}^M X_{i,j}\times w_j|$. It's ...
2
votes
1answer
1k views

Least squares with non-linear constraints

I have the following problem: I want to minimize a least square problem with non-linear restrictions. The start model has the following form: $$w^*=\text{min arg}_w \sum_{i=1}^{N} (y_i-\sum_{j=1}^3 ...
2
votes
1answer
728 views

How can I minimize this least squares problem with inequality constraints?

I have a least square problem with two different inequality problems. I can not use NNLS because its just solves least square problem with equality and inequality problems or just one inequality ...
5
votes
1answer
325 views

Nonlinear regression model linear in some parameters

In a set of lecture notes that I stumbled upon online, the author discusses a nonlinear regression model, which is linear in some parameters, like this $$ y = \theta_1 + \theta_2\exp \left( {\...
3
votes
0answers
368 views

Standard errors in weighted least squares

My data is a panel of countries by year. Suppose my main RHS variable is a country's GDP and my main LHS variable classifies countries by whether the country is a democracy. Is it desirably to ...
17
votes
1answer
7k views

MLE vs least squares in fitting probability distributions

The impression that I got, based on several papers, books and articles that I've read, is that the recommended way of fitting a probability distribution on a set of data is by using maximum likelihood ...
2
votes
1answer
1k views

Deriving OLS estimates using method of moments

I've worked the slope all the way down to $\sum [x_i(y_i - \bar{y})] = \hat\beta_1 \sum[x_i(x_i - \bar{x})]$ But I can not figure out how to show the steps for: $\sum[x_i(y_i - \bar{y})] = \sum(x_i -...
4
votes
2answers
7k views

2 period difference-in-differences fixed effects versus OLS

I have a question on the difference-in-differences estimator. Suppose my data consists of two periods and the treatment is administered to some of the individuals in period $t = 2$. I estimate this ...
20
votes
2answers
6k views

Least squares logistic regression

I have seen it claimed in Hosmer & Lemeshow (and elsewhere) that least squares parameter estimation in logistic regression is suboptimal (does not lead to a minimum variance unbiased estimator). ...
3
votes
0answers
696 views

Pregibon test for linearity vs. Ramsey's RESET test

Does every Ordinary Least Squares (OLS) regression model have to pass both the Pregibon Test for Linearity (sometimes called link-test) and the Ramsey RESET tests? I am working on an OLS model and it ...
3
votes
2answers
442 views

Significance testing of slopes with replicates

I've made replicate measurements of a parameter (fluorescence) which is expected to increase with time, and I'm having a hard time understanding how to test for significance of the slope of the ...
2
votes
1answer
2k views

Quadratic term and variance inflation factor in OLS estimation

One question I have carried around with me for a while is related to including quadratic terms for model specifications. I wonder why it is considered ok to include linear and quadratic terms into OLS ...
0
votes
1answer
1k views

How can I get residuals output in a variable or to act like a data frame in R?

I have an object: noise.lm it's just a simple linear model with an X and Y. when I type in resid(noise.lm) it produces ...
4
votes
1answer
219 views

Least angle regression for a set of vectors?

As far as I know, LARS solves the following problem (using the same notation as Efron et al. Least angle regression): Given a vector y, and a matrix X. Pick some column vectors from X, and express ...
4
votes
2answers
1k views

What are similarity measures between a line and a set of points?

A colleague discussed about a concept of similarity between two different entities line and a set of points. My first guess for solution was considering distances squared ($d^2$) as in LS. So for the ...
2
votes
1answer
131 views

Regression with complex causal structure

I have already run a whole bunch of OLS and found the following Regress P= B1*L+e_1, found b1<0 Regress X= B2*L+e_2, found b2>0 Regress X= B3*P+e_3, found b3<0 I want to build a case with ...
4
votes
2answers
363 views

Formula for single linear regression for dataset that has uncertainties on both x and y

I'm going to teach classes on Physics Laboratory on First year of Bachelor studies. In most of the excercises during data analysis students will have to fit a line to measurements they have taken. I ...
5
votes
2answers
925 views

Multiple regression and OLS. How to choose the best “non-linear” specification?

Let's say I have to make a multiple regression like: $ Y_i = \beta_0 + \beta_1 x_i + \beta_2 w_i + ... +\beta_3 z_i + \epsilon_i $ Then I run a Ramsey RESET test upon it and discover that my linear ...
7
votes
3answers
20k views

Selecting best model based on linear, quadratic and cubic fit of data

I have a Java code that performs a linear regression on a set of data using the Gauss-Jordan elimination. It calculates a linear, quadratic and cubic functions using the least squares method. My ...
0
votes
0answers
213 views

Error amplification in linear regression with uncertainties in design matrix

I am running a least squares regression, and I am trying to estimate the uncertainty on the fit solution. The problem is, I have an uncertainty in my design matrix coefficients as well as in my ...
1
vote
1answer
1k views

How do I handle very different weights in a least squares fit?

I'm performing a weighted linear least squares fit, where the weights correspond to the number of counts of a specific observation. Due to the nature of the data, it is possible that a small handful ...
1
vote
3answers
2k views

Least squared regression where the coefficients switch signs upon the addition of new variable

I am conducting a least square regression using the python library numpy. When I run OLS (ordinary least square regression) over just the variable IE6 I get this output (with the key takeaway being ...
24
votes
2answers
13k views

Why is RSS distributed chi square times n-p?

I would like to understand why, under the OLS model, the RSS (residual sum of squares) is distributed $$\chi^2\cdot (n-p)$$ ($p$ being the number of parameters in the model, $n$ the number of ...
3
votes
0answers
5k views

What to do when ovtest and linktest in Stata suggest model misspecification?

I have a sample that consists of 50 observations. The base model of the OLS-Regression with three control variables, two of them significant, has a $R^2=0.50$ and its F-Value is 7. Both ...
1
vote
2answers
115 views

Fitting a continuous result to categorical predictors semiparametrically

Suppose one has a relatively large number of observations, each of which consists of a continuous result and a small number (2 or perhaps 3) of categorical variables, each of which has a large ...
5
votes
1answer
739 views

Why is semipartial correlation cited so seldom?

In OLS regression, I find the semipartial correlation (a.k.a. part correlation) to be a very useful indicator. When squared, it shows each predictor's unique contribution to explained variance in the ...
1
vote
0answers
467 views

Predictive power (or $R^2$) adjusted for certain variables

I will frame this question for Ordinary Least Square (OLS) regression, but my question is for both OLS and Logistic. Let's say we data over 10000 different individuals. For each person we have three ...
14
votes
2answers
10k views

Why is a projection matrix of an orthogonal projection symmetric?

I am quite new to this, so I hope you forgive me if the question is naïve. (Context: I am learning econometrics from Davidson & MacKinnon's book "Econometric Theory and Methods", and they do not ...
0
votes
0answers
54 views

Summation representation for multivariate regressions (or other time-saving techniques) [duplicate]

Possible Duplicate: Efficient online linear regression Is there a summation representation for multivariate regressions? For example, if I regress $y$ on $X$ instead of using $\hat \beta = (X'X)^...
3
votes
1answer
92 views

Does including both raw and per capita measures as predictors reduce significance of either predictor?

I'm running a regression on independent variables, some of which are measured in different units, for example: The amount of broadband connections in a country The amount of broadband connections in ...